Benchmarking

RunLore is model-agnostic: you pick the model behind the investigation loop. This page shows how to benchmark several models against RunLore’s own eval harness in one command and publish an honest comparison.

RunLore’s own nightly numbers are public: the replay eval publishes a per-scenario scorecard — pass/fail, recall outcomes, confidence calibration, model, date, and cost — on every run, red or green. → The nightly scorecard

One command

lore eval --compare eval/compare.example.yaml --cases examples/eval -n 3

This benchmarks every model in the spec against the same replay cases, grades each run with one fixed judge, and writes an aggregated report to eval/reports/<stamp>-compare.md (human) and .json (machine). It needs no runlore.yaml at all as long as the spec’s judge: block supplies the judge (or you pass --judge-* flags) — the spec already carries its own per-entry models. Without a judge: block, --judge-* flags, or a config file’s config.model, --compare fails fast with a clear error instead of silently grading nothing.

The comparison spec

A small YAML listing the models to benchmark and (optionally) the judge. See eval/compare.example.yaml:

judge:                       # optional; one fixed judge for every entry (blind grading)
  provider: anthropic
  model: claude-opus-4-1
  api_key_env: RUNLORE_JUDGE_API_KEY
models:
  - name: haiku-4.5          # report label (must be unique)
    provider: anthropic      # openai (default) | anthropic | gemini
    model: claude-haiku-4-5-20251001
    api_key_env: RUNLORE_ANTHROPIC_API_KEY
    prices: {input_usd: 1.00, output_usd: 5.00}   # optional → enables the cost column
  - name: gpt-5-mini
    provider: openai
    base_url: https://api.openai.com/v1
    model: gpt-5-mini
    api_key_env: RUNLORE_OPENAI_API_KEY
    effort: medium           # reasoning_effort (openai-compatible only; not gemini)
    prices: {input_usd: 0.25, output_usd: 2.00}
  - name: local-qwen3        # keyless: a local vLLM/Ollama endpoint
    provider: openai
    base_url: http://localhost:8000/v1
    model: qwen3-30b

Entry fields: name (required, unique), model (required), provider, base_url, api_key_env (empty = keyless), effort (openai/anthropic only — gemini is rejected), prices (optional; omit to omit the cost column for that entry). Unknown keys are rejected so a typo in a published spec fails loudly.

The judge precedence is: --judge-* flags → the spec’s judge: block → config.model (only checked when a config file is loaded). Keeping the judge in the spec makes a published comparison self-describing — the judge disclosure travels with the results — and it’s what lets --compare run with no runlore.yaml present. If none of the three supplies a judge, the command errors instead of quietly disabling rubric grading.

The report

The comparison report has one row per model, in spec order (a comparison should not silently reorder by score), with these columns:

A second table gives the per-case pass rate (k-of-n) for each model, with flaky flagged when runs disagree too much to trust. Case rows are sorted by name and the JSON is deterministic, so two reports diff cleanly.

Rubric dimensions and the pass gate are defined in eval/rubric.md. Token usage is the provider-reported count per completion (see providers.Usage), summed by the runner’s CountingModel wrapper. The in tok / out tok columns cover the entry’s own model only — grading is done by one fixed judge shared across entries, so charging its tokens to whichever entry happened to be running would make the comparison meaningless. The judge’s own spend is reported separately on stderr at the end of the run:

eval: judge: model spend: 412340 input / 18220 output tokens (~$1.3762 at model.pricing)

Spend controls

An eval is the easiest way to spend real money quickly: it runs the full investigation loop over a whole corpus, several times per case, unattended. Two independent ceilings bound it, and they answer different questions.

The per-case ceilings are deliberately the same config keys as production, not eval-only ones: an eval replays the production loop, so a case that would be nudged or hard-killed on your cluster must be cut short here too — otherwise the pass rate describes a loop no deployment ever runs. A --compare run with no runlore.yaml still gets the shipped default per case.

--max-total-tokens is the run-level ceiling those cannot express: a 30-case corpus at -n 10 under the default 400000 per-case ceiling authorises 120 million tokens, and nothing about “per case” says otherwise. It is denominated in tokens, not dollars, because a campaign routinely drives several different models at once (the entries under test plus the judge) and a single USD ceiling would have to price all of them from one rate card — misreporting exactly the multi-model run it most needs to bound.

When it trips, every further completion is refused and the campaign stops starting new cases; the report covers the cases that actually ran, and the halt says so:

eval: STOPPED after 12/30 cases — campaign token ceiling exceeded (--max-total-tokens=2000000, spent 2001430).

It cannot be a strict cap — a completion’s size is unknowable before it is made, so the crossing call itself always completes and the overshoot is bounded by one completion. It also covers only model completions: a --live campaign’s setup/teardown shell steps mutate a real cluster and spend no tokens at all, so nothing here bounds them.

Publishing results honestly

RunLore’s positioning is honesty about model performance. When you publish a comparison, disclose:

1. N runs. State the -n you used (median over N; the pass gate is k-of-n at ≥70%). A single run is not a benchmark — one lucky pass hides a flaky model. Report the per-case table so flakiness is visible.
2. The judge model. Grading is by an LLM judge (blind — the judge never sees which model produced a result). The report prints the judge identity; keep it. Prefer a judge stronger than any model under test, and never let a model grade itself in the same run (bias).
3. Replay vs live. These numbers are replay: fixed, recorded evidence. They isolate reasoning quality but do not capture live-cluster tool flakiness, latency, or evidence-gathering gaps. Live-fire (lore eval --live) is the harder, cluster-bound test. Label replay results as replay.
4. ITBench context. Independent work (ITBench, IBM/ICML 2025, see docs/prior-art.md) found frontier models identify the root cause < 50% of the time and fully resolve only ~11–14% of real K8s incidents. Treat sub-50% as the baseline; a high replay pass-rate is a ceiling, not a field number. Design for failure and make honest uncertainty a feature.
5. Versioned report. Commit the generated eval/reports/<stamp>-compare.md and .json so a published claim points at a reproducible artifact (the spec, the case corpus at that commit, the judge, and N are all recoverable).

Offline validation (keyless CI)

The comparison pipeline is covered by an integration test that runs it end-to-end against a local, keyless OpenAI-compatible mock endpoint (internal/app/eval_compare_test.go): the mock streams the tool calls the replay loop needs (what_changed → query_metrics → query_logs → submit_findings) and answers the judge’s forced submit_grade tool with a fixed rubric grade plus a token-usage block. So CI exercises load → per-entry replay → coverage → blind grading → aggregation → report writing with no API key and no network. Run it with:

go test ./internal/app/ -run TestRunEvalCompareOffline

Go micro-benchmarks (hot paths)

Hermetic testing.B benchmarks guard the hot paths the 2026-07-19 audit named: no network, deterministic fixtures, safe to run anywhere. They are deliberately NOT part of the CI gate (numbers on shared runners are noise); run them locally when touching these packages and compare against your own baseline:

go test ./internal/whatchanged/ -bench BenchmarkRemote -benchtime 5x -run '^$'
go test ./internal/catalog/     -bench Benchmark       -benchtime 5x -run '^$'
go test ./internal/outcome/     -bench BenchmarkLedger -benchtime 5x -run '^$'
go test ./internal/trigger/ ./internal/coalesce/ -bench . -benchtime 5x -run '^$'

What each guards: clone-vs-mirror (whatchanged), BM25 rebuild and cold-vs-warm embed cache + BM25/hybrid query cost (catalog), cold-start replay, one compaction cycle, and the O(1) OpenCounts read (outcome), and per-alert admission under storm (trigger, coalesce).