Eval scorecard

The nightly replay eval, published in full — per scenario, red or green.

Everything above the Latest published run heading is written by hand and lives in the repo; everything below it is generated by the nightly and replaced on every publish.

What a scenario is

One recorded incident: the exact output every tool returned during a real failure, frozen into a YAML file, together with the cause the investigation is expected to reach (must_contain, min_confidence, and for the recall cases expect_recall). The nightly replays those files through the real investigation loop — same prompts, same tools, same recall and verify gates — with no cluster attached.

So the pass-rate scores the model and the loop reasoning over fixed evidence. Nothing else in the run can move.

Read the cases before you read the number: examples/eval/. That directory is the suite as it stands today; the published run below states its own date and how many scenarios it covered. If the directory now holds more, the suite has grown since that run and the next nightly picks up the difference.

What the number does not mean

Not a live-cluster result. Replay isolates reasoning over fixed evidence, so it says nothing about tool flakiness, latency, or the evidence gaps that only appear when a query actually runs against a cluster. Live-fire (lore eval --live) is the harder test, and it is not what this page reports.

Not a leaderboard. The corpus is small and hand-built; it exists to catch regressions in RunLore. That makes the pass-rate meaningful only against its own history — the run-history table the scorecard carries. To weigh models against each other, run the benchmarking harness on your own models rather than reading this one number as a ranking.

Not a field result. ITBench (IBM/ICML 2025) found frontier models identify the root cause < 50% of the time and fully resolve only ~11–14% of real K8s incidents. Treat sub-50% as the baseline and a high replay pass-rate as a ceiling — see Prior art.

A run below the 70% CI gate publishes here exactly like a green one — same schedule, same detail, no editorial pass. A pass-rate that drops is visible on this page before it is fixed; that is what publishing it is for.

Latest published run

Published 2026-09-16T11:00:04Z from the eval-scorecard branch. It covered the 6 scenarios the suite held at that time. The case definitions are the suite as it stands today — more files there than this run covered means it has grown since.

Auto-published by .github/workflows/eval.yaml. Reproduce it yourself:

lore eval -config eval/ci.runlore.yaml -cases examples/eval -n 5 -fail-under 0.7

Latest run: 2026-09-16T11:00:02Z · model openai/glm-4.5-air · 2/6 scenarios reached (33%) · n=5 runs/case, k-of-n bar 70% · est. cost $0.31 (1.2M in / 64.1k out tokens)

Scenarios (latest run)

Cost per investigation

Median provider-reported tokens per case on openai/glm-4.5-air, priced at $0.20/MTok in · $1.10/MTok out. Replay evidence, so tool latency and live-cluster variance are excluded.

Confidence calibration

• Confidently wrong (missed with median confidence ≥ 0.70): 4 — harbor-chart-bump, node-eviction-no-commons, poisoned-recall-rejected, poisoned-recall-verify
• Underconfident (reached with median confidence < 0.50): none

History

Newest first, last 30 shown — the full log is history.jsonl. A run that reached no answer to score at all is labelled in the pass-rate column instead of scored — it is not a 0%.