Troubleshooting

How to diagnose the most common “why didn’t RunLore do X?” situations. RunLore exposes two diagnostic channels:

• Structured logs — every decision is a one-line slog event. Set logging.format: json (or text) and logging.level: debug for the most detail. Each section below quotes the exact msg= value to grep for.
• Metrics — runlore_* Prometheus series, exposed when telemetry.metrics_enabled: true. See Observability for the full catalog and a Grafana dashboard.

An alert fired but no investigation started

By far the most common case. RunLore has five independent layers that can each decide not to investigate — dedup, debounce, the coalescer’s cooldown, the per-trigger recurrence cooldown, and a human 🔕 silence — and they answer different questions (“is this the same still-firing alert?”, “did it clear before we even started?”, “did this correlation just get investigated?”, “did THIS trigger just get a conclusive answer?”, “did a human say stop?”). Ruling one out does not rule out another, so an operator chasing “why was it quiet” needs to know all five exist before assuming the layer they found is the only candidate. Work from ingress inward.

1. Is there a per-incident decision line? Every admitted alert produces exactly one log event:

msg=incident alert=<name> severity=<sev> namespace=<ns> investigate=<bool> reason="<reason>"

Trigger filtering and dedup are not counted by any metric — the incident log line is the only place they surface. Grep it first.

2. No incident line at all? The alert never reached the trigger pipeline:

• Source not enabled. The webhook only mounts when sources.alertmanager: {} is set. A typo such as alertmanagr: now fails startup with unknown source(s) [alertmanagr] under \sources:` — known sources are [alertmanager gitops pagerduty]` (older builds silently did nothing). Check the startup logs.
• Webhook rejected at ingress. server.webhook_token_env is mandatory once any model is configured (serve fails closed — an anonymous webhook must not bill the model); it is also required by config.Validate under actions.mode=auto. When set, Alertmanager must send Authorization: Bearer <token>. A 401 means the token is missing or wrong. The request body is also capped at 1 MiB.
• Metric cross-check. runlore_alerts_received_total counts alerts that passed initial decoding and Decide. Flat while alerts are firing ⇒ they’re being rejected at ingress (auth/parse) or the source isn’t mounted.

3. Admitted, but still nothing ran? Compare runlore_investigations_started_total against runlore_alerts_received_total, then:

A GitOps failure didn’t trigger an investigation

The GitOps-failure watcher (sources.gitops: { enabled: true }) debounces before firing, to filter reconcile-churn transients:

• runlore_gitops_failures_debounced_total rising ⇒ the failure cleared within the debounce window and was dropped as transient. Log: msg="gitops-failure cleared within debounce window; dropping transient".
• Tune with triggers.gitops_failures.debounce (default 60s; explicit 0 fires immediately on every Ready=False).

The investigation ran but timed out / came back empty

Check runlore_investigations_completed_total{result=…} — the result label tells you how it ended:

Supporting metrics: runlore_tool_calls_total{tool,result} and runlore_model_requests_total{provider,result} (watch the result="error" slice), runlore_model_responses_truncated_total (completions cut off at the output-token ceiling — a frequent cause of inconclusive), and runlore_investigation_duration_seconds{result}.

result="budget_exceeded" only counts the runs that died. Watch runlore_investigation_budget_trips_total{reason,stage} for the rung above it: stage="nudge" is an investigation that hit a ceiling, was forced to conclude early and still delivered findings — it looks like a normal resolved run in every other series, so a steadily rising nudge rate is how you learn your ceilings are cutting work short before any of them start failing.

The curator didn’t open a PR

RunLore files a KB pull request only for novel, confident findings — by design it does not file for everything. Check runlore_curations_total{kind="pr",result=…} and the curator log:

If you expected a PR and got chat-only or duplicates, the finding simply wasn’t novel/confident enough — tune forge.min_confidence / forge.dup_score if the thresholds are wrong for you.

Knowledge-gap issues (opened by the separate lore curate recurrence agent, not the live curator) are not counted by any metric — they only log msg="opened knowledge-gap issue".

The PR opened, but the entry will never be recalled

A KB pull request can merge cleanly and still be dead on arrival: recall matches an entry to an incident through its resource: frontmatter, so a resource that no workload can ever equal makes the entry unreachable — worse than omitting it, because a non-empty resource also disables the scopeless fallback. RunLore checks each draft before filing and warns rather than blocking, so the PR still opens and a human can fix the frontmatter:

Both carry field=, issue= and title= so you can find the entry. If an already-merged entry never matches anything, this is the first thing to check — it shows up in the recall section below as no_resource_match.

Both are also counted, so you do not have to be watching the log to find out: runlore_kb_draft_defects_total{defect="unrecallable_resource"} for the first row and {defect="merge_gate"} for the second, from both entry writers (the curator and the @runlore note: route). Nothing else would tell you — runlore_curations_total{kind="pr",result="opened"} counts the pull request as a success whichever line fired. Alert on unrecallable_resource: it is the half that merges cleanly and stays silent. See Observability.

Recall never fires (every incident runs the full loop)

Instant recall requires catalog.instant_recall.enabled: true and a confident catalog hit. Check:

• runlore_recall_hits_total is zero, and runlore_recall_rejections_total{reason=…} shows why candidates were rejected:

  reason	meaning	lever
  no_resource_match	no candidate’s stored resource agreed with the incident’s workload — the reranker was never reached	instant_recall.require_workload_match
  rerank_no_signal	retrieval surfaced nothing plausible, so the paid ranking call was skipped	instant_recall.rerank_min_score (default 0.1)
  rerank_low_confidence	the reranker returned no match, a confidence under the bar, or an error	instant_recall.rerank_threshold (default 0.7)
  low_margin	legacy gate only (rerank: false) — top hit too close to the runner-up	instant_recall.margin_gap (default 1.0)
  low_outcome	the entry’s real-world resolve-rate decayed below the floor	instant_recall.outcome_floor (default 0.5)
  rerank_over_budget	the investigation’s spend ceiling was already crossed, so the paid rerank call was declined before it was made — recall falls through to the full loop, which the budget ladder then stops with the same reason	investigation.max_tokens_per_investigation / max_cost_per_investigation

• rerank_over_budget is not a recall problem — no instant_recall.* knob fixes it. It means the investigation had already crossed a spend ceiling before recall got to the reranker, so the call was declined rather than paid for. The run then falls through to the full loop, whose first budget check stops it with the same ceiling. If you see this alongside runlore_investigation_budget_trips_total, raise the ceiling; recall is behaving correctly.
• Check which gate is live before tuning anything. The LLM reranker is on by default once instant recall is enabled, and it replaces the BM25-magnitude gate — so on a default install min_score, margin_gap and solo_floor play no part in the fire decision and tuning them changes nothing. They apply only under instant_recall.rerank: false.
• runlore_recall_score (BM25 at the decision point) is still recorded under the reranker, but there it only ranks candidates: a score too low to spend a call on shows up as rerank_no_signal against rerank_min_score (default 0.1), not against min_score. Either way a cold catalog legitimately won’t recall — recall compounds as merged PRs accrete.
• Decision detail is logged at msg="instant recall decision" with score, margin, confidence; under the reranker the fire decision itself is logged at msg="recall reranker decision" with match, entry_id, confidence and the model’s own one-line reason.
• A recalled answer that fails the adversarial verify pass falls through to a full investigation: msg="instant recall rejected by verify; running full investigation".

Findings were investigated but never delivered to chat

Common causes: wrong notify.slack.channel / bot-token scope, an invalid Matrix room_id / access_token, or a generic-webhook endpoint returning non-2xx. At startup, msg="delivery notifiers" with count= confirms how many sinks were wired — count=0 means none are configured.

A Slack mention in a thread does nothing

@runlore note: … fails silently in both directions, so first establish that Slack can reach the endpoint at all: run the /slack/events pre-flight (one unsigned curl; 401 is the healthy answer). Everything below assumes it passes.

(Matrix thread capture has the same symptom and no HTTP endpoint — skip the pre-flight, the log lines here are shared.)

$ TOKEN=$(kubectl -n runlore get secret runlore-secrets \
    -o jsonpath='{.data.SLACK_BOT_TOKEN}' | base64 -d)
$ curl -sS -X POST -H "Authorization: Bearer $TOKEN" https://slack.com/api/auth.test
{"ok":true,…}

/readyz never goes green

/readyz is gated by catalog warmth (internal/app/runtime.go) — deliberately not by leadership, so every warm replica (leader and standby alike) goes Ready and helm upgrade --wait / Flux kstatus succeeds with replicaCount > 1. It returns 503 ("not ready") until the pod has completed its first catalog index/sync.

• Any pod stuck at 503 ⇒ the catalog never warmed: check catalog.dir / catalog.git (clone failing? token wrong?) and the startup logs. runlore_catalog_invalid_entries_total rising ⇒ malformed OKF entries at load.
• The startupProbe allows ~60s of warm-up; a slow first clone can exceed it — raise the chart’s startupProbe.failureThreshold if needed.
• Who leads is a separate question from readiness: read the Lease (kubectl get lease runlore-leader -o jsonpath='{.spec.holderIdentity}' — <podName>_<podIP>) or the runlore_leader gauge.

Quick reference — metric → meaning

The most useful series for triage (full list in Observability):