zamu On-call agent
An on-call agent on a real server

Break the demo app.
Watch it find out why.

When an app fails, this agent investigates the way an engineer would: it checks health, metrics, logs and traces one step at a time, then writes up the cause with the evidence. It can read. It cannot change anything.

Break something How it works
3 / 4 correct 6 steps median $0.0066 per run
Reported chaos-demo is returning 502s on all routes
Sample data Replay · run 8f2c
  1. 01
    Starting with whether the app is reachable at all.
    get_health(app="chaos-demo", since="30m")
    down since 16:42:10 · 0 of 3 probes passing · timeout 5.0s
  2. 02
    Down could mean crashed. If the container is restarting or out of memory, that changes everything else.
    get_container_stats(app="chaos-demo")
    1 process · 0 restarts · memory 41% · no deploy in window
  3. 03
    The process is up and stable, so requests are hanging rather than failing fast. Checking when errors began.
    get_error_rate(app="chaos-demo", since="15m")
    502: 0/min until 16:41 → 38/min at 16:42 → 61/min at 16:43
  4. 04
    Errors started abruptly with no deploy. The app's own logs should say what it is waiting on.
    get_app_logs(app="chaos-demo", since="15m", contains="error")
    2 distinct messages · 53 lines
  5. 05
    The pool looks exhausted. Traces would confirm whether requests wait on a connection or on the query itself.
    get_slow_traces(app="chaos-demo", since="15m", min_duration="1s")
    5 traces over 1s · slowest 30.0s · 99% in db.pool.checkout
  6. 06
    Enough evidence for a high-confidence cause.
    submit_report
    The database connection pool is exhausted
6 steps Open full run

Break something

Each fault is injected into chaos-demo, a small app on the same server. You see the symptom; the agent has to find the cause. Live runs are not open yet, so each card opens a past run instead.

Replays only
db-leak Everything returns 502 The app is up but every request times out. Replay run 8f2c
crash-config It keeps restarting Health checks fail, the container loops. No run yet
slow-query It's slow, not down Health passes. Users wait 8 seconds. Replay run 71ab memory-leak Errors come and go Intermittent 502s every few minutes. Replay run c09e exception One page is broken 500s on a single route only. Replay run 5d13

How it works

1 · detectA rule firesPlain queries watch health, error rate, restarts and memory. No model involved yet.
2 · decideThe model picks a toolIt forms a hypothesis and chooses the one check that would confirm or rule it out.
3 · readRead-only tools runFixed queries against Prometheus, Loki and Tempo, scoped to one app. The result goes back to step 2.
4 · reportCause, with evidenceA structured report that cites the steps it relied on, and a confidence level.

Recent runs

Sample data
RunFaultAgent saidVerdictStepsTime
8f2c db-leak The database connection pool is exhausted Correct 5 2m 41s
71ab slow-query Unindexed scan on /reports Correct 6 3m 02s
c09e memory-leak Upstream timeouts, cause unknown Wrong 8 3m 50s
5d13 exception KeyError in /orders/{id} Correct 4 1m 58s