chaos-demo is returning 500s on /orders/{id}
Correct.
The injected fault was exception: one route raises an unhandled exception.
Incident report
confidence high · started 10:15:20
Root cause
KeyError in /orders/{id}. The handler reads order['shipping_address'], which is missing on some orders, and the unhandled KeyError returns 500.
category · app_exception
Symptom
500 on GET /orders/{id} from 10:15, about 22 per minute. Other routes are fine.
Proposed fix
Handle orders without a shipping address in get_order (use .get and return the order without it), and add a test for that case.
4tool calls
1m 58sfault to report
39,830tokens in
$0.0044costhaiku-5-5
haiku-5-5model
How it got there
01 get_error_rate 0.8s 500: 0/min until 10:15 → 22/min at 10:16 · no 502s
The alert is about errors, so starting with which statuses and when.
get_error_rate(app="chaos-demo", since="15m")
10:14 500=0
10:15 500=9
10:16 500=22
query · sum by (status) (count_over_time({job="nginx", app="chaos-demo"} | json | status >= 500 [1m]))
02 get_app_logs 0.4s 2 distinct messages · 44 lines
500 rather than 502 means the app answered with an error. Its logs should have the exception.
get_app_logs(app="chaos-demo", since="15m", contains="error")
22× KeyError: 'shipping_address' in get_order (app/routes/orders.py:48)
22× GET /orders/{id} 500 Internal Server Error
query · {job="docker", app="chaos-demo"} |= "error"
03 get_container_stats 0.6s 1 process · 0 restarts · memory 29% · no deploy in window
Checking for a deploy or restart that would explain a new code path.
get_container_stats(app="chaos-demo")
web up memory 148 MiB of 512 MiB (29%) CPU 5% restarts 0
last start 29 Oct 18:02 (before the window)
query · max by (process_type) (container_memory_working_set_bytes{job="cadvisor", app="chaos-demo"}) / max by (process_type) (container_spec_memory_limit_bytes{job="cadvisor", app="chaos-demo"})
sum by (process_type) (rate(container_cpu_usage_seconds_total{job="cadvisor", app="chaos-demo"}[5m]))
max by (process_type) (container_start_time_seconds{job="cadvisor", app="chaos-demo"})
changes(container_start_time_seconds{job="cadvisor", app="chaos-demo"}[30m])
04 get_health 0.4s up · 3 of 3 probes passing · latency 0.07s
Confirming the rest of the app is up, so the fault is limited to one route.
get_health(app="chaos-demo", since="15m")
probe chaos-demo.xodhiambo.com
10:01–now up p50 0.07s
query · probe_success{job="blackbox", instance=~".*chaos-demo.xodhiambo.com.*"}
probe_duration_seconds{job="blackbox", instance=~".*chaos-demo.xodhiambo.com.*"}
05 submit_report Report recorded.
The stack trace names the line. High confidence.
submit_report(cause_category="app_exception", confidence="high")
Queries are shown on each step. Logs, metrics and traces come from the server's own Prometheus, Loki and Tempo.