chaos-demo is returning 502s on all routes
Correct.
The injected fault was db-leak: requests open database sessions and never return them.
Incident report
confidence high · started 16:42:10
Root cause
The database connection pool is exhausted. Sessions are checked out and not returned, so every request waits 30 seconds for a connection and nginx gives up.
category · db_connections
Symptom
502 on all routes from 16:42, rising to 61 per minute. Process up, no restarts, no deploy.
Proposed fix
Close the session at the end of each request (dependency with try/finally). Restarting clears it for now but it will return.
5tool calls
2m 41sfault to report
48,210tokens in
$0.006costhaiku-5-5
haiku-5-5model
How it got there
01 get_health 0.4s down since 16:42:10 · 0 of 3 probes passing · timeout 5.0s
Starting with whether the app is reachable at all.
get_health(app="chaos-demo", since="30m")
probe chaos-demo.xodhiambo.com
16:12–16:42 up p50 0.08s
16:42:10–now down every probe timed out at 5.0s
current latency: timeout (5.0s)
query · probe_success{job="blackbox", instance=~".*chaos-demo.xodhiambo.com.*"}
probe_duration_seconds{job="blackbox", instance=~".*chaos-demo.xodhiambo.com.*"}
02 get_container_stats 0.6s 1 process · 0 restarts · memory 41% · no deploy in window
Down could mean crashed. If the container is restarting or out of memory, that changes everything else.
get_container_stats(app="chaos-demo")
web up memory 210 MiB of 512 MiB (41%) CPU 4% restarts 0
last start 1 Nov 09:12 (before the window)
query · max by (process_type) (container_memory_working_set_bytes{job="cadvisor", app="chaos-demo"}) / max by (process_type) (container_spec_memory_limit_bytes{job="cadvisor", app="chaos-demo"})
sum by (process_type) (rate(container_cpu_usage_seconds_total{job="cadvisor", app="chaos-demo"}[5m]))
max by (process_type) (container_start_time_seconds{job="cadvisor", app="chaos-demo"})
changes(container_start_time_seconds{job="cadvisor", app="chaos-demo"}[30m])
03 get_error_rate 0.9s 502: 0/min until 16:41 → 38/min at 16:42 → 61/min at 16:43
The process is up and stable, so requests are hanging rather than failing fast. Checking when errors began.
get_error_rate(app="chaos-demo", since="15m")
onset 16:42:04 · status 502 only
16:40 502=0
16:41 502=0
16:42 502=38
16:43 502=61
query · sum by (status) (count_over_time({job="nginx", app="chaos-demo"} | json | status >= 500 [1m]))
04 get_app_logs 0.4s 2 distinct messages · 53 lines
Errors started abruptly with no deploy. The app's own logs should say what it is waiting on.
get_app_logs(app="chaos-demo", since="15m", contains="error")
41× sqlalchemy.exc.TimeoutError: QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30.00
12× GET /orders 504 upstream timed out after 30000 ms
query · {job="docker", app="chaos-demo"} |= "error"
05 get_slow_traces 1.3s 5 traces over 1s · slowest 30.0s · 99% in db.pool.checkout
The pool looks exhausted. Traces would confirm whether requests wait on a connection or on the query itself.
get_slow_traces(app="chaos-demo", since="15m", min_duration="1s")
30.02s GET /orders slowest span db.pool.checkout 29.91s (99%)
30.01s GET /orders/{id} slowest span db.pool.checkout 29.90s (99%)
30.01s GET /items slowest span db.pool.checkout 29.89s (99%)
30.00s GET /orders slowest span db.pool.checkout 29.88s (99%)
29.98s GET /items slowest span db.pool.checkout 29.86s (99%)
query · {resource.service.name="chaos-demo"} | duration > 1s
06 submit_report Report recorded.
Enough evidence for a high-confidence cause.
submit_report(cause_category="db_connections", confidence="high")
Queries are shown on each step. Logs, metrics and traces come from the server's own Prometheus, Loki and Tempo.