chaos-demo is returning intermittent 502s
Wrong.
The fault was memory-leak: a route keeps appending to a list until the container hits its memory limit.
Incident report
confidence low · started 14:06:02
Root cause
Upstream timeouts, cause unknown. The app logs no errors, memory is low, the host is normal and the slow traces do not point at the database.
category · unknown
Symptom
Intermittent 502s in short bursts from 14:06, with health probes failing twice in 30 minutes and recovering on their own.
Proposed fix
No change recommended until the cause is known. Check the container's exit reason and watch memory over a longer window.
8tool calls
3m 50sfault to report
74,120tokens in
$0.0093costhaiku-5-5
haiku-5-5model
How it got there
01 get_health 0.4s flapping · 2 failures in 30m · up now, latency 0.09s
Starting with whether the app is reachable.
get_health(app="chaos-demo", since="30m")
probe chaos-demo.xodhiambo.com
14:06:20 down timeout 5.0s
14:06:50 up 0.11s
14:08:05 down timeout 5.0s
14:08:35 up 0.09s
query · probe_success{job="blackbox", instance=~".*chaos-demo.xodhiambo.com.*"}
probe_duration_seconds{job="blackbox", instance=~".*chaos-demo.xodhiambo.com.*"}
02 get_error_rate 0.8s 502 bursts: 14/min at 14:06 → 0 at 14:07 → 9/min at 14:08
It recovers on its own. Checking the shape of the errors over time.
get_error_rate(app="chaos-demo", since="30m")
14:05 502=0
14:06 502=14
14:07 502=0
14:08 502=9
query · sum by (status) (count_over_time({job="nginx", app="chaos-demo"} | json | status >= 500 [1m]))
03 get_app_logs 0.4s 0 lines matched in 15m
Bursts of 502 usually leave a trace in the app's logs.
get_app_logs(app="chaos-demo", since="15m", contains="error")
query · {job="docker", app="chaos-demo"} |= "error"
04 get_container_stats 0.6s 1 process · 1 restart at 14:06:41 · memory 18% · no deploy in window
No error lines. Checking whether the container restarted.
get_container_stats(app="chaos-demo")
web up memory 92 MiB of 512 MiB (18%) CPU 6% restarts 1
last start 14:06:41
query · max by (process_type) (container_memory_working_set_bytes{job="cadvisor", app="chaos-demo"}) / max by (process_type) (container_spec_memory_limit_bytes{job="cadvisor", app="chaos-demo"})
sum by (process_type) (rate(container_cpu_usage_seconds_total{job="cadvisor", app="chaos-demo"}[5m]))
max by (process_type) (container_start_time_seconds{job="cadvisor", app="chaos-demo"})
changes(container_start_time_seconds{job="cadvisor", app="chaos-demo"}[30m])
05 get_slow_traces 1.2s 3 traces over 1s · all GET /items · no database span
One restart, but memory is low, so it does not look like OOM. Looking for slow requests instead.
get_slow_traces(app="chaos-demo", since="15m", min_duration="1s")
5.00s GET /items slowest span GET /items handler 4.61s (92%)
4.98s GET /items slowest span GET /items handler 4.57s (92%)
4.20s GET /items slowest span GET /items handler 3.86s (92%)
query · {resource.service.name="chaos-demo"} | duration > 1s
06 get_app_logs 0.5s 4 distinct messages · 312 lines
Time is spent in the handler, not the database. Reading all recent logs.
get_app_logs(app="chaos-demo", since="15m")
298× GET /items 200
6× GET /healthz 200
4× Started server process
4× Application startup complete
query · {job="docker", app="chaos-demo"}
07 get_host_stats 0.5s disk 61% free · memory 2.9 GiB available · load 0.6 · all normal
Nothing in the app explains it. Checking the host.
get_host_stats(app="chaos-demo")
disk / 61% free ok
memory 2.9 GiB available ok
load (5m) 0.6 on 4 cores ok
query · node_filesystem_avail_bytes{job="node", mountpoint="/"} / node_filesystem_size_bytes{job="node", mountpoint="/"}
node_memory_MemAvailable_bytes{job="node"}
node_load5{job="node"}
08 get_error_rate 0.8s 502: 9/min at 14:08 → 0 at 14:09
Checking whether the errors are still happening.
get_error_rate(app="chaos-demo", since="5m")
14:07 502=0
14:08 502=9
14:09 502=0
query · sum by (status) (count_over_time({job="nginx", app="chaos-demo"} | json | status >= 500 [1m]))
09 submit_report Report recorded.
Out of checks with no clear cause. Reporting it as unknown.
submit_report(cause_category="unknown", confidence="low")
Queries are shown on each step. Logs, metrics and traces come from the server's own Prometheus, Loki and Tempo.