Latency alert fired
Checkout p99 crossed 2s in us-east. Error rate stayed flat.
01 / impact trace
The 43-minute window
Checkout API p99, one-minute rollup. The rollback cleared queued connections without dropping writes.
8.4s peak p99
degradedbaseline
320ms → 8.4s after pool capacity was halved0 failed writes · 0 data loss
02 / event log
Timeline
Mara took incident command
Declared SEV-2. Priya joined as first responder; agent opened this record.
Cache churn looked guilty
Miss rate had moved, but pinning cache nodes did nothing. Wrong lead closed.
Turning point: pool saturation found
Connection wait time matched p99. All 30 checkout connections were pinned.
Config delta confirmed
Morning push changed DB_POOL_MAX from 60 to 30 on every checkout task.
Turning point: rollback started
Inez reverted the pool parameter. New tasks entered rotation at the prior size.
Queues draining
Pool saturation dropped below 70%; p99 fell through 2s and kept falling.
Service recovered
p99 held at 320ms for five minutes. No lost orders or duplicate charges.
All clear
Status moved to resolved. Support received the final customer-impact note.
04 / living follow-ups
Close the loop
05 / postmortem notes
What we learned
Editable in place. The agent keeps these notes with the incident, even after the channel and call are gone.
Live documentLesson 01 We reviewed the config syntax, not the operational effect. A valid value can still be an unsafe change.
Lesson 02 Our first ten minutes went to cache churn because that graph was familiar. Connection wait time needs to sit beside checkout latency, not three dashboards away.