Incident INC-2041 · Severity 2 · Resolved
Checkout slowed for 47 minutes after a pool change
A connection pool limit shipped in release 4.2 starved the payments service under peak load. No orders were lost; 3.1% of checkouts timed out and were retried.
Summary
- Customer impact
- 47 min
- Checkouts retried
- 3.1%
- Orders lost
- 0
- Time to mitigate
- 22 min
Timeline
Release 4.2 reaches 100%
The pool change ships with the rollout; canary traffic had been too light to show it.
Latency alert fires
Checkout p95 passes the 1 s threshold and pages the on-call engineer.
Pool exhaustion identified
The payments dashboard shows every connection in use and a growing wait queue.
Pool limit restored
A config-only rollback returns the pool to 64 connections.
Incident resolved
p95 is back under 400 ms for 20 minutes and the incident is closed.
Evidence
payments-api, 18:02 to 18:26
7 lines ·
- 4 info
- 1 warn
- 2 error
- INFOpool
- WARNpool
- ERRORcheckout
- ERRORcheckout
- INFOoncall
- INFOdeploy
- INFOpool
Load test after the fix
No failures in 5 tests
- Failed
- 0
- Flaky
- 1
- Skipped
- 1
- Passed
- 3
- Duration
- 7.7 min
checkout under peak load1 flaky, 2 passed · 7.7 min
- Flakyretry storm stays under 2%1.6 min
- Passedp95 under 500 ms at 1,200 rps3 min
- Passedno pool wait over 100 ms3 min
pool configuration guard1 skipped, 1 passed · 120 ms
- Passedrejects pool.max below 32 in production120 ms
config/pool.test.ts - Skippedalerts when waiting exceeds 10
POST /v1/payments
The endpoint whose calls timed out. Clients retry a timed-out call with the same idempotency key, so no payment was taken twice.
Parameters
| Name | In | Type | Description |
|---|---|---|---|
Idempotency-Key required | header | string | Makes retries safe; the same key never charges twice. |
Request body
{ "order_id": "ord_8812", "amount": 4200, "currency": "USD" }Responses
201Payment created
504Timed out waiting for a database connection
Follow-ups
To do2 cards
Peak-load stage in canary
Runbook for pool exhaustion
In progress1 card
Alert on pool wait queue
Done2 cards
Restore pool.max to 64
Guard pool.max in production config
Remediation plan
- Week 41
- Week 42
- Week 43
- Week 44
- Pool wait alert
- Canary peak-load stage
- Pool exhaustion runbook
- Quarterly load test review
Week 41
- Pool wait alert
Week 42
- Canary peak-load stage
- Pool exhaustion runbook
Week 44
- Quarterly load test review