Skip to main content

Incident INC-2041 · Severity 2 · Resolved

Checkout slowed for 47 minutes after a pool change

A connection pool limit shipped in release 4.2 starved the payments service under peak load. No orders were lost; 3.1% of checkouts timed out and were retried.

Summary

Customer impact
47 min
Checkouts retried
3.1%
Orders lost
0
Time to mitigate
22 min

Timeline

  1. Release 4.2 reaches 100%

    The pool change ships with the rollout; canary traffic had been too light to show it.

  2. Latency alert fires

    Checkout p95 passes the 1 s threshold and pages the on-call engineer.

  3. Pool exhaustion identified

    The payments dashboard shows every connection in use and a growing wait queue.

  4. Pool limit restored

    A config-only rollback returns the pool to 64 connections.

  5. Incident resolved

    p95 is back under 400 ms for 20 minutes and the incident is closed.

Evidence

payments-api, 18:02 to 18:26

7 lines ·

  • 4 info
  • 1 warn
  • 2 error
  1. INFOpoolpool stats: in_use=16 idle=0 waiting=3
  2. WARNpoolconnection wait exceeded 500ms (waiting=41)
  3. ERRORcheckoutPOST /v1/payments timed out after 5000ms
  4. ERRORcheckoutPOST /v1/payments timed out after 5000ms
  5. INFOoncallpool exhaustion confirmed on payments-db primary
  6. INFOdeployconfig rollback applied: pool.max=64
  7. INFOpoolpool stats: in_use=22 idle=42 waiting=0

Load test after the fix

No failures in 5 tests

Failed
0
Flaky
1
Skipped
1
Passed
3
Duration
7.7 min
checkout under peak load1 flaky, 2 passed · 7.7 min
  • Flakyretry storm stays under 2%1.6 min
    Passed on the second run; the first exceeded 2% for 40 seconds.
  • Passedp95 under 500 ms at 1,200 rps3 min
  • Passedno pool wait over 100 ms3 min
pool configuration guard1 skipped, 1 passed · 120 ms
  • Passedrejects pool.max below 32 in production120 ms
    config/pool.test.ts
  • Skippedalerts when waiting exceeds 10
    Alert rule lands with the follow-up below.

POST /v1/payments

The endpoint whose calls timed out. Clients retry a timed-out call with the same idempotency key, so no payment was taken twice.

Parameters

Parameters
NameInTypeDescription
Idempotency-Key requiredheaderstringMakes retries safe; the same key never charges twice.

Request body

Requestjson
{ "order_id": "ord_8812", "amount": 4200, "currency": "USD" }

Responses

  • 201Payment created

  • 504Timed out waiting for a database connection

Follow-ups

  • To do2 cards

    • Peak-load stage in canary

      Releaseplanned

    • Runbook for pool exhaustion

      SREplanned

  • In progress1 card

    • Alert on pool wait queue

      SREactive

  • Done2 cards

    • Restore pool.max to 64

      Paymentsdone

    • Guard pool.max in production config

      Platformdone

Remediation plan

  1. Week 41
  2. Week 42
  3. Week 43
  4. Week 44
  1. Pool wait alertWeek 41In progress
  2. Canary peak-load stageWeek 42 – Week 43Planned
  3. Pool exhaustion runbookWeek 42Planned
  4. Quarterly load test reviewWeek 44Planned
  1. Week 41

    • Pool wait alertIn progress
  2. Week 42

    • Canary peak-load stageuntil Week 43Planned
    • Pool exhaustion runbookPlanned
  3. Week 44

    • Quarterly load test reviewPlanned