detail package

Why checkout got twice as slow on Tuesday

The 4.2 release added the currency to the cart cache key, so no cached cart matched and every checkout read the cart from Postgres. Rolling the key back restored latency; the lasting fix needs your pick below.

Impact

For 3 hours 40 minutes, checkout p99 sat at 410 ms instead of 180 ms; no orders failed, but conversion fell 2.1% against the same hours last week.

The slowdown began at 14:02, the minute 4.2 reached the first region, and ended at 17:44 when the rollback finished. Every region was affected once its deploy landed, in the order the release train visits them.

Error rates stayed flat: Postgres absorbed the extra reads with headroom to spare, peaking at 61% CPU. The cost was time, which shoppers felt as a pause after pressing Pay. The conversion figure compares the affected hours with the same weekday window a week earlier and carries the usual ±0.8% weekly noise.

Measure Before During
Checkout p99 180 ms 410 ms
Cart cache hit rate 94% 3%
Postgres CPU 22% 61%

Cause

The release appended the currency to the cart's cache key, and nothing warmed the new keys, so the cache missed on every cart until it refilled one shopper at a time.

Multi-currency carts needed the currency in the key so a shopper switching currency would not see stale prices. The change was correct for that case. What it missed is that every existing entry became unreachable at once: 1.9 million warm carts turned into misses the moment the new code read them.

A cold cart refills on its first read, so the hit rate would have recovered on its own as shoppers returned, but most carts are read once per session, and the refill was slower than the rate at which new sessions arrived.

The key, before and after 
cart:{user_id}            →  cart:{user_id}:{currency}
      cart:8812004              →  cart:8812004:EUR

Why the checks missed it

Staging starts every test run with an empty cache, so it measures the cold path both before and after the change and could not see a warm cache go cold.

The load test compares p99 against the previous release on the same staging stack. Both runs started cold, so both paid the database read on every cart, and the difference was within noise. Production is the only place the cache is ever warm.

The canary region did show the jump, but its alert compares against a fixed 500 ms threshold, which 410 ms stayed under.

What to do next

Rolling back fixed Tuesday; shipping multi-currency carts again needs either a dual-read migration or a cache warmed before the switch.

How should 4.3 change the cache key?

Both ship multi-currency carts without a cold cache; they differ in risk and effort.

Read both keys for a week Two days of work; old entries keep hitting while new ones fill. Warm the new keys before switching Half a day; one backfill job, but a slow backfill delays the release.

Either way, the canary alert should compare against the previous release rather than a fixed threshold, so a 2× jump fires whatever the absolute number. That change is independent of the choice above and is already in review.

Ruled out: a short cache lifetime during the rollout

Expiring entries after five minutes would bound staleness but keeps the cold path for every returning shopper, which is the cost we are trying to avoid.

Timeline

From the first region's deploy to a finished rollback took 3 hours 42 minutes, 2 hours of it before anyone connected the slowdown to the release.

  1. 14:02 4.2 reaches the first region; p99 starts climbing.
  2. 14:40 Support reports slow checkouts in two tickets.
  3. 16:05 On-call links the slowdown to the deploy times.
  4. 16:30 Cache hit rate found at 3%; key change identified.
  5. 17:44 Rollback complete in every region; p99 back to 180 ms.
The on-call channel, abridged 
16:05 jo: checkout p99 tracks the deploy wave region by region
      16:12 sam: postgres reads 3x, nothing else changed
      16:30 jo: cart hit rate is 3%. 4.2 changed the key.
      16:34 sam: rolling back