postmortem · final review

The 12 June export delays

One export that could never succeed returned to the head of the queue for 94 minutes. This is the settled incident record: mechanism, impact, chronology, and the changes that prevent one bad job from holding every export behind it.

Detected
09:14, queue-depth alert
Resolved
10:48, poison job quarantined
Blast radius
exports only; imports and sync unaffected

What happened

A workspace uploaded attachment metadata that our validator accepted but the export worker could not serialize. The worker treated every failure as retryable, so the same job returned to the front of the queue every 45 seconds and all later exports waited behind it. 

graph LR
  Q[queue head] --> W[worker]
  W -->|serialize fails| R[retry in 45s]
  R --> Q
  Q -.->|starved| B[1,204 jobs behind]
Why a declared attachment size of −1 fails every attempt

The upload predates the size check added in March. Two other workspaces held the same shape; both were quarantined before their next export. The customer data triggered the path, but our validator and retry policy made it an incident.

Impact

Impact on 12 June, 09:14 to 10:48 UTC.
Exports delayed 1,204
Exports lost 0
Workspaces affected 412
Longest delay 81 minutes

Timeline

The first fix stays in the record because it briefly drained the queue without removing the job that was filling it again.

Queue-depth alert fired Export delay crossed ten minutes. Cause found One export was retrying at the front. Adding workers did not hold The queue drained briefly, then the same job blocked it again. Stuck job set aside The queue resumed its normal rate. Every delayed export delivered

What changed

  1. A job that cannot succeed is now set aside and pages a person instead of retrying.
  2. Every retry rejoins at the back, so a single export cannot hold the queue. Both changes shipped on 13 June.
  3. A new alert fires when one job fails repeatedly, before queue depth becomes the first visible symptom.