payments on-call · alert review · week 38

Which alerts should still page?

Payments paged 158 times last week, and 17 of those pages needed a person. Pick what each noisy alert should do; I'll change every rule in one pull request.

5 caught incidents 7 mostly noise 2 wrong owner

Alerts by pages

Evidence and decision

Canary health check

Rule
payments_canary_probe_failed
Fires on
One failed probe against the canary pod
Pages
47, none of which needed a person

Forty-four of the 47 pages came within three minutes of a deploy, while the restarting canary missed one probe; the 10:00, 14:00 and 18:00 trains alone account for 39. It was never down long enough to act on.

How should the canary probe page?

last week: 0 pages recommended Require three failed probes in a row Pages when the canary is down for 90 seconds, which no deploy restart reached. last week: 3 pages Mute for five minutes after each deploy Keeps single-probe sensitivity between trains, but depends on deploy events arriving on time. last week: 47 pages Keep paging on one failed probe No change to the rule.

Checkout p99 latency

Rule
checkout_latency_p99_high
Fires on
p99 above 800 ms for two minutes
Pages
31, one of which needed a person

Most pages came from the lunchtime traffic peak and recovered on their own in a median of four minutes. The one real page was Thursday's database failover, which the 5xx alert also caught a minute earlier.

What should the latency alert page on?

Checkout's latency objective is 1.2 seconds at p99 over ten minutes.

last week: 1 page recommended Page on the objective itself 1.2 seconds sustained for ten minutes; peaks that recover stop paging. last week: 12 pages Keep 800 ms, page only in working hours Nights and weekends open a ticket instead. last week: 31 pages Keep the current threshold No change to the rule.

Webhook queue depth

Rule
webhook_queue_depth_high
Fires on
More than 5,000 queued deliveries
Pages
22, two of which needed a person

Twenty pages were one partner's bulk retries, which drained within ten minutes each time. The two that mattered were Wednesday's stuck consumer: the queue was short, but its oldest delivery was 40 minutes old.

What should the webhook alert measure?

Depth tracks partner traffic; age tracks whether we are keeping up.

last week: 2 pages recommended Page on the oldest delivery's age Ten minutes old trips it; a burst that drains does not. last week: 6 pages Raise the depth to 20,000 Fewer burst pages, and still blind to a stuck consumer on a short queue. last week: 22 pages Keep paging at 5,000 No change to the rule.

Ledger export lag

Rule
ledger_export_lag_high
Fires on
Export more than 30 minutes behind
Pages
18, none of which needed a person

Every page fell between 01:00 and 03:00 UTC, while the nightly settlement batch holds the ledger lock. The lag cleared by 03:20 each night without anyone touching it. 

hour, pages
22:00, 0
23:00, 0
00:00, 0
01:00, 7
02:00, 9
03:00, 2
04:00, 0
05:00, 0
06:00, 0

What should happen during the nightly batch?

Finance reads the export at 07:00 UTC.

recommended Mute from 01:00 to 03:30 UTC Lag outside the batch still pages. Open a ticket instead of paging Someone checks it before Finance's 07:00 read. Keep paging No change to the rule.

Replica disk usage

Rule
ledger_replica_disk_80
Fires on
Ledger read replica above 80% full
Pages
9, none of which needed a person

Autovacuum reclaimed the space within the hour every time. Growth is already tracked in the monthly capacity review, and the 95% critical alert on the same disk stays either way.

Should the 80% disk alert page at all?

The 95% alert has not fired this quarter.

recommended Delete it The capacity review and the 95% alert cover the disk. Raise it to 90% Leaves an hour's warning before the critical alert. Keep paging at 80% No change to the rule.

Sandbox TLS expiry

Rule
tls_cert_expiring_14d
Fires on
A certificate within 14 days of expiry
Pages
7, one a day for the same certificate

The certificate belongs to the partner sandbox host, which Platform owns and renews automatically. Their renewal job is failing, and PLAT-2291 already tracks the fix.

Who should hear about the sandbox certificate?

It expires in six days.

recommended Route it to Platform's queue The team that can renew it sees it; payments stops paging. Page both teams until it renews Payments keeps waking up for a fix it cannot make.

Settlement job failed

Rule
settlement_job_failed
Fires on
Any failed settlement attempt
Pages
6, three of which needed a person

The three pages nobody needed were first attempts that the scheduler retried successfully five minutes later. The other three were Friday's expired bank credential, which failed every retry.

Should a settlement failure wait for its retry?

The scheduler retries once, five minutes after a failure.

recommended Page when the retry fails too Real failures arrive five minutes later. Keep paging on the first failure No change to the rule.

Ledger consumer lag

Rule
ledger_events_consumer_lag
Fires on
More than 50,000 unread events on the ledger topic
Pages
5, none of which needed a person

Each page was a broker rebalance on the shared event cluster, which Data Platform runs. The consumer caught up once partitions were reassigned; nothing on the payments side changed.

Who should hear about ledger consumer lag?

Lag that outlasts a rebalance does delay the ledger export.

recommended Page Data Platform during rebalances Payments hears only when lag outlasts 15 minutes. Keep paging payments No change to the rule.

Payments API 5xx rate

Rule
payments_api_5xx_rate
Pages
4, all of which needed a person

Caught Tuesday's card-network outage six minutes before the first customer ticket, and paged for Thursday's failover a minute ahead of the latency alert. It keeps paging unchanged.

3-D Secure failure rate

Rule
threeds_challenge_failure_rate
Fires on
More than 8% of challenges failing over five minutes
Pages
3, one of which needed a person

Two pages came from one issuing bank's challenge outage, which only that bank could fix. The third was our own expired signing key, which failed every issuer at once.

Should one issuer's outage page us?

Per-issuer failure rates are already on the payments dashboard.

recommended Page when three or more issuers fail A single bank's outage opens a ticket instead. Keep paging on the overall rate No change to the rule.

Fraud score drift

Rule
fraud_score_distribution_drift
Pages
2, both of which needed a person

Both pages were Friday's stale feature pipeline, which left the model scoring on day-old velocity counts. It keeps paging unchanged.

Payout file rejected

Rule
payout_file_rejected
Pages
2, both of which needed a person

The bank rejected Monday's payout file over a changed date format, and the page arrived in time to resubmit before the cutoff. It keeps paging unchanged.

Refund processor timeouts

Rule
refund_processor_timeout_rate
Pages
1, which needed a person

Saturday's page was the processor's own degradation, confirmed on their status page twelve minutes later. It keeps paging unchanged.

Dispute deadline near

Rule
dispute_evidence_due_24h
Pages
1, which needed a person

A chargeback's evidence was due within a day and had no owner; the page got it filed with four hours to spare. It keeps paging unchanged.