payments-api · alerting

Tune payments-api's alert thresholds

On-call asked for at most 10 pages a month, every page a real incident, and a ticket for anything real but not urgent. Each alert below shows what it would have done between 12 September and 11 October at each threshold.

Which threshold and route should each alert use?

Each alert's threshold and route stand beside its replay; the page budget, and whether finance-ops pages count against it, apply to every alert.

Payments on-call pages: 37 of 10 allowed · today 37
False pages: 31 today 31
Finance-ops pages: 0 today 0
Tickets: 4 today 4
Real incidents missed: 2 today 2

`payments-5xx-rate`

Today: over 1%, pages

Share of requests returning a 5xx, over 5 minutes.

Threshold	Fired	Real
over 0.5%	31	3
over 1% today	12	3
over 2%	4	3
over 5%	1	1

Over 5% would have missed the 19 September partial outage and the 26 September card-network incident.

12 pages to payments on-call: 3 real, 9 false · catches all 3 real

`payments-p99-latency`

Today: over 800 ms, pages

p99 latency over 10 minutes.

Threshold	Fired	Real
over 600 ms	44	2
over 800 ms today	19	2
over 1,200 ms	5	2
over 2,000 ms	0	0

19 pages to payments on-call: 2 real, 17 false · catches all 2 real

`card-auth-decline-spike`

Today: over 2×, opens a ticket

Card authorization declines against their 7-day baseline, for 15 minutes.

Threshold	Fired	Real
over 1.5×	9	1
over 2× today	3	1
over 3×	1	1

3 tickets: 1 real, 2 false · catches the real

`webhook-backlog`

Today: over 5,000, pages

Undelivered merchant webhooks in the queue.

Threshold	Fired	Real
over 1,000	22	0
over 5,000 today	6	1
over 20,000	1	1

6 pages to payments on-call: 1 real, 5 false · catches the real

`ledger-reconciliation-lag`

Today: over 90 min, off

Minutes since the last successful ledger reconciliation run.

Threshold	Fired	Real
over 60 min	14	2
over 90 min today	7	2
over 180 min	2	2

Off since August, after a noisy weekend. Both real firings were reconciliation runs that silently stopped; finance found them two days later.

Off: nothing fires · misses 2 real incidents

Finance owns this alert: its pages go to the finance-ops rotation, not payments on-call.

`gateway-cert-expiry`

Today: under 14 days, opens a ticket

Days until the payment gateway’s TLS certificate expires.

Threshold	Fired	Real
under 30 days	1	1
under 14 days today	1	1
under 7 days	1	1

It fires once per renewal whatever the threshold; only the warning time changes.

1 ticket: 1 real, 0 false · catches the real

Apply these alert settings to payments-api's monitoring config and show me the config diff.