Which alerts should still page?

00 of 00 to answerNothing to answer 7 mostly noise 2 wrong owner 5 caught incidents

How should the canary probe page?

Rule: payments_canary_probe_failed
Fires on: One failed probe against the canary pod
Pages: 47, none of which needed a person

Forty-four of the 47 pages came within three minutes of a deploy, while the restarting canary missed one probe; the 10:00, 14:00 and 18:00 trains alone account for 39. It was never down long enough to act on.

{
  ariaLabel: "Canary pages by hour, UTC: 14 at 10:00, 13 at 14:00 and 12 at 18:00, the three deploy trains; 2 at 11:00 and 3 at 16:00, after other deploys; one each at 03:00, 08:00 and 20:00",
  marginBottom: 40,
  x: {
    domain: Array.from({length: 24}, (_, h) => `${String(h).padStart(2, "0")}:00`),
    ticks: ["00:00", "06:00", "10:00", "14:00", "18:00", "22:00"],
    label: "hour (UTC)",
  },
  y: {grid: true, label: "pages"},
  marks: [
    Plot.barY(
      [
        {hour: "03:00", pages: 1, deploy: false},
        {hour: "08:00", pages: 1, deploy: false},
        {hour: "10:00", pages: 14, deploy: true},
        {hour: "11:00", pages: 2, deploy: true},
        {hour: "14:00", pages: 13, deploy: true},
        {hour: "16:00", pages: 3, deploy: true},
        {hour: "18:00", pages: 12, deploy: true},
        {hour: "20:00", pages: 1, deploy: false},
      ],
      {x: "hour", y: "pages", fill: (d) => (d.deploy ? "var(--series-1)" : "var(--series-3)")},
    ),
    Plot.ruleY([0]),
  ],
}

last week: 0 pages recommended Require three failed probes in a row Pages when the canary is down for 90 seconds, which no deploy restart reached.

last week: 3 pages Mute for five minutes after each deploy Keeps single-probe sensitivity between trains, but depends on deploy events arriving on time.

last week: 47 pages Keep paging on one failed probe No change to the rule.

What should the latency alert page on?

Checkout's latency objective is 1.2 seconds at p99 over ten minutes.

Rule: checkout_latency_p99_high
Fires on: p99 above 800 ms for two minutes
Pages: 31, one of which needed a person

Most pages came from the lunchtime traffic peak and recovered on their own in a median of four minutes. The one real page was Thursday's database failover, which the 5xx alert also caught a minute earlier.

last week: 1 page recommended Page on the objective itself 1.2 seconds sustained for ten minutes; peaks that recover stop paging.

last week: 12 pages Keep 800 ms, page only in working hours Nights and weekends open a ticket instead.

last week: 31 pages Keep the current threshold No change to the rule.

What should the webhook alert measure?

Depth tracks partner traffic; age tracks whether we are keeping up.

Rule: webhook_queue_depth_high
Fires on: More than 5,000 queued deliveries
Pages: 22, two of which needed a person

Twenty pages were one partner's bulk retries, which drained within ten minutes each time. The two that mattered were Wednesday's stuck consumer: the queue was short, but its oldest delivery was 40 minutes old.

last week: 2 pages recommended Page on the oldest delivery's age Ten minutes old trips it; a burst that drains does not.

last week: 6 pages Raise the depth to 20,000 Fewer burst pages, and still blind to a stuck consumer on a short queue.

last week: 22 pages Keep paging at 5,000 No change to the rule.

What should happen during the nightly batch?

Finance reads the export at 07:00 UTC.

Rule: ledger_export_lag_high
Fires on: Export more than 30 minutes behind
Pages: 18, none of which needed a person

Every page fell between 01:00 and 03:00 UTC, while the nightly settlement batch holds the ledger lock. The lag cleared by 03:20 each night without anyone touching it.

{
  ariaLabel: "Export-lag pages by hour, UTC: none from 22:00 to 00:00, 7 at 01:00, 9 at 02:00, 2 at 03:00, and none from 04:00 to 06:00",
  marginBottom: 40,
  x: {
    domain: ["22:00", "23:00", "00:00", "01:00", "02:00", "03:00", "04:00", "05:00", "06:00"],
    ticks: ["22:00", "00:00", "02:00", "04:00", "06:00"],
    label: "hour (UTC)",
  },
  y: {grid: true, label: "pages"},
  marks: [
    Plot.barY(
      [
        {hour: "22:00", pages: 0},
        {hour: "23:00", pages: 0},
        {hour: "00:00", pages: 0},
        {hour: "01:00", pages: 7},
        {hour: "02:00", pages: 9},
        {hour: "03:00", pages: 2},
        {hour: "04:00", pages: 0},
        {hour: "05:00", pages: 0},
        {hour: "06:00", pages: 0},
      ],
      {x: "hour", y: "pages", fill: "var(--series-1)"},
    ),
    Plot.ruleY([0]),
  ],
}

recommended Mute from 01:00 to 03:30 UTC Lag outside the batch still pages.

Open a ticket instead of paging Someone checks it before Finance's 07:00 read.

Keep paging No change to the rule.

Should the 80% disk alert page at all?

The 95% alert has not fired this quarter.

Rule: ledger_replica_disk_80
Fires on: Ledger read replica above 80% full
Pages: 9, none of which needed a person

Autovacuum reclaimed the space within the hour every time. Growth is already tracked in the monthly capacity review, and the 95% critical alert on the same disk stays either way.

recommended Delete it The capacity review and the 95% alert cover the disk.

Raise it to 90% Leaves an hour's warning before the critical alert.

Keep paging at 80% No change to the rule.

Should a settlement failure wait for its retry?

The scheduler retries once, five minutes after a failure.

Rule: settlement_job_failed
Fires on: Any failed settlement attempt
Pages: 6, three of which needed a person

The three pages nobody needed were first attempts that the scheduler retried successfully five minutes later. The other three were Friday's expired bank credential, which failed every retry.

recommended Page when the retry fails too Real failures arrive five minutes later.

Keep paging on the first failure No change to the rule.

Should one issuer's outage page us?

Per-issuer failure rates are already on the payments dashboard.

Rule: threeds_challenge_failure_rate
Fires on: More than 8% of challenges failing over five minutes
Pages: 3, one of which needed a person

Two pages came from one issuing bank's challenge outage, which only that bank could fix. The third was our own expired signing key, which failed every issuer at once.

{
  ariaLabel: "Challenge failure rate by issuer during Wednesday's first page: Northbank 34%, Atlas 3%, Crestline 2%, Meridian 2%, Pioneer 1%; the alert pages above 8% overall",
  marginLeft: 80,
  marginTop: 24,
  x: {grid: true, label: "challenges failing (%)", domain: [0, 40]},
  y: {label: null},
  marks: [
    Plot.barX(
      [
        {issuer: "Northbank", rate: 34},
        {issuer: "Atlas", rate: 3},
        {issuer: "Crestline", rate: 2},
        {issuer: "Meridian", rate: 2},
        {issuer: "Pioneer", rate: 1},
      ],
      {y: "issuer", x: "rate", fill: "var(--series-1)", sort: {y: "-x"}},
    ),
    Plot.ruleX([8], {stroke: "var(--series-2)"}),
    Plot.text([8], {x: (d) => d, frameAnchor: "top", dy: -14, dx: 4, textAnchor: "start", text: () => "pages above 8% overall"}),
    Plot.ruleX([0]),
  ],
}

recommended Page when three or more issuers fail A single bank's outage opens a ticket instead.

Keep paging on the overall rate No change to the rule.

Who should hear about the sandbox certificate?

It expires in six days.

Rule: tls_cert_expiring_14d
Fires on: A certificate within 14 days of expiry
Pages: 7, one a day for the same certificate

The certificate belongs to the partner sandbox host, which Platform owns and renews automatically. Their renewal job is failing, and PLAT-2291 already tracks the fix.

recommended Route it to Platform's queue The team that can renew it sees it; payments stops paging.

Page both teams until it renews Payments keeps waking up for a fix it cannot make.

Who should hear about ledger consumer lag?

Lag that outlasts a rebalance does delay the ledger export.

Rule: ledger_events_consumer_lag
Fires on: More than 50,000 unread events on the ledger topic
Pages: 5, none of which needed a person

Each page was a broker rebalance on the shared event cluster, which Data Platform runs. The consumer caught up once partitions were reassigned; nothing on the payments side changed.

recommended Page Data Platform during rebalances Payments hears only when lag outlasts 15 minutes.

Keep paging payments No change to the rule.

Rule: payments_api_5xx_rate
Pages: 4, all of which needed a person

Caught Tuesday's card-network outage six minutes before the first customer ticket, and paged for Thursday's failover a minute ahead of the latency alert. It keeps paging unchanged.

Rule: fraud_score_distribution_drift
Pages: 2, both of which needed a person

Both pages were Friday's stale feature pipeline, which left the model scoring on day-old velocity counts. It keeps paging unchanged.

Rule: payout_file_rejected
Pages: 2, both of which needed a person

The bank rejected Monday's payout file over a changed date format, and the page arrived in time to resubmit before the cutoff. It keeps paging unchanged.

Rule: refund_processor_timeout_rate
Pages: 1, which needed a person

Saturday's page was the processor's own degradation, confirmed on their status page twelve minutes later. It keeps paging unchanged.

Rule: dispute_evidence_due_24h
Pages: 1, which needed a person

A chargeback's evidence was due within a day and had no owner; the page got it filed with four hours to spare. It keeps paging unchanged.