Alert on the error class, not the error
Our client contact forms sit behind Cloudflare Turnstile, the human-verification step that keeps bots out of the inbox. For 2 weeks, one pipeline rejected every single submission, including the real humans. Nobody got paged. The forms looked fine, the site looked fine, and the inquiries just stopped arriving.
The failure was infrastructure: a verification secret had gone invalid, so the service rejected everything it was asked to verify. The reason nobody noticed is the interesting part. Verification failures happen all day, every day, because bots hammer public forms and expired tokens time out. Our monitoring saw this failure the same way it saw all of those: one more rejection in a stream of rejections that are supposed to happen.
Burnt toast and house fires ring the same bell
An alert that fires on every failed verification is a smoke detector that goes off for toast. Within a week someone pulls the battery, and now it misses the kitchen fire too. Silencing the whole stream was the reasonable move right up until it cost us 2 weeks of leads.
The fix was not more alerting. It was classifying. Every error the verification service returns falls into one of two classes, and the service's own error codes tell you which:
User-class failures. Expired tokens, malformed tokens, bot traffic getting caught. These mean the system is working. They should stay silent forever, at any volume.
Infrastructure-class failures. Invalid secret, misconfigured widget, the verification service unreachable. These mean no human can get through, and even one of them is an emergency. This class pages a person immediately.
The rebuilt alerting reads the specific error code on every failure and routes by class. Bot noise stays noise. The first invalid-secret response wakes someone up, because that error can never be the user's fault.
The rule travels
Any third-party verification or gating service works this way: payment processors, login providers, spam filters, the lot. Each one returns failures constantly as part of normal operation, and each one has a small set of failure codes that mean the gate itself is broken. Read the provider's error-code list once, sort every code into "their fault" or "our fault," and alert only on ours.
The question to ask of any alert you are about to build: if this fires 400 times a day when everything is healthy, what does firing tell me? If the answer is nothing, the alert is not protecting you. Classification is what turns a noise stream back into a signal.
Two weeks of silence taught us that one. Cheap lesson at twice the price.
Related: the contact form backend we build for every client site, and where form submissions should go.