Technical note

Simulate the burst before you arm the timer

We run a posting scheduler on a 30-minute timer. It wakes up, asks the database what is due, posts it, goes back to sleep. In one week it burst on us 3 separate times. Once it flushed a backlog as 5 posts in a row. Once it treated unscheduled approved posts as due-now and burned 4 overnight. Once it published 4 LinkedIn posts in 2 hours.

Every one of those versions passed its test before deploy. We ran the function once, it did the right thing, we shipped it.

The guard that lied every 30 minutes

The worst incident came down to one timestamp. The scheduler asked "did I already post today?" by reading the approval timestamp on each row. But approval happened days earlier, so the answer was never today, so every tick concluded nothing had run yet and posted again. Approved-at is not posted-at. The burst lived entirely in the gap between those two columns.

A timer job with a bad guard is a sprinkler with a stuck valve. Turn it on once by hand and the lawn gets watered, everything looks right. Leave it on the timer and you wake up to a flooded yard, because the valve made the same wrong decision every cycle all night.

That is the property single-run tests cannot see. The bug is not in any one run. It is in what the runs do to each other.

The rule: run the tick loop, not the function

Before we deploy any scheduler change now, we simulate 3 to 5 consecutive timer ticks against a copy of live state, with state carried forward between ticks, and assert the total sends stay at or under the cap. Not a unit test of the function. A rehearsal of the actual timer.

Two audits ride along with it. First, every "did X happen today" guard gets traced to its column: decision timestamps and execution timestamps look interchangeable and are not. Second, confirm the config actually reaches the scheduled process. One of our rate caps lived in an env file the timer job never loaded, so the job posted at 2 a day during a 1-a-day ramp. The cap existed. The process never met it.

The simulation costs 10 minutes. Each burst cost us a day of cleanup and made the accounts we post to look spammy, which is the one thing a posting system exists not to do.

If it runs on a timer, test it like a timer. Your future 8 a.m. self will be stoked.