a short defence of slow systems
On why I distrust the word real-time, and what twenty milliseconds of patience can buy you.
Twenty milliseconds is a long time, if you spend it well — and the cult of the synchronous has cost us more than it has saved. Payments do not need to be instantaneous. They need to be correct, and known to be correct, which is a slower and far more interesting thing.
The word real-time has done a great deal of work in my industry, almost all of it rhetorical. We say a system is real-time the way we say a person is busy: to end the conversation, not to describe anything. What we usually mean is that a number moves on a screen before the reader has finished doubting it.1
There is a difference between a system that is fast and one that is merely in a hurry. The first has decided what it can safely skip. The second has not; it only hopes. I have kept both kinds running, and I can tell you that hope never appears in the postmortem under its own name. It appears as unexpected load, or as a race we had agreed was benign.2
Consider the retry. A request fails, so the client sends it again, and again — each one perfectly reasonable on its own, and collectively a stampede. The remedy is not to try harder but to wait: a little randomness, a little patience, the interval doubling each time. We call it backoff, which makes it sound like a retreat. It is closer to manners.3
A system in a hurry will hand you a lie quickly. A patient one makes you wait for the truth — and admits it when it isn’t sure.
The twenty milliseconds, then, is not a latency I am apologising for. It is where the system checks itself: confirms the ledger still balances, that the message was written down before it was acted upon, that nobody downstream is about to be surprised. Spent on certainty, it is the cheapest insurance I know.4
None of this is an argument against speed. It is an argument against pretending. Be as fast as the problem honestly allows, and not one millisecond faster. The rest is only a number, moving on a screen, hoping you don’t look too closely.