Ten Out Of Twelve Is A Failure, Not A Result

An integration job that loads data for ten of twelve expected sites and reports itself as finished has not mostly succeeded. It has silently failed for two sites, and the only reason it doesn't look that way is that the job was written to describe completion in terms of throughput rather than in terms of correctness. Those are very different measures, and the gap between them is where a lot of quiet, expensive incidents live.

I watched this shape appear again recently in an integration alert: a scheduled load across a set of facilities came back with most sites processed and a couple stuck on lock-wait timeouts, and the system's own summary of the run was something close to "10 of 12 roots loaded." Read quickly, that phrasing sounds like a result. Ten twelfths of the work got done. Read carefully, it's a status report from a system that doesn't actually know what "done" means for the thing it's moving. If two of those twelve sites are hospitals, and the missing data is patient movement or medication administration records, then the run didn't mostly work. It worked for ten places and failed for two, and nothing downstream can tell the difference between "this site had no data today" and "this site's data didn't arrive" unless someone goes looking.

That distinction is the entire problem. A percentage is a fine way to describe throughput — how much of a queue got processed, how many files got copied, how much of a batch made it through a pipeline stage. It is a dangerous way to describe an obligation, because an obligation doesn't partially exist. Either every site's clinical data arrived where it needs to be today, or it didn't, and describing that as "83% complete" invites everyone downstream to treat the run as broadly fine rather than as two specific, nameable failures that each need a specific, nameable fix. The moment a completeness requirement gets expressed as a percentage, you've quietly converted a binary outcome into a scale, and scales are much easier to feel comfortable about than zeros.

I think the reason this keeps happening isn't carelessness, it's that the engineering effort naturally goes into the part of the system that's hardest to build, which is moving the data at all. Retry logic, timeout handling, and partial-failure recovery around a batch job take real design work, and once a team has built a job that keeps going instead of falling over completely when one site chokes on a lock, that resilience feels like the win. It is a win, operationally — a job that keeps making progress despite one bad node is much better than one that stops dead. But resilience at the transport layer and completeness at the business layer are two separate concerns, and it's very easy for a summary line written by the person who solved the first one to accidentally answer for the second one too, using language ("10 of 12 loaded") that sounds like a proportion of the same kind of thing when it's actually a headline number hiding two entirely different outcomes underneath it.

The fix isn't more retries, and it isn't a stricter timeout, although both of those help around the edges. The fix is separating the two questions the run is actually answering and refusing to let one satisfy the other. "Did the job run without crashing" is an operational question, and a percentage or a log line is a perfectly good answer to it. "Did every site's data land correctly today" is a completeness question, and the only honest answer to it is a list — the sites that didn't make it, named individually, with the reason, sitting somewhere impossible to miss rather than folded into an average. A dashboard that shows "92% healthy" across a fleet of integration points is quietly asking the reader to do arithmetic they shouldn't have to do: is the missing 8% one site having a bad day, or is it the two sites that matter most this week having a bad month? You cannot tell from the percentage. You can only tell from the list, and if the list doesn't exist as its own artefact, nobody checks for it, because there's nothing to check.

This matters more, not less, the higher the stakes of what's missing. In consumer software, a delayed sync that catches up an hour later is usually a shrug. In healthcare integration, a missing day of records from one facility is the kind of gap that a clinician downstream has no way of detecting on their own, because absence of data looks exactly like absence of activity. Nobody gets paged when a chart is quietly incomplete. Someone just makes a decision later with less information than they believed they had, and by the time anyone notices, the lock-wait timeout that caused it is long gone from the logs. The cost of the failure doesn't land on the team that wrote the integration. It lands on someone who never saw the alert, working from data they had every reason to trust completely.

None of this requires exotic engineering. It requires treating "which specific things didn't happen" as a first-class output of every batch process that moves data anyone relies on, not a detail buried in a log line that only gets read after someone already suspects a problem. Name the two sites. Don't average them into the ten that worked. A run that fails for two out of twelve should look like a run that failed for two out of twelve, loudly and specifically, right up until someone has actually gone and fixed those two, rather than being quietly filed away as an 83% success the moment the job stops running and moves on to tomorrow.