A Fix Has To Reach Where It's Broken

The most dangerous bug is the one that gets fixed correctly and still doesn't work, because everyone involved is now confident about something that isn't true. I spent part of this week chasing exactly that shape of problem in an authentication handshake between a backend service and an external client, and it's worth writing down because the pattern shows up everywhere, not just in OAuth plumbing: a change can be entirely correct in isolation and still be indistinguishable from no change at all, if it never reaches the moment where the actual failure occurs.

The setup was ordinary enough. A service publishes a small, standard discovery document that tells any client how to authenticate against it: here is where you get a token, here is where you register, here is who I am. The document was wrong in a way that looked simple from the outside. Some of the addresses it advertised were missing a prefix that a reverse proxy strips before the request ever reaches the application, so a client that dutifully followed the published address would sail straight past the API and land on the front-end instead, which answered every request with the same helpful, completely wrong web page. First fix: change the code that builds those addresses so it derives them from the configured public address rather than the address the incoming request happened to arrive on. Correct diagnosis, correct fix, tested and verified against a live document. Ship it.

It made no observable difference. The document still had the same problem, in a different corner of it.

The reason took longer to find than the first bug had, and it's the part worth keeping. The handler responsible for shaping that document had a guard clause at the top: if a piece of framework state wasn't populated yet, return early and do nothing. That state gets filled in by the framework later in the request lifecycle than this particular handler runs, so the guard was true on every single request, always, everywhere. The handler never executed its body. Not sometimes, not under load, not in one environment and not another — never. So a second fix, in a second file, changed the address-building logic that determined what the handler should write, and it was completely correct. It just wrote its answer into a room nobody ever opened. Fixing the read produced zero net effect, because the write never happened.

That's the trap. Two defects sat on top of each other, and each one perfectly disguised the existence of the other. Fixing the address logic while the early return is still in place looks like nothing happened, so a reasonable engineer starts doubting their diagnosis of the address problem — when the actual issue is that the diagnosis was right and simply hadn't been given a chance to run. Fixing the early return without also fixing the address logic would have looked almost like success — the document would populate — but with several of its values subtly, silently wrong, in a way that would only surface once a client tried to use them and got redirected into the wrong application entirely. Neither fix on its own was falsifiable from the code. Both required watching what the system actually said to a real caller, not what the diff implied it should say.

This is the part that generalises well beyond authentication documents. In any system with layers — middleware wrapping a handler, a proxy in front of a service, a cache in front of a database, a UI reading from a store that a background job also writes to — a fix that is logically sound can still be operationally inert, because "correct" is a property of the code and "effective" is a property of where that code sits in the actual sequence of events. Code review can confirm the first. Only running the system and inspecting what comes out the other end confirms the second. I've come to treat "the fix is obviously right" and "the fix works" as two separate claims that need two separate kinds of evidence, and the second one only ever comes from outside the code, from a request-response pair or a log line or a document fetched over the wire, never from reading the change a second time more carefully.

There's a second lesson sitting just underneath the first, about silence. The early return that swallowed the handler didn't throw, didn't log, didn't fail a test — it just quietly did nothing, which is the most expensive kind of bug to find because there is no error message pointing at it. A crash is a gift. It tells you exactly where to look. A guard clause that fires every time and produces no visible symptom other than "the thing I expected to happen didn't" leaves you searching the wrong half of the system, because everything downstream of the silent handler looks fine on its own terms — it's just building on an input that was never actually updated. Anywhere a piece of code can decide to do nothing based on state it doesn't control the timing of, it is worth asking what evidence exists that it ever does something. If the answer is "we'd notice if it stopped," treat that as an assumption to test, not a fact.

None of this required exotic tooling to untangle. It required refusing to trust that a correct-looking change had done its job until the actual output — the document a real client would receive — was fetched and read again, address by address, after each change rather than after all of them together. That habit is slower in the moment. It is much faster than the alternative, which is stacking three plausible fixes on top of each other, watching the symptom persist, and then not knowing which of the three to distrust because none of them were wrong.

The commercial version of this lesson is just as real. A team under time pressure will happily accept "we found the bug and fixed it" as a closed ticket, because that's what the diff says happened. The people who actually have to support the system in production care about a different sentence: did the behaviour change. Those two sentences are usually the same thing. The interesting failures are the ones where they quietly stop being the same thing, and nobody notices until a customer does.