Move One Thing And Watch What Happens

Three weeks of clean telemetry taught us less than twenty minutes of moving one person to a different server while they kept working.

I've been part of an investigation into an intermittent freezing problem across a multi-site application environment. Staff would be mid-task and the screen would simply lock up for anywhere from a few seconds to half a minute, several times a day on bad days, and then release as if nothing had happened. The infrastructure side had already been through upgrades, resource increases and configuration reviews before I got involved, none of which had touched the underlying problem. So we went looking properly: application servers, the database engine, the network path, the file and print server, all reviewed over windows from seven days out to three months. Every one of those layers came back inside normal thresholds. CPU headroom to spare. Disk latency in single-digit milliseconds. Memory comfortable. No discards, no saturation, nothing that a monitoring dashboard would ever flag red.

That cleanliness is exactly what makes intermittent problems hard. A dashboard tells you whether each measured thing is within its expected range. It does not tell you why two things that look equally healthy behave differently when a real person is using them. We had a strong circumstantial lead — one particular application server was consistently associated with worse freezing than the others, even though it had the most CPU and memory of the lot — but circumstantial is not causal, and staring at more graphs of that server wasn't going to convert one into the other.

So we stopped watching and started intervening. During a live session, with a practice's staff going about their normal work and several workstations open in parallel through remote observation, we moved one user from the suspect server to an alternative one mid-session. Within the same twenty-minute window, a freeze hit multiple workstations still pointed at the original server. The one we'd just moved stayed responsive throughout. It happened again shortly after: two machines froze, the moved one didn't. That is not proof — one afternoon of correlation in a genuinely intermittent system never is — but it told us more in that window than three weeks of infrastructure metrics had, because it isolated the one variable that mattered and watched it in real conditions rather than inferring it from aggregates.

The reason this works, and the reason it's easy to forget it works, is that passive monitoring and live experimentation answer different questions. Monitoring answers "is this component behaving abnormally." Experimentation answers "does changing this one thing change the outcome." Those overlap far less than they feel like they should. A server can be running comfortably within every resource threshold you'd ever alert on and still be doing something structurally different from its neighbours — an extra background sync process, a leftover integration nobody remembered was still pointed there, a scheduled job with a different footprint. None of that shows up as a threshold breach. It shows up as a difference in behaviour under load, which you only see if you deliberately create the comparison and watch it happen.

There's a temptation, especially once you've built good observability, to treat more data as the default next move whenever something is unexplained. It usually isn't. More data helps you rule things out and build confidence about where to look — which is genuinely what those three weeks bought us, even though they didn't find the cause directly. They meant that when we finally did the live comparison, we weren't flailing; we already knew the network was fine, the database host was fine, the disks were fine, so a behavioural difference between two servers running the same software pointed hard at something specific to one of them. The telemetry narrowed the search space. The experiment answered the question inside it. Neither step would have been as useful without the other, but the second one was the one that actually moved the investigation forward, and it cost a lot less than another round of tracing would have.

This has a commercial shape to it too, which is easy to lose sight of when you're several weeks into a technical investigation. Exhaustive monitoring across every layer of a production environment is not free — someone has to extract it, someone has to review it, and someone has to sit with a client who is watching their staff lose time to freezes every day while the investigation continues. A team that keeps asking for one more data pull because it feels more rigorous than a live test is optimising for the appearance of thoroughness over the speed of an actual answer. The live session cost one afternoon, a handful of remote sessions, and the cooperation of a practice that was already frustrated. It bought more clarity than the previous three weeks combined. If you're advising on how to run one of these investigations, that trade-off is worth stating plainly rather than assuming more measurement is automatically the safer, more professional choice.

None of this closes the case. Knowing that a server is associated with the problem is not the same as knowing which specific process on it is responsible, and intermittent problems have a way of punishing anyone who declares victory after one good afternoon. But it changed the shape of the remaining work, from "keep collecting telemetry and hope the pattern eventually explains itself" to "compare the handful of things that differ between this server and its healthier neighbours." That's a much smaller, much more tractable problem, and it only became tractable because at some point we stopped observing the system from the outside and changed one variable inside it while someone was actually using it.

The broader habit worth keeping is this: when something behaves badly only sometimes, and every passive signal you can collect says the system is healthy, the next useful step is rarely a wider net. It's a narrower, live comparison — move the one thing you suspect, keep everything else constant, and watch what actually happens to real work in real time. Dashboards are excellent at telling you nothing is obviously wrong. They are much weaker at telling you what's different. For that, you still have to go and look.