A Test Needs A State It Owns

A flaky test is almost never a flaky assertion. I watched someone this week describe the exact failure mode I see over and over: a suite of integration tests that kept failing intermittently, not because the logic under test was wrong, but because every test run started from whatever data was left lying around from the last one. Nobody had written a setup step. Each test just reused what was already there, and "what's already there" is not a fixed thing. It changes with the order tests run in, with what a previous failed run left half-finished, with what a colleague's branch happened to seed. You cannot reliably test against a moving target, and no amount of cleverer assertions fixes that, because the assertions were never the problem.

The instinct when a test suite gets flaky is to look at what the test checks. People tighten tolerances, add retries, insert waits, or rewrite the assertion to be "more robust." All of that treats the symptom. The actual defect is upstream of the assertion entirely: the test never established a known starting point, so it has no fixed thing to compare against. A test is a claim of the form "given this state, this action produces that result." If you cannot guarantee the "given this state" part, the whole claim is unfalsifiable. It will pass sometimes and fail sometimes for reasons that have nothing to do with whether your code is correct, and a test suite that does that stops being evidence. People learn to re-run the pipeline until it goes green, which is the moment a test suite quietly turns into theatre.

The fix is unglamorous and usually gets skipped because it looks like more work up front: put the system into a well-known state before the test runs, and do it directly rather than through the product. If a test needs a customer with three open orders and one cancelled one, write that data straight into the store the test actually reads from. Don't drive the UI or the API to construct it turn by turn, because now you have two systems that both need to be correct before your test can even start — the thing you meant to test, and the thing you used to set it up. Direct state seeding decouples those. It also happens to be much faster, since walking through screens or endpoints to arrange fixtures is often the slowest part of an integration suite, slower than the actual test.

This is where people push back, reasonably: doesn't writing straight to the database mean you're not testing the real path data takes into the system? Sometimes, yes, and that's fine — that's a different test's job. A test that verifies "orders can be created through the API and end up correctly billed" should absolutely go through the API. A test that verifies "the reconciliation job correctly handles a cancelled order sitting next to open ones" doesn't need to re-prove that orders can be created; it needs a cancelled order and some open ones sitting in front of it, reliably, every time. Conflating "prove the system can reach this state" with "prove the system behaves correctly once in this state" is exactly how test suites end up slow, coupled, and still flaky despite the extra effort, because every test is quietly depending on every other layer being healthy just to get to its own starting line.

There's a cleanup half to this too, and it matters more than it looks like it should. If a test seeds state directly, it should also be able to remove it, or run inside something that resets cleanly — a transaction that rolls back, a database that gets reset between runs, a namespace that's torn down afterwards. Tests that seed state but never clean it up just relocate the original problem one level down: now instead of inheriting unknown leftover state from a previous manual run, you inherit unknown leftover state from a previous automated one. The number of "flaky" suites I've seen that are actually just accumulating years of undeleted fixture data is not small. Ownership of state has to include ownership of its removal, or the guarantee you were trying to buy quietly erodes back to zero over a few months of CI runs.

What I find interesting about this pattern is how often it survives review. Nobody sets out to write a flaky test. The test passes locally, it passes in isolation, it gets merged, and the flakiness only shows up under the specific conditions of shared CI infrastructure running things in an order nobody chose and in parallel with other people's work. That delay between cause and symptom is what makes it hard to catch in code review — the reviewer is looking at the assertion, which is fine, not at the implicit assumption baked into "this test presumes the database already contains X," which is invisible unless you go looking for it. The question worth asking of any new integration test isn't "does this check the right thing" — it's "if I ran this on a completely empty environment, right now, would it still pass." If the honest answer is "only if someone else's test ran first," you don't have a test yet. You have a test that's borrowing someone else's homework and hoping it's still correct by the time you get to copy it.

None of this is really about testing frameworks or tooling. It's about whether you're willing to pay a small, boring cost up front — writing explicit setup — in exchange for a much larger cost never showing up later, which is a team that has learned to distrust its own test suite and started treating red builds as noise. Trust in automated tests is not free, and it does not regenerate on its own. It gets spent every time someone re-runs a failing pipeline out of habit rather than curiosity, and it gets rebuilt, slowly, by tests that always start from a state they actually own.