01 Overview
Intermittent bugs are the ones that make teams reach for ritual instead of engineering. Someone sees a failure once, writes it up as a mystery, and the report sits in a queue while people argue about whether it is real. The argument is unwinnable because the report never contained the one thing that would settle it: a rate. A rate turns a ghost story into a measurement. It says that under these conditions, this failure happened this many times out of this many attempts.
That single number changes every downstream decision. It tells a reviewer whether the defect is a rounding error or a slow bleed. It tells an engineer whether to hunt for a race condition, a timing dependency, a resource leak, or an environment difference. It tells a product owner whether the fix can wait for the next cycle or needs to interrupt the current one. Without a rate, all of those conversations collapse into opinion, and opinion is the most expensive input a triage queue can receive.
The craft here is not exotic. It is the discipline of converting a vague observation into a bounded claim, then defending that claim with evidence someone else can reproduce. This article walks through what an intermittent report is actually asserting, how severity and priority diverge, how to write steps that survive handoff, how to run the triage queue, and what to do when a defect you thought was fixed comes back.
02 What the report is actually claiming
When someone writes that a feature is flaky, they are usually making three claims at once without separating them. The first is that a failure occurred. The second is that the failure is caused by the product rather than by the test environment, the data, or the observer. The third is that the failure will occur again under conditions the team can control. Only the first claim is directly observed. The other two are inferences, and they need evidence before anyone treats them as fact.
A well-formed intermittent report separates observation from inference explicitly. The observation is a timestamped event: this build, this environment, this account, these actions, this outcome. The inference is the suspected mechanism: a race between two writes, a cache that expires at an inconvenient moment, a retry that duplicates a request, a clock skew between services. Marking the mechanism as suspected rather than confirmed keeps the investigation honest and prevents the team from fixing the wrong thing with great confidence.
The report should also state what was ruled out. If the failure did not reproduce on a clean profile, say so. If it disappeared when a background process was stopped, say so. Negative evidence narrows the search space as much as positive evidence does, and it is routinely discarded because it feels like failure. It is not failure; it is the boundary of the claim. A report that says where the bug is not is far more useful than one that only says where someone saw it.
03 Severity versus priority
Severity describes damage. Priority describes scheduling. Teams conflate them constantly, and intermittent defects are where the confusion does the most harm. A defect that silently drops one in a thousand transactions may look minor because it is rare, but if those transactions are payments or state changes, the damage per occurrence is severe. Conversely, a defect that makes a button flicker on every load is highly visible and low in damage. Visibility is not severity.
Priority is a business and engineering decision about when the work happens relative to everything else. It depends on the rate, the blast radius, the availability of a workaround, and the cost of delay. A low-rate defect on a critical path can outrank a high-rate cosmetic one, and a high-rate defect with a one-line workaround can wait behind a lower-rate defect with no workaround at all. The rate is the input that makes this comparison possible; without it, priority becomes a measure of who complained loudest.
A useful habit is to record severity and priority as separate fields with separate justifications. Severity should cite the damage: data loss, incorrect state, blocked workflow, degraded performance, cosmetic error. Priority should cite the scheduling rationale: release blocker, next sprint, backlog with a review date, or accepted risk. When the two disagree, that disagreement is information. It usually means the team has decided to live with a known amount of damage for a while, and that decision should be written down rather than implied.
04 Writing steps someone else can follow
Reproduction steps for an intermittent defect are not a script for a single run. They are a protocol for many runs, with a way to count outcomes. Start with the exact build identifier and environment, because a rate measured on the wrong build is worse than no rate at all. Then describe the starting state precisely: which account, which data set, which feature flags, which background services were running, and whether any caches were warm or cold. Ambiguity here produces noise that looks like flakiness.
Next, list the actions in order, with timing where timing matters. If two operations must overlap, say so and describe how the overlap was achieved. If a delay was introduced, give the duration and explain why. If the failure only appeared after the tenth repetition, that repetition count belongs in the steps, not in a footnote. The goal is that a different person, on a different machine, can execute the same protocol and produce a comparable count.
Finally, define the observation and the counting method. State what counts as a failure, what counts as a pass, and how many attempts were made. Report the raw numbers rather than a percentage alone, because small samples make percentages misleading. If ten attempts produced one failure, say ten and one. If a thousand attempts produced one failure, say that too. The rate is the headline, but the sample size is the credibility behind it, and both belong in the report.
05 Running the triage queue
A triage queue for intermittent defects needs a different rhythm than a queue for deterministic ones. The first pass should not try to fix anything. It should classify each report by whether it contains a rate, a bounded environment, and a defined observation. Reports missing those elements go back to the reporter with a specific request, not a generic demand for more information. A specific request might ask for twenty attempts on a named build with the outcome of each recorded. That is a task someone can complete.
Reports that do contain a rate move to a second pass, where the team compares rates across reports that may share a root cause. Two low-rate failures in the same subsystem are often one defect with two symptoms, and treating them separately wastes investigation effort. This is also where severity and priority get assigned with written justifications, and where the team decides whether to attempt a fix now, instrument the code to gather more evidence, or accept the risk with a review date.
The queue also needs an exit for reports that cannot be reproduced. Closing as not reproducible is a legitimate outcome, but it should be recorded with the conditions that were tried and the number of attempts made. That record is what allows a future report to be linked to the old one if the failure returns. A closed report with a documented protocol is an asset. A closed report with no detail is a future argument waiting to happen.
06 When a defect comes back
A reopened defect is not a failure of the process; it is the process working. The important question is whether the new occurrence matches the old one or merely resembles it. Compare the build, the environment, the rate, and the observation against the original report. If the rate has changed, that is a clue. If the environment has changed, the old fix may have been correct for conditions that no longer exist. If the symptom is similar but the mechanism is different, the team is looking at a new defect wearing an old name.
Reopening should also trigger a review of the original verification. A fix that was verified with a handful of attempts on a fast machine may not have been verified at all for a defect that appears once in a thousand runs. Verification for intermittent defects needs the same statistical honesty as the original report: a stated number of attempts, a stated environment, and a stated observation. If the original verification was thin, say so plainly and strengthen it rather than blaming the fix.
Finally, resist the urge to treat recurrence as evidence that the team is incompetent. Intermittent defects often have multiple contributing causes, and removing one cause can lower the rate without eliminating it. A falling rate is progress even when it is not zero. The ledger should record that trajectory, because the next person who picks up the defect needs to know whether they are starting from scratch or continuing a hunt that has already narrowed the field.
07 Frequently asked questions
How many attempts are enough to report a rate? There is no universal number, but the report should state the count and let reviewers judge. Ten attempts with one failure is a weak signal; a hundred attempts with ten failures is stronger. What matters is that the sample size is disclosed, because a rate without a denominator cannot be interpreted.
What if the defect only happens in production? Then the report should describe the production conditions as precisely as possible and note which of them cannot be replicated elsewhere. The rate may be estimated from logs or monitoring rather than from manual attempts, and that estimate should be labeled as an estimate. The distinction between measured and inferred rates belongs in the report.
Should every intermittent defect get a rate? Ideally yes, but effort is finite. A reasonable rule is that any defect likely to affect users or data deserves a rate, while purely cosmetic intermittency can be logged and watched. The decision to skip measurement should be explicit, not accidental.
What about defects that disappear on their own? Record them anyway, with the conditions and the attempts made. A defect that vanished may return when the environment changes, and the record is what makes the return recognizable. The ledger exists for exactly this reason: to keep the evidence alive longer than the memory of the person who saw it.