Replicate before you report

A bug report nobody has reproduced is only a rumour, whether a person or an agent wrote it. Here's the rule my team adopted, and the bug that proved it.

By Subash Adhikari · · 6 min read

Share on X


A bug report that nobody has reproduced is only a rumour. It doesn't matter whether a person wrote it or an agent did. Until someone has made it happen again, on purpose, it's a story about something that might be wrong.

Rumours are expensive. Someone has to read them, guess at them, and decide whether to act. And a confident rumour is worse than a vague one, because it sends people off to fix the wrong thing.

On 19 November 2026 I made reproduce first the standard for every ticket my team files. The next morning gave us a good example of why.

The rule#

My bots file tickets for my own projects and for work I've been assigned. Writing a clear ticket is easy. Proving the bug is real is the hard part. The rule now covers every ticket and every issue, and it goes like this.

Only live products get tickets. If something is still being built and nobody is using it yet, the finding goes in the pull request or the notes. In my words when I set it: "You don't have to report issues for things that are actively being worked on and under development and no live customers are using it."

Reproduce it safely. That means on a dev or staging environment running the same release as production, and writing down the version you tested. Production is read only. Nobody writes to it, sends real email from it, or touches real users while chasing a bug.

Attach evidence. A screenshot or a screen recording of the repro goes on the ticket. Secrets never appear in it.

Say how sure you are. Every ticket carries one of three tags. It was reproduced on production. It was reproduced on dev at the production release, with the version. Or it's suspected. Suspected is allowed. Sometimes you only have a log line or a piece of code that looks wrong. But it has to be labelled that way and kept apart from confirmed bugs, and never written up as if it were proven.

Every ticket also says who is affected, and has a priority from P1 to P4. A P1 means I get pinged straight away.

And nobody opens a fix pull request unless I ask for one. The ticket's job is to describe the problem well, not to rush a fix for a problem we haven't confirmed.

None of this is new. Good testers have always worked this way. Writing it down matters more with bots, because a bot can write a confident, well formatted ticket very quickly, whether or not the bug is real.

The flaky test that wasn't#

My blog has a nightly test run. At 2am it uses Playwright, a browser testing tool, to run the end to end tests against the dev site. On 20 November it failed.

The first failure was boring. The test runner couldn't find Bun, the JavaScript runtime the blog's tooling uses, because it wasn't on the nightly's path. The ops bot fixed the path and reran the nightly at 6:41am.

That run got further. 59 tests passed and one failed. It was the test that checks the links inside posts. It threw an error saying a path from a GitHub link "cannot be parsed as a URL". The link was in one of my posts, and it pointed at an example in a public GitHub repo.

Then the guessing started. The ops bot's first thought was a bad link in a post, or a bug in the test's link parser. The engineer bot that owns the nightly reran the test three times against dev. It passed all three. The link returned a normal 200. So it called the test flaky, probably a GitHub redirect that slipped past the check. It filed an issue, and that summary reached me. A test that fails once and passes three times in a row really does look flaky.

To its credit, the issue was tagged suspected, not confirmed. It said plainly that the engineer couldn't reproduce it. That's the rule doing its job. But a suspected issue is still a rumour, and this one had a wrong cause attached.

Then the engineer went back to replicate it properly. It looked at the saved trace from the failed run, not just the error. There was no redirect at all. GitHub had answered 200 and set some cookies. The error came straight after.

The difference was the runtime. The nightly runs the tests under Bun, because Node isn't on its path. The engineer's three reruns were in an interactive shell, where they ran under Node. So the reruns weren't replicating the failure. They were running a different setup that happened to share the same test file.

Under Bun, when a response sets cookies, the response URL comes back as a bare path, where Node gives an empty value. Playwright then tries to turn that bare path into a URL, which throws. That happens outside the promise the test was guarding with a catch, so the test fails. The engineer showed it with a small local server that returns 200 with a cookie. Under Node, that passes. Under Bun, it fails with the same error. The real test was less tidy. The tester ran the four affected tests five times each under Bun. Fifteen of the twenty runs passed and five failed with that error, because GitHub doesn't set a cookie on every response. With the fix, all twenty passed.

So it wasn't flaky in the way we meant. It failed whenever the test ran under Bun and GitHub happened to set a cookie, which is exactly the nightly's setup. Our three reruns under Node could never have caught it, however many times we ran them.

How it ended#

Less than a quarter of an hour after the flaky report, the engineer had a test-only fix up and had corrected its own earlier call on the issue. External links are now checked with a plain fetch that never throws and only adds a note to the report. The checks on the blog's own links stayed strict. No tests were skipped and the baseline didn't drop. The tester and the reviewer both passed it, and it merged at 7:09am.

The issue is still open, on purpose. The same Bun behaviour could hit the checks on the blog's own links if the dev site ever starts setting cookies. Running the nightly under Node would avoid the whole class of problem, but that's a runner decision for later, not something to slip into a test fix.

What I took from it#

Three explanations went past in under an hour. A bad link. A flaky GitHub redirect. Then the real one. The first two were reasonable, and both were wrong. What settled it wasn't a better guess. It was making the failure happen again under the same conditions as the original.

That's the part I'd underline. Reproducing isn't rerunning the test until it goes green. It's rerunning it the way it actually failed: the same runtime, the same environment and the same release. If your repro passes and the original failed, you haven't disproved the bug. You've found a difference between the two runs, and that difference is usually the bug.

Before you file the ticket, make it happen again. If you can't, say so, and call it suspected.

Reflect

Your notes stay in this browser. Nothing you write here is sent anywhere.

This browser is not letting the page store anything, so notes will not be kept.

This browser is not letting the page store anything, so notes will not be kept.

Come back to this

Thanks for reading. You can also follow along on X or with RSS.