Can You Trust AI-Written Tests?
Have you ever asked an AI agent to add tests, seen them all go green, and wondered if they catch bugs? Mutation testing checks that by planting a deliberate bug, called a mutant, in your code and seeing if a test fails. By the end, you will run it and find tests that prove nothing.
Does a green test run really mean there are no bugs?
No. "All tests pass" only says nothing the tests look at is broken. A test that asserts nothing is green. 100% coverage says every line ran, like every smoke detector plugged in, not that anything was checked. An agent that writes a support bot and its tests together makes tests that agree with the code, bugs included.
Green also depends on the data you feed it, like empty text or another language, and the environment it runs in, like your timezone or clock.
What can you tell an agent to do when green proves so little?
On our support bot, you can send an agent where green cannot look with three prompts, each with a data line and an environment line.
First: "open the app in a browser, no test plan, catch bugs." A QA agent, an AI that tests like a tester, runs an end-to-end test, a robot user clicking the real page. Playwright, a browser-testing tool, has a planner agent that does it, exploring your app and writing a plan.
Second: "plant bugs in my code, tell me which data no test used." That is mutation testing: a test that turns red kills the mutant, one that stays green lets it survive. mutmut plants mutants in Python, and StrykerJS and PIT do it in JavaScript and Java.
Third: a different agent gets only a written plan of what the code must do, as in The Plan is King, and writes tests from it.
Will an agent find the bugs if you only say "catch bugs"?
Back to the first prompt: an agent told "catch bugs" sees what your machine shows and misses the rest. Our bot has two functions: route sends a message to the billing, refund, cancel or human queue if its confidence, how sure it is, reaches 0.70, and trim_reply cuts the answer to one 280-character chat bubble. An agent asked to "add tests" wrote six:
📄Text12bot_guards.py 18 0 6 0 100% 6 passed in 0.43s
We planted two bugs: the plan wants a confidence of 0.80 where the code accepts 0.70, and the bot's web page decides "open now" from the visitor's clock, not Tel Aviv time. We ran prompt A in our script, not Playwright, with trimmed reports:
📄Text1You are a QA agent with a real browser. You are given a URL and NO test plan, NO use cases and NO spec. Your only job: catch bugs.
📄Text1Bugs recorded: (1) the input is not cleared after sending, (2) there is no conversation history, as each reply replaces the previous one and the user's own messages are not shown, (3) empty messages are accepted and routed to the human queue, (4) the 'We are open now.' status appears only after the first send and its basis is unverified
Did it find the clock bug? No, the clock is right in Tel Aviv, where we ran it. Prompt B adds two lines:
📄Text1Also test with different DATA (empty, very long, other languages, special characters, values at the edge of a rule) and in different ENVIRONMENTS (language, timezone, time of day, screen size). For every bug say which data and which environment it needs.
📄Text123Open/closed status ignores the stated Tel Aviv timezone and uses the browser's clock. ... set_env locale en-US, timezone UTC, time 2024-06-15T06:30:00Z (09:30 Tel Aviv, should be OPEN) ... shows 'We are closed now, a human answers tomorrow.' Conversely, at time 2024-06-15T15:30:00Z (18:30 Tel Aviv, should be CLOSED) the page says 'We are open now.' Empty message is accepted. ... Data needed: empty string, works in any environment.
Like a detector tested in one room, each prompt misses what the other sees, so run both. Neither checks the 0.80.
Can mutation testing show which of the agent's tests are empty?
No browser agent checks the 0.80, but mutation testing shows you if the six tests notice a change: each surviving bug is data no test used, like a smoke detector that stays silent.
📄Text12You are a QA agent. Your job is to find out which of my tests are empty. Run mutation testing on bot_guards.py with the tests in tests/ (mutmut does the planting). For every mutant that survived, answer in plain words, no code, one line each: what was changed, and which DATA or ENVIRONMENT no test used, so that nothing turned red. Then say how many of the planted bugs the tests caught, out of how many.
📄Text1234567891. route, mutmut_12: the default for a missing confidence was changed from zero to "nothing", and no test sent a classifier reply with the confidence field left out entirely. 2. route, mutmut_14: the default for a missing confidence was deleted altogether, and again no test sent a reply without a confidence field. 3. route, mutmut_17: a missing confidence now counts as full confidence instead of zero, and no test sent a valid intent with no confidence field. 4. route, mutmut_18: the "too low" check now also rejects a confidence exactly equal to the 0.70 cutoff, and no test used a confidence of exactly 0.70 (the tests only used 0.9 and 0.2). 5. trim_reply, mutmut_3: a reply exactly 280 characters long now gets cut and gets an ellipsis, and no test used a reply of exactly the limit length (the tests used 6 and 500 characters). 6. trim_reply, mutmut_5: after cutting, it strips whitespace from the front of the text instead of the end, and no test used a long reply where the cut lands right after a space. 7. trim_reply, mutmut_7: it now keeps one character fewer than it should before the ellipsis, and no test checked the exact length or exact content of a trimmed reply. Score: the tests caught 23 of the 30 planted bugs, 7 slipped through. ...
📄Text130/30 🎉 23 🫥 0 ⏰ 0 🤔 0 🙁 7 🔇 0 🧙 0
🎉 means killed, 🙁 means survived. A missing confidence field got a made-up default, and no test noticed. Numbers exactly on a limit, 0.70 and 280, went unchecked. A trim cutting the wrong end survived, and so did a reply one character short. A flipped condition was caught: that is what a catch looks like. One more line:
📄Text1Then write one test per survivor.
📄Text1211 passed in 0.11s 30/30 🎉 30 🫥 0 ⏰ 0 🤔 0 🙁 0 🔇 0 🧙 0
Foster et al.'s "Mutation-Guided LLM-based Test Generation at Meta" shows Meta's system doing the same, planting bugs and writing tests for the survivors, with a language model. But one of our five new tests sends 0.70 to "refund", and nobody checked that against the plan.
📄Text1The agent's test for exactly 0.70: a refund with a confidence of exactly 0.7 must be routed to "refund", not sent to a human.
Who decides what correct means, you or the code?
The plan decides. Tests written from the code can only agree with it, so a plan-only agent gets just the plan:
📄Text123456789You are a QA agent. You get a plan and you never see the code. Write pytest tests for the plan: from bot_guards import route, trim_reply. Cover DATA: empty, missing fields, wrong types, other languages, values exactly on, just below and just above every number in the plan. Cover ENVIRONMENT: anything that could change the result from one machine to another (clock, locale, timezone, environment variables) and say in a comment when a rule cannot depend on it. Reply with one Python file only. PLAN (support bot guards, module bot_guards.py) - route(raw): raw is the JSON text the classifier prompt returned. It returns the intent only when the intent is one of billing, refund, cancel, other AND the confidence is at least 0.80. In every other case it returns "human". - trim_reply(text): returns the persona prompt's reply cut to at most 280 characters, ending with "…" when it was cut.
Its 167 checks, sorted by hand:
📄Text1234536 failed, 131 passed 5 the plan's number: 0.79 goes to "billing" instead of "human" 20 valid JSON of the wrong shape: AttributeError: 'NoneType' object has no attribute 'get' and TypeError: unhashable type: 'list' 11 rules the plan never states: missing plan lines, not bugs
Did it find what we planted? It found the 0.80 and the odd data, but its environment checks stayed green, because the clock bug lives in the page. The limits: one browser run is one sample, a plan-only agent needs a good plan, and planting bugs costs time, though 6,000 Google engineers use it, say Petrovic and Ivankovic in "State of Mutation Testing at Google".
Conclusion
Does a green test run on our support bot mean there are no bugs? No. Green and 100% only showed the lines ran on one machine, like a smoke detector never tested in a steamy kitchen. The browser agent found the clock bug, the mutation testing agent went from catching 23 planted bugs to catching 30, and the plan-only agent found the 0.80, which the plan owns.
Sources
- Playwright Test Agents, Playwright docs: the planner agent explores an app and produces a Markdown test plan.
- mutmut README, Boxed: how mutmut plants mutants, mutates only inside functions, and needs fork support (WSL on Windows).
- State of Mutation Testing at Google, Petrovic and Ivankovic, ICSE-SEIP 2018: mutation analysis on code changes for 6,000 engineers.
- Mutation-Guided LLM-based Test Generation at Meta, Foster et al., arXiv 2025: Meta's ACH generates mutants nobody catches and tests that kill them.
Enjoyed this post?
Get the next one in your inbox. No spam, unsubscribe any time.
Comments
0 comments

