Loading...

Loading...

Can You Trust AI-Written Tests?
Loading like and share buttons...
#ai-written-tests#qa-agent#test-coverage#ai-coding

Can You Trust AI-Written Tests?

Vibki AI
Vibki AI
Nov 11•9 min read

Have you ever asked an AI agent to add tests, seen them all go green, and wondered if they catch bugs? Mutation testing checks that by planting a deliberate bug, called a mutant, in your code and seeing if a test fails. By the end, you will run it and find tests that prove nothing.

Does a green test run really mean there are no bugs?

No. "All tests pass" only says nothing the tests look at is broken. A test that asserts nothing is green. 100% coverage says every line ran, like every smoke detector plugged in, not that anything was checked. An agent that writes a support bot and its tests together makes tests that agree with the code, bugs included.

Green also depends on the data you feed it, like empty text or another language, and the environment it runs in, like your timezone or clock.

What can you tell an agent to do when green proves so little?

On our support bot, you can send an agent where green cannot look with three prompts, each with a data line and an environment line.

First: "open the app in a browser, no test plan, catch bugs." A QA agent, an AI that tests like a tester, runs an end-to-end test, a robot user clicking the real page. Playwright, a browser-testing tool, has a planner agent that does it, exploring your app and writing a plan.

Second: "plant bugs in my code, tell me which data no test used." That is mutation testing: a test that turns red kills the mutant, one that stays green lets it survive. mutmut plants mutants in Python, and StrykerJS and PIT do it in JavaScript and Java.

Third: a different agent gets only a written plan of what the code must do, as in The Plan is King, and writes tests from it.

Will an agent find the bugs if you only say "catch bugs"?

Back to the first prompt: an agent told "catch bugs" sees what your machine shows and misses the rest. Our bot has two functions: route sends a message to the billing, refund, cancel or human queue if its confidence, how sure it is, reaches 0.70, and trim_reply cuts the answer to one 280-character chat bubble. An agent asked to "add tests" wrote six:

📄Text
1
2
bot_guards.py      18      0      6      0   100%
6 passed in 0.43s

We planted two bugs: the plan wants a confidence of 0.80 where the code accepts 0.70, and the bot's web page decides "open now" from the visitor's clock, not Tel Aviv time. We ran prompt A in our script, not Playwright, with trimmed reports:

📄Text
1
You are a QA agent with a real browser. You are given a URL and NO test plan, NO use cases and NO spec. Your only job: catch bugs.
📄Text
1
Bugs recorded: (1) the input is not cleared after sending, (2) there is no conversation history, as each reply replaces the previous one and the user's own messages are not shown, (3) empty messages are accepted and routed to the human queue, (4) the 'We are open now.' status appears only after the first send and its basis is unverified

Did it find the clock bug? No, the clock is right in Tel Aviv, where we ran it. Prompt B adds two lines:

📄Text
1
Also test with different DATA (empty, very long, other languages, special characters, values at the edge of a rule) and in different ENVIRONMENTS (language, timezone, time of day, screen size). For every bug say which data and which environment it needs.
📄Text
1
2
3
Open/closed status ignores the stated Tel Aviv timezone and uses the browser's clock. ... set_env locale en-US, timezone UTC, time 2024-06-15T06:30:00Z (09:30 Tel Aviv, should be OPEN) ... shows 'We are closed now, a human answers tomorrow.' Conversely, at time 2024-06-15T15:30:00Z (18:30 Tel Aviv, should be CLOSED) the page says 'We are open now.'

Empty message is accepted. ... Data needed: empty string, works in any environment.

Like a detector tested in one room, each prompt misses what the other sees, so run both. Neither checks the 0.80.

Can mutation testing show which of the agent's tests are empty?

No browser agent checks the 0.80, but mutation testing shows you if the six tests notice a change: each surviving bug is data no test used, like a smoke detector that stays silent.

📄Text
1
2
You are a QA agent. Your job is to find out which of my tests are empty.
Run mutation testing on bot_guards.py with the tests in tests/ (mutmut does the planting). For every mutant that survived, answer in plain words, no code, one line each: what was changed, and which DATA or ENVIRONMENT no test used, so that nothing turned red. Then say how many of the planted bugs the tests caught, out of how many.
📄Text
1
2
3
4
5
6
7
8
9
1. route, mutmut_12: the default for a missing confidence was changed from zero to "nothing", and no test sent a classifier reply with the confidence field left out entirely.
2. route, mutmut_14: the default for a missing confidence was deleted altogether, and again no test sent a reply without a confidence field.
3. route, mutmut_17: a missing confidence now counts as full confidence instead of zero, and no test sent a valid intent with no confidence field.
4. route, mutmut_18: the "too low" check now also rejects a confidence exactly equal to the 0.70 cutoff, and no test used a confidence of exactly 0.70 (the tests only used 0.9 and 0.2).
5. trim_reply, mutmut_3: a reply exactly 280 characters long now gets cut and gets an ellipsis, and no test used a reply of exactly the limit length (the tests used 6 and 500 characters).
6. trim_reply, mutmut_5: after cutting, it strips whitespace from the front of the text instead of the end, and no test used a long reply where the cut lands right after a space.
7. trim_reply, mutmut_7: it now keeps one character fewer than it should before the ellipsis, and no test checked the exact length or exact content of a trimmed reply.
Score: the tests caught 23 of the 30 planted bugs, 7 slipped through.
...
📄Text
1
30/30  🎉 23 🫥 0 ⏰ 0 🤔 0 🙁 7 🔇 0 🧙 0

🎉 means killed, 🙁 means survived. A missing confidence field got a made-up default, and no test noticed. Numbers exactly on a limit, 0.70 and 280, went unchecked. A trim cutting the wrong end survived, and so did a reply one character short. A flipped condition was caught: that is what a catch looks like. One more line:

📄Text
1
Then write one test per survivor.
📄Text
1
2
11 passed in 0.11s
30/30  🎉 30 🫥 0 ⏰ 0 🤔 0 🙁 0 🔇 0 🧙 0

Foster et al.'s "Mutation-Guided LLM-based Test Generation at Meta" shows Meta's system doing the same, planting bugs and writing tests for the survivors, with a language model. But one of our five new tests sends 0.70 to "refund", and nobody checked that against the plan.

📄Text
1
The agent's test for exactly 0.70: a refund with a confidence of exactly 0.7 must be routed to "refund", not sent to a human.

Who decides what correct means, you or the code?

The plan decides. Tests written from the code can only agree with it, so a plan-only agent gets just the plan:

📄Text
1
2
3
4
5
6
7
8
9
You are a QA agent. You get a plan and you never see the code.
Write pytest tests for the plan: from bot_guards import route, trim_reply.
Cover DATA: empty, missing fields, wrong types, other languages, values exactly on, just below and just above every number in the plan.
Cover ENVIRONMENT: anything that could change the result from one machine to another (clock, locale, timezone, environment variables) and say in a comment when a rule cannot depend on it.
Reply with one Python file only.

PLAN (support bot guards, module bot_guards.py)
- route(raw): raw is the JSON text the classifier prompt returned. It returns the intent only when the intent is one of billing, refund, cancel, other AND the confidence is at least 0.80. In every other case it returns "human".
- trim_reply(text): returns the persona prompt's reply cut to at most 280 characters, ending with "…" when it was cut.

Its 167 checks, sorted by hand:

📄Text
1
2
3
4
5
36 failed, 131 passed
 5  the plan's number: 0.79 goes to "billing" instead of "human"
20  valid JSON of the wrong shape: AttributeError: 'NoneType' object has no attribute 'get'
    and TypeError: unhashable type: 'list'
11  rules the plan never states: missing plan lines, not bugs

Did it find what we planted? It found the 0.80 and the odd data, but its environment checks stayed green, because the clock bug lives in the page. The limits: one browser run is one sample, a plan-only agent needs a good plan, and planting bugs costs time, though 6,000 Google engineers use it, say Petrovic and Ivankovic in "State of Mutation Testing at Google".

Conclusion

Does a green test run on our support bot mean there are no bugs? No. Green and 100% only showed the lines ran on one machine, like a smoke detector never tested in a steamy kitchen. The browser agent found the clock bug, the mutation testing agent went from catching 23 planted bugs to catching 30, and the plan-only agent found the 0.80, which the plan owns.

Sources

Enjoyed this post?

Get the next one in your inbox. No spam, unsubscribe any time.

Comments

0 comments

Enter your comment. Maximum 2000 characters.

No comments yet. Be the first to share your thoughts!
Vibki logo wave icon
Vibki Surfing Bot
Vibki

Vibki is a developer blog where you will find tutorials on web development and AI integration. Each article is crafted with clear explanations, practical examples, and ready-to-use source code.

Talk to me
Copy email
LinkedIn
GitHub
Dev.to
Privacy Policy