LLM Evals: Is Your Prompt Really Getting Better?
Have you ever changed a prompt, liked a few answers, and shipped it without knowing what else broke? LLM evals fix that: an eval is an automated test for your prompt, with fixed inputs, a success criterion, and a score. By the end you will build your first one and know if a change helped.
One tiny prompt tweak, what could possibly break?
Meet Brewly, a coffee shop with an LLM (Large Language Model) support bot made of two prompts. The classify prompt, the router, returns JSON like {"intent": ..., "needs_human": ...}. The persona prompt is Mia, a warm barista: at most 3 sentences, always the exact menu path such as "Settings > Billing", never a promised refund.
The team asks for "shorter and more playful", so you edit both prompts. Harmless, right?
πText12345678910# classify_v1.md (last two lines) {"intent": "cancel_subscription | refund | shipping_status | other", "needs_human": true or false} Set needs_human to true if the customer is angry or mentions a lawyer. # classify_v2.md (last two lines) Read the customer message and reply with JSON: a short snake_case "intent" and a boolean "needs_human". Keep it brief. # persona_v2.md (whole file) You are Mia, Brewly's support barista. Be playful and keep replies short.
Three test chats look fine, and that is the mirror: it shows today's face, not your weight. In production the router answers cancel_plan, a label no code handles, and Mia drops the menu path.
So what are LLM evals, actually?
What would have caught it? LLM evals give you three parts: cases (fixed inputs, here five messages in cases.json), a success criterion (what counts as right) and a score (a number to compare across edits).
π¦JSON1234567{ "id": "cancel", "message": "How do I stop my coffee box?", "intent": "cancel_subscription", "needs_human": false, "golden": "Sure! You can cancel any time in Settings > Billing > Cancel plan, and your next box still ships until the period ends. Anything else I can help with?" }
Anthropic's docs say a good criterion is specific, measurable, achievable and relevant. "Brewly replies should be good" is none of those, while "the router returns one of four labels, the right one, and Mia gives the menu path for every account action" is all four. The router's half, intent.schema.json:
π¦JSON123456789{ "type": "object", "required": ["intent", "needs_human"], "additionalProperties": false, "properties": { "intent": {"enum": ["cancel_subscription", "refund", "shipping_status", "other"]}, "needs_human": {"type": "boolean"} } }
Mia gets a golden answer, a trusted reference reply, and a set of them is a golden dataset (per the Langfuse guide: inputs with reviewed reference outputs). That covers one prompt, and Brewly ships two.
Is one check really enough for a production bot?
No, because each kind of eval answers a different question, like gauges on one scale:
- Code-based: code checks a measurable fact.
- Classification: exact match on the label, then accuracy and recall on
needs_human(angry customers caught). - Prompt review: lint the prompt text, no model call.
- Golden answer: ROUGE-L F1 (word overlap, 0 to 1) similarity to the reference.
- LLM-as-judge: a second model grades against a rubric (a written rule).
Pairwise asks which reply wins, a human just reads them (as in Human-in-the-Loop), regression reruns old cases, and consistency asks the same question twice. Anthropic's docs say to pick the fastest, most reliable, most scalable grader, so you want the cheapest one that sees the failure.
πPython12345678910111213141516# evals/test_support_bot.py, part 1: the router tests import json, os, re, pathlib HERE = pathlib.Path(__file__).parent V = os.environ.get("BOT_VERSION", "v1") # v1 or v2 CASES = json.loads((HERE / "cases.json").read_text()) SCHEMA = json.loads((HERE / "intent.schema.json").read_text()) LOGGED = json.loads((HERE / "logged" / f"{V}.json").read_text()) # logged model outputs, one per case PROMPT = (HERE.parent / "prompts" / f"classify_{V}.md").read_text() def schema_errors(obj, schema): # the few JSON Schema rules we use errs = [f"missing key {k!r}" for k in schema["required"] if k not in obj] errs += [f"extra key {k!r}" for k in obj if k not in schema["properties"]] for k, rule in schema["properties"].items(): ...
πPython12345678910111213141516# part 2: Mia's tests and a tiny runner def test_persona_rules(): # 4. code-based: the persona's hard rules for c in CASES: reply = LOGGED[c["id"]]["reply"] assert len(re.findall(r"[.!?]", reply)) <= 3, f"{c['id']}: more than 3 sentences" if "golden" in c: assert "Settings > Billing" in reply, f"{c['id']}: no menu path" def lcs(a, b): # longest common subsequence length prev = [0] * (len(b) + 1) for x in a: cur = [0] for j, y in enumerate(b): cur.append(prev[j] + 1 if x == y else max(prev[j + 1], cur[j])) ...
πText123456789101112131415# real output: BOT_VERSION=v1 python evals/test_support_bot.py, then v2 --- bot v1 PASS test_output_schema PASS test_intent_accuracy PASS test_classify_prompt_review PASS test_persona_rules PASS test_persona_golden F1 {'cancel': 0.91, 'double': 0.79} SKIP test_persona_judge: needs a judge model --- bot v2 FAIL test_output_schema: ["cancel: 'cancel_plan' is not one of ['cancel_subscription', 'refund', 'shipping_status', 'other']", "double: 'billing_issue' is not one of ['cancel_subscription', 'refund', 'shipping_status', 'other']"] FAIL test_intent_accuracy: accuracy 3/5 FAIL test_classify_prompt_review: prompt never mentions label cancel_subscription FAIL test_persona_rules: cancel: no menu path PASS test_persona_golden F1 {'cancel': 0.77, 'double': 0.83} SKIP test_persona_judge: needs a judge model
On v2 the router fails three ways: cancel_plan and billing_issue break the schema, accuracy is 3/5, and the prompt never mentions cancel_subscription. Mia fails with cancel: no menu path, and guess which test still passes? The golden answer one. That is 96 lines of plumbing.
Do I really have to hand-roll all of this?
Not really, so let's run the same LLM evals on Brewly in frameworks. DeepEval is pytest-style: a Golden is an input plus its expected output, ExactMatchMetric and PatternMatchMetric need no LLM, and GEval is the LLM-judged golden metric, so it needs a judge key:
πPython12345678910111213141516import json import pytest from deepeval import assert_test from deepeval.dataset import EvaluationDataset, Golden from deepeval.metrics import ExactMatchMetric, PatternMatchMetric, GEval from deepeval.test_case import LLMTestCase, LLMTestCaseParams from conftest import CASES # The golden dataset: input + the answer we agreed is right. dataset = EvaluationDataset(goldens=[ Golden(input=c["message"], expected_output=c["intent"], name=c["id"]) for c in CASES]) @pytest.mark.parametrize("golden", dataset.goldens, ids=lambda g: g.name) def test_intent_exact_match(golden, logged): # no-LLM metric: score is 0 or 1 got = json.loads(logged[golden.name]["raw"])["intent"] ...
πText12345# real run on the v2 logged outputs (v1: 6 passed) FAILED evals/test_deepeval.py::test_intent_exact_match[cancel] - AssertionError: Metrics: Exact Match (score: 0.0, threshold: 1.0, strict: False, error: None, reason: The actual and expected outputs are different.) failed. FAILED evals/test_deepeval.py::test_intent_exact_match[double] - AssertionError: Metrics: Exact Match (score: 0.0, threshold: 1.0, strict: False, error: None, reason: The actual and expected outputs are different.) failed. FAILED evals/test_deepeval.py::test_reply_has_menu_path - AssertionError: Metrics: Pattern Match (score: 0.0, threshold: 1.0, strict: False, error: None, reason: The actual output does not match the pattern.) failed. 3 failed, 3 passed, 1 deselected, 3 warnings in 0.38s
promptfoo does it in YAML, grading through its echo provider with no API call:
βοΈYAML12345678910111213141516description: Brewly intent router prompts: ["{{raw}}"] providers: [echo] # echo = grade logged outputs, no API call defaultTest: assert: - type: is-json # schema check, no LLM value: type: object required: [intent, needs_human] properties: intent: {enum: [cancel_subscription, refund, shipping_status, other]} needs_human: {type: boolean} - type: javascript # classification: label equals the expected label value: "JSON.parse(output).intent === context.vars.expected" tests: ...
πText12345678# real run: npx promptfoo@0.120.0 eval -c promptfooconfig.yaml, plus the v2 reasons Successes: 1 Failures: 1 Errors: 0 Pass Rate: 50.00% is-json: JSON does not conform to the provided schema. Errors: data/intent must be equal to one of the allowed values javascript: Custom function returned false
Inspect AI builds a task from a dataset, a solver that answers and a scorer that grades, and Ragas covers LLM apps. Frameworks give you history and CI (continuous integration) gating, but green is only as honest as your cases.
When does a green score lie to you?
Not always, because green in LLM evals only means the checks you wrote found nothing. On v2 the golden-answer test passed at F1 0.77 and 0.83 (threshold 0.7) and rated v2's double reply above v1's (0.83 against 0.79) while the menu path was gone, so similarity misses the one fact that matters. Every kind is blind somewhere: a schema accepts a valid wrong label, exact match fails a correct paraphrase, a judge needs checking against a human. A scale only measures what you put on it.
Repeat each run, because temperature varies answers, and never tune until only your own cases pass.
Conclusion
So would an eval have caught the "shorter and more playful" edit on the Brewly bot? Yes: accuracy 3/5, 'cancel_plan' is not one of [...] and cancel: no menu path all show before it ships, the way a scale beats the mirror. Cases, a criterion and a score, graded per prompt: that is what LLM evals are. Tomorrow, write 5 failure cases for your JSON prompt and your user-facing prompt, add a schema, label, rule and golden check, and run them before your next edit. Next time: prompt caching, and why a cache read costs a fraction of normal input.
Sources
Enjoyed this post?
Get the next one in your inbox. No spam, unsubscribe any time.
Comments
0 comments

