How to Test LLM Applications: The Behavioral Testing Framework

A traditional unit test asserts that a function returns the same value for the same input. That assertion breaks on the first line of an LLM-backed service. The model is probabilistic. The same prompt produces different outputs across runs. Your test suite — the one that gave you confidence to ship — is now lying to you.

This is the problem most teams discover the week they try to put their first LLM feature behind a CI pipeline. The tests pass, then fail, then pass, then fail. Engineers start marking them as flaky and skipping them. By the third week, no one knows whether the model is actually working.

Ali Parandeh — chartered mechanical engineer turned AI engineer, head of engineering at a London AI consultancy, and author of Building Generative AI Services with FastAPI (O’Reilly, ~500 pages) — has a framework for this that comes from an academic paper, not from vibes. He calls it behavioral testing, and it splits into three distinct test types that each measure something different about how the model actually behaves.

The framework comes from the CheckList paper

The behavioral testing approach Ali uses is grounded in a NLP research paper called “CheckList” (Ribeiro et al., ACL 2020). The paper proposes testing language models the way you’d test any black-box system — by probing behaviors, not internals. Ali adapted it for generative AI services and walks through it in the book.

There are three tests. Each one answers a different question. You need all three to know whether your model is actually production-ready.

Minimum Functional Test (MFT). Does the model do the basic thing you’re paying it to do? If you built a summarizer, does it summarize? If you built a classifier, does it classify? MFT is the LLM equivalent of a unit test — but because the model is non-deterministic, you have to run it many times against the same input and check the pass rate, not a single boolean.

Invariance Test. Does the output stay stable when the input changes in ways that shouldn’t matter? Change “color” to “colour.” Add a typo. Swap a synonym. The output should not meaningfully change. If it does, the model is brittle to surface variation, which means production users — who don’t write in your test vocabulary — will get inconsistent results.

Directional Expectation Test. When you change the input in a way that should change the output in a specific direction, does it? Ask for a simplified version of a text — the output should be shorter. Add a more negative input to a sentiment classifier — the score should drop. The direction is the assertion, not the exact value.

You can’t pass a single run — you have to test statistically

“Models are probabilistic,” Ali says. “They’re statistical under the hood. Sometimes you need to run your MFT test like 10 times again and again on a model to make sure that every time you’re getting consistent results, to get a statistical confidence that the model is working.”

This changes how you write the test. A traditional test passes once or fails once. An MFT runs N times, you measure the pass rate, and you set a threshold — maybe 9 out of 10 has to pass for the test to count as green. The threshold becomes a parameter you tune based on the cost of a false positive in your domain.

Statistical testing also means your CI gets slower and more expensive. Each LLM call costs money and time. The teams Ali sees getting this right are the ones that build a smaller, faster eval set for every PR and run the full behavioral suite on a schedule — nightly, or before a release.

Invariance is the test most teams skip

Invariance is the boring middle child of the three. It’s also the one that catches the most production bugs. Users do not write the way your engineers write. They use slang, regional spellings, abbreviations, typos. If your model performs well on engineer-written test inputs and poorly on user-written real inputs, your test suite is structurally lying to you.

“If you play around a little bit with the inputs, do you confuse the model or not?” Ali explains. “If you change the vocabulary from American English to British English, the results shouldn’t change. That’s an example of an invariance test.”

The trick is to generate the input variations programmatically. You don’t write 100 versions of each test input by hand. You write a generator that takes a single canonical input and produces variations — typos, regional spelling, paraphrases, capitalization changes. You run all of them. You assert the output is the same. That’s invariance.

Directional tests force you to define what should change

The third test is the one that forces you to be precise about what the model is supposed to do. If you ask for a simpler version of a text, what should change? The length should drop. The reading level should drop. The vocabulary should get more common. If none of those things measurably change, the model didn’t follow instructions, even if it produced fluent output.

“If I ask the model to give me a simpler version, simplify this text, I should expect the output to be shorter in length,” Ali says. “So that’s how the direction would change.”

Directional tests work for any task where the input has a parameter the model is supposed to respond to. Sentiment intensity. Formality. Length. Specificity. You write one test per parameter, change the input along that axis, and assert the output moves the right direction.

You can do test-driven development with this

The interesting consequence: once you have the three tests in place, you can do TDD on LLM applications. Write the test before the prompt. Tune the prompt until the tests pass. Switch the model later and re-run the suite. You’ll know within minutes whether the new model degrades behavior.

That’s the actual goal. Not a green checkmark on a single run, but a stable measurement of behavior that lets you upgrade models, change prompts, and ship features without breaking what works.

FAQ

How do you test an LLM application in production?

Use behavioral testing — three test types adapted from the CheckList paper. Minimum Functional Tests check basic capability. Invariance Tests check robustness to input variation. Directional Expectation Tests check that the model responds correctly to input changes. Because LLMs are probabilistic, each test runs multiple times and you measure pass rate, not a single boolean.

What is a Minimum Functional Test for an LLM?

A Minimum Functional Test (MFT) checks whether the model does the basic task you built it for. A summarizer summarizes, a classifier classifies. MFTs run N times against the same input — typically 10 — and measure the pass rate. The threshold depends on your domain: high-stakes applications might require 95%+ consistency, while lower-stakes ones can accept lower.

What is invariance testing for language models?

Invariance testing checks whether the model gives stable output when the input changes in ways that shouldn’t matter — typos, regional spellings, paraphrases, capitalization. The assertion is that output should not change meaningfully. Failing invariance tests means the model is brittle to surface variation, which will surface as inconsistent user experience in production.

What is directional expectation testing?

Directional expectation testing changes the input in a way that should change the output in a specific direction, then asserts the direction. Ask for a simpler version, the output should be shorter. Add a negative phrase, sentiment should drop. The direction is the assertion. This catches models that produce fluent output without actually following instructions.

Why don’t traditional unit tests work for LLM apps?

Traditional unit tests assert deterministic equality — the same input always produces the same output. LLMs are non-deterministic, so the same prompt produces different outputs across runs. Tests pass intermittently and the suite becomes unreliable. Behavioral testing replaces equality assertions with statistical assertions over many runs.

How many times should I run each LLM test?

Ali’s heuristic: 10 runs minimum for statistical confidence. The right number depends on temperature settings and the cost of a false positive. High-stakes domains (medical, legal, financial) need more runs and tighter thresholds. Lower-stakes domains can use fewer runs. The point is to measure pass rate, not pass/fail.

Can I do test-driven development on LLM applications?

Yes. Behavioral testing makes TDD possible for LLM apps. Write the three test types first — MFT, Invariance, Directional. Tune the prompt until tests pass at your target threshold. When you swap models or update prompts later, the suite tells you whether behavior degraded. This is how you ship model upgrades without breaking what worked.

Where did behavioral testing for LLMs come from?

The CheckList paper (Ribeiro et al., ACL 2020) introduced the three-test framework — Minimum Functional, Invariance, Directional Expectation — for NLP models. Ali Parandeh adapted it for generative AI services in Building Generative AI Services with FastAPI (O’Reilly), mapping each test type to concrete patterns in FastAPI and pytest.

What tools should I use for LLM evaluation?

Langfuse, Logfire, and OpenAI evals are common starting points. Ali also references pytest-style behavioral test suites you can build yourself. The right tool depends on whether you want hosted evaluation (managed pipeline, less control) or self-hosted (more setup, full visibility). For early-stage teams, a pytest harness running behavioral tests on a schedule is often enough.

How does behavioral testing relate to LLM observability?

Observability tells you what your model did in production. Behavioral testing tells you what it should have done. The two work together: observability surfaces unexpected behaviors from real users, and behavioral tests get added to the suite to prevent regressions. Without observability, you don’t know what new tests to write.