metamorph

Metamorphic (property-based) testing for LLMs — and it shrinks every failure to a minimal prompt.

You can't unit-test a language model by pinning exact outputs — the "right" wording drifts and any answer key rots. Metamorphic testing asserts something sturdier: not what the answer is, but how it must change (or not) when you transform the prompt in a way whose meaning you already know.

• Rephrase a question → the answer must stay the same.
• Shuffle multiple-choice options → the chosen option must stay the same.
• Add an irrelevant sentence → the answer must stay the same.
• Negate the question → the answer must change.

When a system breaks one of these, metamorph shrinks the failure to the smallest prompt that still triggers it — so you get is an apple red?, not the 300-character prompt it first showed up on.

It's property-based, chaos-style testing — the parag-labs deterministic-sim-testing / chaos-mesh-lite DNA — pointed at LLMs, down to the same shrink-to-a-minimal-reproducer move.

$ metamorph study --system negation-blind

=== negation-blind  [FAIL] ===
  ok   paraphrase-invariance      0/10 failing
  ok   option-order-invariance    0/10 failing
  ok   distraction-robustness     0/25 failing
  ok   case-whitespace-invariance 0/25 failing
  FAIL negation-sensitivity       15/15 failing
       minimal failing prompt: 'is an apple red?'

---

Why

The failure modes that actually bite LLMs in production aren't "wrong facts" — they're inconsistencies: the same question answered differently when rephrased, an answer that flips because the options were reordered, a model that sails past the word "not". None of those need an answer key to detect; they need invariance checks. metamorph is a small, honest set of them, plus the machinery to make a violation minimal enough to act on.

How it works

flowchart TB
    C["Prompt (structured: question, options, context)"]:::blue --> T["Transform<br/>paraphrase · reorder · distract · negate"]:::purple
    C --> S1["system(original)"]:::green
    T --> S2["system(transformed)"]:::green
    S1 --> CMP{"change matches<br/>the expectation?"}:::blue
    S2 --> CMP
    CMP -->|yes| OK["relation holds"]:::green
    CMP -->|no| SH["shrink to a<br/>minimal failing prompt"]:::red
    classDef blue fill:#dbeafe,stroke:#3b82f6,color:#1e3a8a;
    classDef green fill:#dcfce7,stroke:#22c55e,color:#14532d;
    classDef purple fill:#ede9fe,stroke:#8b5cf6,color:#4c1d95;
    classDef red fill:#fee2e2,stroke:#ef4444,color:#7f1d1d;

Loading

The prompt is structured data, not a string, which is what lets transforms act on its parts and lets the shrinker remove parts (drop a distractor, drop an option, collapse whitespace) while the failure survives.

One subtlety done right

Reordering multiple-choice options changes the label of the right answer — what was "C" may become "A". So option-order invariance resolves each label back to the chosen option's content before comparing. A system that answers by content passes; one that always picks "the first option" (a real LLM bias) is caught. Comparing raw labels would raise false failures — this is the correct formulation, and it's tested.

Verifying the tester

A test framework you can't trust is worse than none, so metamorph's own correctness is tested: it runs against five deterministic mock systems — a sound reference and four each broken in exactly one way (negation-blind, option-order-biased, distractible, paraphrase-brittle). The suite asserts the reference passes every relation and each flawed system is caught on precisely its relation, with no false alarms on the others.

$ metamorph study
=== reference           [PASS] ===   (all relations ok)
=== negation-blind      [FAIL] ===   negation-sensitivity   -> 'is an apple red?'
=== order-biased        [FAIL] ===   option-order-invariance
=== distractible        [FAIL] ===   distraction-robustness
=== brittle-paraphrase  [FAIL] ===   paraphrase-invariance

Use it

pip install -e ".[dev]"
pytest -q                    # 19 tests, incl. "catches each flaw, no false alarms"

metamorph study              # run every relation against every mock system
metamorph relations          # list the built-in relations

Point it at your own model — the system under test is just str -> str:

from metamorph import run_study, ALL_RELATIONS, base_cases

def my_model(prompt: str) -> str:
    return call_your_llm(prompt)          # your API call

study = run_study("my-model", my_model, ALL_RELATIONS, base_cases())
for r in study.reports:
    if not r.passed:
        print(r.relation, "->", r.minimal_failure.render())

Design decisions

• Assert behaviour, not answers — relations carry a same/different expectation, never an answer key, so they don't rot.
• Structured prompts — so transforms act on parts and failures can be shrunk.
• Shrink every failure — delta-debugging reduces a violation to a minimal prompt.
• Tested against known-broken systems — the framework proves it catches the bug and clears the sound system.

Trade-offs and non-goals (not a benchmark, not a fixed-answer harness, not a raw fuzzer) are argued out in DESIGN.md.

Layout

metamorph/
├── src/metamorph/
│   ├── case.py         # structured prompt + size metric the shrinker minimises
│   ├── relations.py    # the metamorphic relations (paraphrase, order, negation, …)
│   ├── shrink.py       # delta-debugging shrinker -> minimal failing prompt
│   ├── engine.py       # run relations, aggregate, shrink the first failure
│   ├── systems.py      # reference + four known-broken mock systems
│   ├── corpus.py       # base cases to run against
│   └── cli.py          # study / relations
└── tests/              # 19 tests: catches each flaw, no false alarms, shrinking

References

Metamorphic testing (Chen et al. 1998); delta debugging (Zeller & Hildebrand 2002); invariance/behavioural testing of NLP models (CheckList, Ribeiro et al. 2020).

License

MIT — see LICENSE.