Decide how to write e2e tests

What problem are you facing?

Our e2e test is a shell script, e2e/run.sh. It brings up a control plane and a workload cluster on kind, then checks that one model serves end to end through the InferenceGateway.

Open PRs extend it in different ways:

Assert what the served endpoints return, not just a 200 #420 moves the serving checks into a Python unittest suite that runs in a pod.
Exercise each clouds serving stack placement in the local e2e #472 adds a Chainsaw test for per-cloud serving stack placement.

#367 and #368 track more tests we want. I'd like to decide how we write e2e tests before they grow further, so new tests follow one pattern.

That starts with what e2e tests are for, as opposed to the composition functions' unit tests. Candidates include:

Integration. The pieces work together on a real cluster: RBAC, provider wiring, cross-cluster access, readiness.
Lifecycle. Resources behave correctly when updated or deleted, or when something fails.
Serving. The gateways authenticate, route, translate and meter requests.
Placement. Pods schedule onto the right nodes, with the right tolerations.
Clouds. Modelplane provisions real cloud clusters, in runs a maintainer triggers rather than on every e2e run.

How could Modelplane help solve your problem?

Whatever we choose has to run two kind clusters and reach gateway addresses on the kind Docker network. It also has to give the same result locally as in CI.

Options include:

Keep extending run.sh.
Keep bring-up in shell, and write the checks with Python's unittest, as Assert what the served endpoints return, not just a 200 #420 does.
Write bring-up and tests in Python with pytest, in the style of Crossplane's e2e tests.
Use Chainsaw, as Exercise each clouds serving stack placement in the local e2e #472 does.

I ran a bake-off between the four options, to see how each one handles the tests we want.

A coding agent built each option on its own branch, from one brief. Every option had to cover the same checks:

Serving. Authentication, model rewriting, Anthropic translation and the usage record, as run.sh --verify checks them today.
Lifecycle. Deploy a second model and check it serves. Delete its ModelService and check routing stops, then foreground-delete its ModelDeployment and check its replicas, endpoints and engine pods go.
Placement. Engine pods run on the GPU pool with a DRA device allocated, and no pod tolerates every taint.
Clouds. An EKS InferenceCluster test that runs only when a maintainer asks for it. Nobody ran it.

The pytest option also ported bring-up to Python. The others kept it in run.sh.

I ran each prototype three times against one shared kind environment: healthy, then with two faults injected, then healthy again. The faults changed the caller's API key and added a pod that tolerates every taint. Three more agents then scored each prototype from 1 to 5, each seeing the options in a different order. All the agents ran on the same model, and I went in leaning toward pytest, so treat the scores as one input among several.

They scored these criteria, which count equally in the totals:

Legibility
Failure output
Cost of adding the next test
How much code we'd own
Toolchain fit
Coverage
Robustness

All four prototypes passed both healthy runs, and failed the right checks under the faults.

Option	Helper lines	Faulted run	Judges' totals, out of 35
shell	159	461s	26, 29, 28
unittest	205	244s	30, 29, 28
pytest	446	715s	28, 29, 28
Chainsaw	60	150s	22, 23, 21

Helper lines are code beyond the tests and bring-up, including license headers. Healthy runs took 107 to 153 seconds. Chainsaw runs its tests in parallel and the others run them serially, so the times aren't like for like. Judges 2 and 3 tied shell, unittest and pytest on points, and broke the tie for pytest. Judge 1 ranked unittest first. All three put shell third and Chainsaw last.

In short:

shell extends run.sh with a small helper library. It adds nothing new to the repo, but it checks JSON by matching text, which gets harder as checks look deeper into responses.
unittest writes the checks in Python's standard library, and leaves bring-up in run.sh. It doesn't add dependencies, but it has nothing like pytest's fixtures, so each scenario writes its own setup and cleanup.
pytest writes bring-up and checks in Python, with fixtures for shared setup and teardown. It was the cheapest to add a test to, but it has the most helper code and adds pytest as a dependency.
Chainsaw writes tests as YAML. Checks on Kubernetes resources are declarative, but requests to the gateways and log checks become shell scripts inside the YAML.

The brief, the run logs and each judge's full scoring are in bakeoff/ on the soggy-bottom branch.

Match the bake-off report to the comment posted on modelplaneai#473