[nicknikolakakis]

Fixes #368.

The e2e tests from #482 check that the InferenceGateway routes a request and rewrites its model, but not what comes back. A gateway that mangled the body, buffered a stream into one response, or turned a bad request into a 200 would pass them.

The mock engine now echoes the last user turn, streams chat.completion.chunk frames ending in [DONE] when a request sets stream, and answers 400 to a body that isn't JSON or has no messages. Usage stays fixed, so test_metering is unchanged. A new e2e/tests/test_serving.py checks the OpenAI and Anthropic response shapes, the echoed turn, the event-stream frames, and that a malformed body, a request without messages and an unrouted path all fail. Client.request gains raw to send a non-JSON body, and Response gains content_type for the streaming test.

This replaces the earlier unittest suite in e2e/verify, which #482 superseded. I ran the mock locally against every branch, but not the full two-cluster e2e, so it needs a test-e2e run. The rejection tests assert >= 400 rather than an exact status, since the gateway may refuse a malformed body before the engine sees it.

I have:

• Read and followed Modelplane's contribution process.
• Run nix flake check (or ./nix.sh flake check) and made sure it passes.
• Added or updated tests covering any composition function changes. This PR changes only the e2e tests and their mock engine.
• Signed off every commit with git commit -s.

[haarchri]

can you please add the checklist, rebase and run the test one time locally that will help to review the PR after that we can add the test-e2e label - thanks

[nicknikolakakis]

Checklist added to the description, and rebased onto main. Clean, no conflicts, and the diff is unchanged even though 4e3b603 touched e2e/run.sh and 798fcd1 touched nix/checks.nix.

On running it locally: ./nix.sh flake check passes on the rebased branch, which covers the ruff and license checks now that e2e/verify is in their scope (it checks aarch64-linux in the container and skips aarch64-darwin). The two-cluster e2e itself I still cannot run here, since crossplane project run needs the Nix-built function images. What I did verify is the pod wiring on a throwaway kind cluster: the suite goes green against the new mock with the pod reaching Succeeded, a deliberately wrong MODELPLANE_MODEL puts it in Failed, and the suite fails 5 of its 8 cases against the old mock. So the test-e2e label is still what would prove the full path.

[haarchri]

tests are failing with:

==> Running the behavioral suite (e2e/verify) against http://172.18.255.200/ml-team/mock
configmap/e2e-verify-suite created
pod/e2e-verify created
setUpModule (__main__) ... ERROR

======================================================================
ERROR: setUpModule (__main__)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/suite/test_serving.py", line 89, in setUpModule
    raise AssertionError(
AssertionError: http://172.18.255.200/ml-team/mock did not serve a chat completion within 300s (last status 400)

----------------------------------------------------------------------
Ran 0 tests in 300.376s

FAILED (errors=1)
verify: the behavioral suite did not pass (pod phase: Failed)
Error: Process completed with exit code 1.

[nicknikolakakis]

Thanks, that log had everything needed. The 400 was the mock engine's own, and the bug is mine.

From the run's e2e-diagnostics artifact, every readiness attempt in the gateway's access log:

"response_code":400, "response_code_details":"via_upstream",
"bytes_received":70, "bytes_sent":33,
"x-envoy-origin-path":"/v1/chat/completions",
"upstream_host":"10.244.0.25:8000"

via_upstream puts the 400 at the engine rather than the proxy. bytes_received: 70 is the readiness payload exactly, and bytes_sent: 33 is exactly {"error": "messages is required"}, which only the mock emits. The EPP agreed: "modelName":"mock","targetModelName":"mock" on every request, so routing and model resolution were never the problem.

The cause is that the endpoint picker is an ext_proc filter, and Envoy strips content-length from a request it hands to one unless allow_content_length_header is set, which defaults to false. The body reaches the engine chunked. The mock read it by content-length only, so it saw zero bytes, parsed {}, found no messages, and answered 400 to everything. The old mock never noticed because it ignored the body.

The mock now decodes a chunked body too. Reproduced and checked locally by running the engine straight out of the manifest:

request	before	after
POST with content-length	200	200
POST chunked, the shape CI sends	400	200
chunked stream, Anthropic, /v1/models	-	200
chunked malformed body / no messages	400	400
unknown path	404	404

The suite is 8/8 against the fixed mock, and ./nix.sh flake check passes. It is folded into the mock commit rather than added on top, per CONTRIBUTING. Ready for another test-e2e run whenever you are.

[haarchri]

with #473 we updated our test implementation, are you willing to double check the for needed changes ?

[nicknikolakakis]

Rebased onto main and rewritten as pytest in e2e/tests/test_serving.py. #482 already covers the served model, /v1/models and unclaimed 404, so I dropped those, along with the e2e/verify runner and the run.sh and nix changes. What's left is echo of the last user turn, SSE streaming, response shapes, and 400s for bad requests. nix flake check passes; the full run needs test-e2e.

[haarchri]

LGTM - thanks for multiple iterations