[MaxGhenis]

Problem

Three follow-ups from #6 (the settled-rejection port) in CollectorDelivery, the telemetry service's collector delivery. All three are in Microcosm's copy of this code too; PolicyEngine/microcosm#1179 makes the same change there.

1. A refused registration warned about the wrong thing. _register called _make_local_only without a message, so the service's own run got the default REJECTED_CREDENTIAL_MESSAGE: "the ambient Hugging Face credential was not accepted as a PolicyEngine organization member". By then the token exchange had accepted that credential as a member. The warning was false for a 409 (the run's owner registering it with other metadata), a 422 (a registration outside the collector's schema) and any other settled 4xx. It was also loose for a 403 on registration or on events, where the collector accepted the login and refused it this run.
2. Batches were bounded by count, not bytes. EventSpool.batch returns up to 100 events. The collector answers a request body over 1 MiB with 413 before reading it, and since Make a settled collector rejection local-only instead of retrying it forever #6 a 413 makes the run local-only. At the service's own bounds (message cut to 500 characters, details to 8,192 bytes, identifiers at the collector's 192-character maximum), a full batch is 964,904 bytes with ASCII messages, 1,214,904 with 2-byte messages and 1,514,904 with emoji, because json.dumps escapes each non-ASCII character to 6 or 12 bytes.
3. A run the collector lost went local-only. The collector answers events for a run it does not have with 404, before it checks the owner. The delivery keeps the runs it registered in memory, so if the collector loses one (a database restored from before the run was registered, say), every later batch got 404 and the run went local-only (collector_rejected_events).

Change

All in src/microcosm_emitter/telemetry/service/collector.py and constants.py.

1. Warnings say what the collector refused. _make_local_only's message no longer has a default; every caller names one.
   • Registration refused with 409, 422 or any other settled 4xx: REJECTED_REGISTRATION_MESSAGE, "the collector rejected its registration (HTTP {status}), so it will not be retried".
   • Registration or events refused with 403: REJECTED_RUN_USER_MESSAGE, "the collector refused the ambient Hugging Face user for it (HTTP 403). A run belongs to the user who registered it first."
   • Token exchange refused with 401 or 403: unchanged, the credential message. It is the only one that mentions the credential or organization membership.
   • Missing credential: unchanged, NO_CREDENTIAL_MESSAGE once per service and nothing per run, now passed as message=None.
   • The rule that only the service's own run is warned about is unchanged.
2. Requests are cut to the collector's limit. MAX_EVENTS_REQUEST_BYTES = 1_048_576, the collector's own figure, with no margin: the contract test below has the collector accept a body of exactly that size. _events_fitting_request keeps the longest run of leading events whose request body fits. It measures with _request_body, the same function _http_post now sends, so the bytes measured are the bytes sent. Events go in sequence order, one request after another, with none skipped.
   • An event too large to send alone is still sent, alone, once the events before it are delivered. Every event is bounded before it is queued, so this needs an event from outside those bounds: another version's service on the shared spool, say. The collector answers 413, which makes the run local-only, as the settled-rejection rule does for any event the collector will not accept (a 422 for one malformed event does the same). Dropping it locally instead would leave a hole in the run's sequence that nothing records, so the event goes to the collector to refuse.
3. A 404 for a run's events registers the run again, once. The delivery forgets the run's registration and remembers that it did (_registering_again). The next pass registers the run and sends the same batch, with no retry delay. A second 404 with no batch of the run acknowledged in between is the collector's settled answer, and the run goes local-only as before. An acknowledged batch clears the mark.

README and a changelog.d fragment describe all three.

Is registering again safe?

The follow-up asked for this to be worked out before anything was decided. The answer does not turn on a judgment call, so it is implemented here rather than queued:

• Who owns the run. The collector gives a run it does not have to whoever registers it first (register_run: insert on conflict do nothing, then compare owners). So the re-registered run belongs to the login that registers it again. For a build's own run that is the build's login. For a run a service adopted, it is the adopter's login, which the collector had already accepted for that run: an adopter reaches the events request only after its own registration of the run was answered 201. If another user registered the lost run id first, registering again gets 403 and is handled as any registration 403 is: the build's own run goes local-only with the user warning, and an adopted run is passed over for that login. This is the collector's existing first-registrant rule, applied to a run it lost; nothing new decides ownership.
• No loop. Between two registrations of a run that the collector accepts, the service has had a 404 for the run's events; from the second on, an acknowledged batch as well. So the collector accepts a run's registration from one service at most two more times than it acknowledges that service a batch of the run, and each acknowledged batch removes events from the spool. (Attempts answered 503, 429 or 408 are retried after the retry delay, as every registration is.) The property test below checks this after every step.
• A 404 that does not mean a lost run (a collector without the events route, a proxy in front of it) costs one more registration and one more events request, then the run goes local-only as before. If the registration is refused too, it settles on that answer.
• What the dashboard shows. A recovered run holds only the events sent after the loss: those before it were acknowledged and removed from the spool. Every event keeps its sequence number, so the gap is visible. Before this change none of the later events reached the collector at all.

How the collector answers (read for this change)

From PolicyEngine/calibration-diagnostics telemetry-service/telemetry_collector/ at 80a2a4ac, the commit this repo's CI pins. app.py, which holds the body limit and every status below, is unchanged from there to that repo's main (280459c).

Request	Answer
Any body with Content-Length over 1,048,576, or a streamed body that grows past it	413, from RequestBodyLimitMiddleware, before routing or authentication. A body of exactly 1,048,576 bytes passes.
Registration of a run another user owns	403
Registration by the owner with other metadata	409
Registration by the owner with the same metadata	201 again (repeatable)
Registration outside the schema	422
Events for a run the collector does not have	404, before the owner check
Events for another user's run, or from a producer not registered for the run	403

Invariants

Each is checked by the tests named. The scope is stated per item and nothing is claimed beyond it.

1. One warning, about the request refused. For a service's own run, for every status that makes it local-only (401 and 403 on the token exchange; every 4xx but 401, 408 and 429 on registration and on events), exactly one warning is printed, and its text is fixed by the request and the status as listed under Change. Only the token-exchange refusal mentions the credential or organization membership. Exhaustive over those statuses, fake transport: test_every_refusal_prints_one_warning_about_the_request_refused; exact texts for 409, 422 and 403: test_a_rejected_registration_warning_says_what_the_collector_refused; the same three against the real collector: test_a_refused_registration_warns_about_the_registration_not_the_credential (CI collector job). Another build's run is never warned about, and a missing credential is reported once per service, not per run (existing tests, now also killing the mutants listed below).
2. Every request fits, in order, none skipped. For any queue of events bounded as the service bounds them: every events request holds between 1 and 100 events; its body is at most 1,048,576 bytes; it holds fewer than 100 only when the queue has fewer left or the next event would not fit; and the collector receives every event once, in sequence order. Hypothesis, up to 230 events per queue in up to five shapes (messages of 1 to 600 characters of ASCII, 2-byte, emoji, control or quote characters; details of up to 60 keys of 2,100 characters, cut to 8,192 bytes), plus the all-largest and alternating examples: test_every_request_fits_and_events_arrive_in_order.
   • The cut itself, for any list of JSON events and any limit from 0 to 400 bytes, against the bytes that would be sent: the result is a prefix, empty only for an empty list, within the limit unless it is a single event, and one event more would exceed the limit (test_the_events_sent_are_the_longest_leading_run_that_fits, 400 examples plus the exact 18- and 17-byte edges).
   • The boundary itself: two events whose request is exactly 1,048,576 bytes go together, and one byte more splits them (test_two_events_of_exactly_the_limit_go_together_and_a_byte_more_apart). _http_post sends exactly _request_body's bytes and says so in Content-Length (test_the_bytes_measured_are_the_bytes_sent, a loopback server).
   • Against the real collector: 100 of the largest events arrive in two requests, both accepted; a body of exactly the limit is accepted (CI collector job).
3. An event too large to send alone is sent alone after everything before it is delivered, and the collector's 413 makes the run local-only, with nothing sent after (test_an_event_too_large_to_send_alone_is_left_to_the_collector). The limit is the collector's to enforce: were it raised there, such an event would be delivered like any other (test_an_event_over_the_limit_that_the_collector_takes_does_not_stop_the_run).
4. A lost run costs a bounded number of registrations. For one service and one run: between two registrations the collector accepts there is a 404 for the run's events, and from the second on an acknowledged batch too; a 404 makes the run local-only only when the service's previous answer for the run's events was also 404 (in this test, where the collector answers only 201, 202 and 404, the three answers before are exactly events 404, registration 201, events 404); nothing about the run is sent after; and what the collector acknowledged is always a prefix of the queue, in order. Checked after every step of a Hypothesis test in which the collector loses the run, or stops keeping registrations, at any point (test_a_lost_run_costs_a_bounded_number_of_registrations, 200 examples plus two explicit). No other answer to events (every status from 200 to 599 but 202 and 404) registers a run again (test_no_other_answer_to_events_registers_the_run_again, exhaustive).
5. Registering again is answered like any registration. 403, 409 and 422 settle the run with the registration warning for that status; 503, 429 and 408 wait out the retry delay and try again (test_registering_again_is_answered_like_any_registration, test_registering_again_waits_out_a_transient_answer). The first 404 sets no retry delay: the next pass, with no time passed, registers and delivers (test_a_run_the_collector_lost_is_registered_again_and_delivered).
6. Another build's run, as in Keep one build's telemetry login from deciding other builds' runs #5 and Make a settled collector rejection local-only instead of retrying it forever #6. The multi-build property test's fake collector can now lose any build's run at any step (forget) and can be set never to find one run's events. With that, after every step: another producer's run goes local-only only on an answer about the run itself, which now includes a second 404 with no batch acknowledged in between, never because of this service's login; and once every service has gone, each pending run is delivered in full by a login that owns it, unless the collector refuses the run itself (now including never finding it). test_no_service_decides_another_producers_run_from_its_own_credential, with an explicit example where Alice's run is lost and Bob's build of the same run id registers it first. Deterministic cases: test_an_adopted_orphan_the_collector_loses_is_registered_again, test_a_lost_run_another_user_registers_first_is_refused_its_first_owner, and an unfindable_run case in the orphan-refusal tests.

Both packages. Microcosm's copy of this code gets the same change in PolicyEngine/microcosm#1179. With #6 now on #5's 093efb1, which mirrors the rest of microcosm#1177, the two collector.py files are identical apart from the package path. Checked with:

diff <(sed 's/microcosm\.build\.telemetry_emitter_service/microcosm_emitter.telemetry.service/g' microcosm/packages/microcosm-build/src/microcosm/build/telemetry_emitter_service/collector.py) microcosm-emitter/src/microcosm_emitter/telemetry/service/collector.py

which prints nothing at #1179's b3710aa04 and this PR's 5df012b. constants.py differs only by #5's comment on the lease-file layout. One script applied each change to its base (615d84f89 there, 9d4ee06 here) and reproduces each head's two files byte for byte.

tests/telemetry/test_collector_delivery.py is Microcosm's test_telemetry_collector_delivery.py with its import block swapped, and otherwise identical.

Tests

• tests/telemetry/test_collector_delivery.py (new): the warning, request-size and lost-run tests named above, against a fake collector that answers as the hosted one does, including its body limit and repeatable registration.
• tests/telemetry/test_foreign_runs.py: the fake collector can lose a run (forget) or never find one (unfindable_run_ids); the invariant check accepts a second 404 as a refusal about the run, and only that; the property test draws both; two deterministic tests and an explicit example as above.
• tests/telemetry/test_delivery.py: 404 leaves the settled-rejection table (it is now registered again first), and the exhaustive 4xx table expects one re-registration for events 404, the status named in every warning, and no "organization member" in any.
• tests/test_collector_contract.py (CI collector job only): the 422 case kept, the 404 case replaced by one where the real collector accepts the second registration and the batch; refused registrations (409, 422, 403) and their warnings; 100 of the largest events in two accepted requests; a body of exactly the limit accepted. The lost run is simulated by answering the first registration without passing it on, so the test does not depend on how the delivery keys runs.

Verification

Local results, the mutation pass and CI are being added here as they finish. Until this section says otherwise, treat the PR as unverified.

Limits and follow-ups

• The contract tests run only in CI's collector job, which needs the pinned collector and PostgreSQL; locally they were collected and skipped.
• A token exchange answered with a status other than 200, 401 or 403 is still retried without end, as in Microcosm. Unchanged here.
• The registration memory is per process. A new service, or a run adopted by another service, starts with a fresh one, so each registers the run once at start, as before.
• Merge order. This is stacked on Make a settled collector rejection local-only instead of retrying it forever #6, which is stacked on Keep one build's telemetry login from deciding other builds' runs #5. Merge Keep one build's telemetry login from deciding other builds' runs #5, retarget Make a settled collector rejection local-only instead of retrying it forever #6 to main, merge Make a settled collector rejection local-only instead of retrying it forever #6, then retarget this to main and merge. Don't delete a base branch through the API or --delete-branch before the PRs on it are retargeted: that closes them.

🤖 Generated with Claude Code