fix: report the cause when the committer main task dies - #82
Merged
Merged
Conversation
An unhandled exception from consumer.commit() ends the streaming loop, and spawn() registered no done-callback, so the death itself logged nothing. The only ERROR a dead committer produced was the send_task guard's CommitterIsDeadError on the next message — a symptom, not a cause — while the real exception surfaced later at WARNING from close(). Error reporters promote ERROR to an event and keep lower levels as breadcrumbs, so a production incident arrived with no root cause attached. _on_committer_done now reports the cause at ERROR when the loop ends, and is the only place that does; close()'s duplicate branch is gone. Behaviour is unchanged: a dead committer stays dead. Widening _call_committer's except list to Exception would keep the loop alive by discarding batches it cannot commit, but an error that repeats every round becomes an infinite redelivery — the consumer looks healthy, offsets never advance, and nothing reaches the liveness probe. Respawning is worse, because _pending, _messages_queue and _uncommitted_count may already describe offsets partly committed before the throw, so a fresh loop can commit past work that never finished.
lesnik512
force-pushed
the
fix/committer-death-cause
branch
from
September 22, 2026 13:46
551cd4c to
761eec4
Compare
This was referenced Sep 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A production incident on 0.6.6 + faststream 0.7.6 surfaced in Sentry as
CommitterIsDeadError: Committer main task is not running, with nothing attached explaining whatkilled the committer. Capturing every log record from death to first guard failure on
mainshowswhy:
spawn()registered no done-callback, so the death itself emitted nothing.send_taskguard — a symptom, not a cause.close(), under a message aboutclose().Error reporters promote ERROR to an event and keep lower levels as breadcrumbs, which is exactly
why the incident report carried no root cause.
Change
_on_committer_donereports the cause at ERROR when the loop ends, and is now the only place acommitter death is reported —
close()'s duplicate branch is gone, so one death no longer logstwice at two levels. Cancellation and clean exit stay silent.
After:
What this deliberately does not change
Behaviour. A dead committer stays dead. Two repairs were considered and rejected:
_call_committer'sexceptlist toExceptionwould keep the loop alive bydiscarding batches it cannot commit, but an error that repeats every round becomes an infinite
redelivery: the consumer looks healthy, offsets never advance, and nothing reaches the liveness
probe.
_pending,_messages_queueand_uncommitted_countmayalready describe offsets partly committed before the throw, so a fresh loop can commit past work
that never finished.
Tests
Three added, one retargeted:
test_committer_reports_the_cause_when_the_main_task_dies— the regression guard, driving thereal spawned loop with a non-Kafka exception from
consumer.commitand asserting on the record'slevel and carried exception rather than on
caplog.text.test_committer_death_is_reported_before_any_later_message— ordering: the cause lands beforethe first
CommitterIsDeadError.test_committer_cancellation_is_not_reported_as_a_death.test_unhealthy_when_the_committer_died_under_a_running_handler— pins the chain a livenessprobe depends on: a dead committer fails the probe on its own, with
_is_runningstill true andno message having arrived.
test_committer_close_logs_when_task_already_diedretargeted totest_committer_close_is_a_noop_when_the_task_already_died, since the message it asserted on isthe duplicate being removed.
194 passed (190 before), coverage still 100%,
just lint-ciclean.Considered and dropped
Hardening
MockAIOKafkaConsumer.commitwith aiokafka's realcommit_structure_validate, to stop awrong
TopicPartitiontype reaching PyPI again. Tested by re-introducing the original bug at itsreal source (
processing.py's import) and running the unit suite: the only test that caught it wasthe existing
test_commit_task_partition_is_the_type_the_client_library_commits. The mock changedetected nothing extra, because the tests that reach
consumer.commitbuild their ownaiokafka.structs.TopicPartition, while the handler path that constructs the wrong type is testedagainst
MockKafkaBatchCommitterand never reaches a consumer. Dropped rather than shipped asfalse assurance.
Noted, not addressed
_streaming_iterationdiscards_commit_ready's return value —committed = Truewheneverreadywas non-empty, regardless of commit success — so a failed round re-arms the scheduler'stimeout deadline as if it had committed. Unrelated to this incident and touches
CommitScheduler'sinvariants.