Skip to content

fix: report the cause when the committer main task dies - #82

Merged
lesnik512 merged 1 commit into
mainfrom
fix/committer-death-cause
Sep 22, 2026
Merged

lesnik512 merged 1 commit into
mainfrom
fix/committer-death-cause

Conversation

@lesnik512

@lesnik512 lesnik512 commented Sep 22, 2026 •

Copy link
Copy Markdown
Member

Problem

A production incident on 0.6.6 + faststream 0.7.6 surfaced in Sentry as
CommitterIsDeadError: Committer main task is not running, with nothing attached explaining what
killed the committer. Capturing every log record from death to first guard failure on main shows
why:

--- logs emitted AT the moment of death: 0 ---

ERROR    carries=CommitterIsDeadError   Kafka middleware. Committer is dead
INFO     carries=-                      Kafka middleware. Shutting down middleware handler
WARNING  carries=RuntimeError           Committer task had already died before close()
  1. spawn() registered no done-callback, so the death itself emitted nothing.
  2. The only ERROR was the send_task guard — a symptom, not a cause.
  3. The real exception surfaced later, at WARNING, from close(), under a message about close().

Error reporters promote ERROR to an event and keep lower levels as breadcrumbs, which is exactly
why the incident report carried no root cause.

Change

_on_committer_done reports the cause at ERROR when the loop ends, and is now the only place a
committer death is reported — close()'s duplicate branch is gone, so one death no longer logs
twice at two levels. Cancellation and clean exit stay silent.

After:

ERROR    carries=RuntimeError           Committer main task died; offsets will no longer be committed
ERROR    carries=CommitterIsDeadError   Kafka middleware. Committer is dead

What this deliberately does not change

Behaviour. A dead committer stays dead. Two repairs were considered and rejected:

  • Widening _call_committer's except list to Exception would keep the loop alive by
    discarding batches it cannot commit, but an error that repeats every round becomes an infinite
    redelivery: the consumer looks healthy, offsets never advance, and nothing reaches the liveness
    probe.
  • Respawning the loop is worse — _pending, _messages_queue and _uncommitted_count may
    already describe offsets partly committed before the throw, so a fresh loop can commit past work
    that never finished.

Tests

Three added, one retargeted:

  • test_committer_reports_the_cause_when_the_main_task_dies — the regression guard, driving the
    real spawned loop with a non-Kafka exception from consumer.commit and asserting on the record's
    level and carried exception rather than on caplog.text.
  • test_committer_death_is_reported_before_any_later_message — ordering: the cause lands before
    the first CommitterIsDeadError.
  • test_committer_cancellation_is_not_reported_as_a_death.
  • test_unhealthy_when_the_committer_died_under_a_running_handler — pins the chain a liveness
    probe depends on: a dead committer fails the probe on its own, with _is_running still true and
    no message having arrived.
  • test_committer_close_logs_when_task_already_died retargeted to
    test_committer_close_is_a_noop_when_the_task_already_died, since the message it asserted on is
    the duplicate being removed.

194 passed (190 before), coverage still 100%, just lint-ci clean.

Considered and dropped

Hardening MockAIOKafkaConsumer.commit with aiokafka's real commit_structure_validate, to stop a
wrong TopicPartition type reaching PyPI again. Tested by re-introducing the original bug at its
real source (processing.py's import) and running the unit suite: the only test that caught it was
the existing test_commit_task_partition_is_the_type_the_client_library_commits. The mock change
detected nothing extra, because the tests that reach consumer.commit build their own
aiokafka.structs.TopicPartition, while the handler path that constructs the wrong type is tested
against MockKafkaBatchCommitter and never reaches a consumer. Dropped rather than shipped as
false assurance.

Noted, not addressed

_streaming_iteration discards _commit_ready's return value — committed = True whenever
ready was non-empty, regardless of commit success — so a failed round re-arms the scheduler's
timeout deadline as if it had committed. Unrelated to this incident and touches CommitScheduler's
invariants.

An unhandled exception from consumer.commit() ends the streaming loop, and
spawn() registered no done-callback, so the death itself logged nothing. The
only ERROR a dead committer produced was the send_task guard's
CommitterIsDeadError on the next message — a symptom, not a cause — while the
real exception surfaced later at WARNING from close(). Error reporters promote
ERROR to an event and keep lower levels as breadcrumbs, so a production
incident arrived with no root cause attached.

_on_committer_done now reports the cause at ERROR when the loop ends, and is
the only place that does; close()'s duplicate branch is gone.

Behaviour is unchanged: a dead committer stays dead. Widening _call_committer's
except list to Exception would keep the loop alive by discarding batches it
cannot commit, but an error that repeats every round becomes an infinite
redelivery — the consumer looks healthy, offsets never advance, and nothing
reaches the liveness probe. Respawning is worse, because _pending,
_messages_queue and _uncommitted_count may already describe offsets partly
committed before the throw, so a fresh loop can commit past work that never
finished.
@lesnik512
lesnik512 force-pushed the fix/committer-death-cause branch from 551cd4c to 761eec4 Compare September 22, 2026 13:46
@lesnik512
lesnik512 merged commit b936d06 into main Sep 22, 2026
11 checks passed
@lesnik512
lesnik512 deleted the fix/committer-death-cause branch September 22, 2026 13:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant