Skip to content

_commit_ready's success flag is computed, tested, and then discarded by its only caller #84

Description

@lesnik512

_commit_ready returns all(results) — false when any _call_committer in the round failed
(batch_committer.py:117). Its only production caller drops it on the floor
(batch_committer.py:174-187):

committed = False
if decision.should_commit:
    ready = self._pending.take_ready()
    if ready:
        await self._commit_ready(ready)   # return value discarded
        committed = True

self._scheduler.note_committed(
    now=loop.time(), committed=committed,
    timeout_fired=decision.timeout_fired, pending_empty=not self._pending,
)

So committed means "a round was attempted", not "a round succeeded". The tests do pin the flag
(test_commit_ready_returns_false_on_commit_failure, test_kafka_committer.py:725), which makes
the discard easy to misread as an oversight.

Where it could actually matter

committed only feeds the deadline re-arm in note_committed
(_commit_scheduler.py:93-95):

if committed or timeout_fired:
    self._timeout_deadline = (now + self._batch_timeout) if not pending_empty else None

In the common failure shapes it makes no difference. A KafkaError re-queues the batch and a
CommitFailedError discards it; either way pending drains, pending_empty is true, and the
deadline is cleared to None whichever value committed held.

The one shape where it diverges is a partially-ready multi-partition round: take_ready()
lifted the ready prefixes, the commit for them failed, and pending still holds unfinished tasks on
other partitions. committed=True then pushes the deadline out a full commit_batch_timeout_sec
even though nothing was committed, delaying the next attempt for work that is already finished.

Why the obvious fix is probably wrong

Passing committed=all(results) through means that on a failed round with pending non-empty,
neither committed nor timeout_fired is set, so the branch is skipped and _timeout_deadline
keeps its old, now-past value. wait_timeout then returns zero or negative every iteration and
the loop spins — the same busy-retry failure mode #62 documents as the reason its own obvious fix
was rejected. Any real fix has to distinguish "retry promptly" from "back off", which is the
retry-with-backoff feature #62 also lands on.

Where this came from

Noticed while fixing #82 (committer death reporting) and deliberately left out of that PR's scope.
Not a cause of, and not related to, the CommitterIsDeadError incident in #83.

Triage question: is the discard deliberate — holding the "pending empty ⇒ deadline None"
invariant and avoiding a stale past deadline — or is the partially-ready case a real delay worth
fixing? If deliberate, a sentence in _streaming_iteration saying so would stop the next reader
re-filing this. If it is bundled into any future retry-with-backoff work, fold it in with #62.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    wontfixThis will not be worked on

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions