_commit_ready returns all(results) — false when any _call_committer in the round failed
(batch_committer.py:117). Its only production caller drops it on the floor
(batch_committer.py:174-187):
committed = False
if decision.should_commit:
ready = self._pending.take_ready()
if ready:
await self._commit_ready(ready) # return value discarded
committed = True
self._scheduler.note_committed(
now=loop.time(), committed=committed,
timeout_fired=decision.timeout_fired, pending_empty=not self._pending,
)
So committed means "a round was attempted", not "a round succeeded". The tests do pin the flag
(test_commit_ready_returns_false_on_commit_failure, test_kafka_committer.py:725), which makes
the discard easy to misread as an oversight.
Where it could actually matter
committed only feeds the deadline re-arm in note_committed
(_commit_scheduler.py:93-95):
if committed or timeout_fired:
self._timeout_deadline = (now + self._batch_timeout) if not pending_empty else None
In the common failure shapes it makes no difference. A KafkaError re-queues the batch and a
CommitFailedError discards it; either way pending drains, pending_empty is true, and the
deadline is cleared to None whichever value committed held.
The one shape where it diverges is a partially-ready multi-partition round: take_ready()
lifted the ready prefixes, the commit for them failed, and pending still holds unfinished tasks on
other partitions. committed=True then pushes the deadline out a full commit_batch_timeout_sec
even though nothing was committed, delaying the next attempt for work that is already finished.
Why the obvious fix is probably wrong
Passing committed=all(results) through means that on a failed round with pending non-empty,
neither committed nor timeout_fired is set, so the branch is skipped and _timeout_deadline
keeps its old, now-past value. wait_timeout then returns zero or negative every iteration and
the loop spins — the same busy-retry failure mode #62 documents as the reason its own obvious fix
was rejected. Any real fix has to distinguish "retry promptly" from "back off", which is the
retry-with-backoff feature #62 also lands on.
Where this came from
Noticed while fixing #82 (committer death reporting) and deliberately left out of that PR's scope.
Not a cause of, and not related to, the CommitterIsDeadError incident in #83.
Triage question: is the discard deliberate — holding the "pending empty ⇒ deadline None"
invariant and avoiding a stale past deadline — or is the partially-ready case a real delay worth
fixing? If deliberate, a sentence in _streaming_iteration saying so would stop the next reader
re-filing this. If it is bundled into any future retry-with-backoff work, fold it in with #62.
_commit_readyreturnsall(results)— false when any_call_committerin the round failed(
batch_committer.py:117). Its only production caller drops it on the floor(
batch_committer.py:174-187):So
committedmeans "a round was attempted", not "a round succeeded". The tests do pin the flag(
test_commit_ready_returns_false_on_commit_failure,test_kafka_committer.py:725), which makesthe discard easy to misread as an oversight.
Where it could actually matter
committedonly feeds the deadline re-arm innote_committed(
_commit_scheduler.py:93-95):In the common failure shapes it makes no difference. A
KafkaErrorre-queues the batch and aCommitFailedErrordiscards it; either way pending drains,pending_emptyis true, and thedeadline is cleared to
Nonewhichever valuecommittedheld.The one shape where it diverges is a partially-ready multi-partition round:
take_ready()lifted the ready prefixes, the commit for them failed, and pending still holds unfinished tasks on
other partitions.
committed=Truethen pushes the deadline out a fullcommit_batch_timeout_seceven though nothing was committed, delaying the next attempt for work that is already finished.
Why the obvious fix is probably wrong
Passing
committed=all(results)through means that on a failed round with pending non-empty,neither
committednortimeout_firedis set, so the branch is skipped and_timeout_deadlinekeeps its old, now-past value.
wait_timeoutthen returns zero or negative every iteration andthe loop spins — the same busy-retry failure mode #62 documents as the reason its own obvious fix
was rejected. Any real fix has to distinguish "retry promptly" from "back off", which is the
retry-with-backoff feature #62 also lands on.
Where this came from
Noticed while fixing #82 (committer death reporting) and deliberately left out of that PR's scope.
Not a cause of, and not related to, the
CommitterIsDeadErrorincident in #83.Triage question: is the discard deliberate — holding the "pending empty ⇒ deadline None"
invariant and avoiding a stale past deadline — or is the partially-ready case a real delay worth
fixing? If deliberate, a sentence in
_streaming_iterationsaying so would stop the next readerre-filing this. If it is bundled into any future retry-with-backoff work, fold it in with #62.