Skip to content

Account for full protobuf size when batching topic writes - #929

Merged
vgvoleg merged 3 commits into
mainfrom
fix-topic-writer-message-size
Oct 28, 2026
Merged

vgvoleg merged 3 commits into
mainfrom
fix-topic-writer-message-size

Conversation

@vgvoleg

@vgvoleg vgvoleg commented Oct 26, 2026 •

Copy link
Copy Markdown
Member

Topic write batching counted payload bytes and a fixed overhead, so messages with metadata could be grouped into a request exceeding the gRPC size limit.

Build each message's protobuf representation once and pack it directly into requests using its serialized size. Include message metadata, transaction identity, and nested field framing in the size calculation, then pass the prepared protobuf requests to the stream without rebuilding them.

Related to #930.

@codecov

codecov Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.64%. Comparing base (d6391fb) to head (901791f).

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #929      +/-   ##
==========================================
- Coverage   82.67%   82.64%   -0.04%     
==========================================
  Files          99       99              
  Lines       12961    12950      -11     
  Branches     1269     1266       -3     
==========================================
- Hits        10716    10703      -13     
- Misses       1794     1797       +3     
+ Partials      451      450       -1     
Flag Coverage Δ
integration 80.33% <80.00%> (-0.11%) ⬇️
unit 49.25% <100.00%> (+0.03%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
ydb/_grpc/grpcwrapper/common_utils.py 88.76% <100.00%> (ø)
ydb/_topic_common/test_helpers.py 87.75% <100.00%> (+0.25%) ⬆️
ydb/_topic_writer/topic_writer.py 85.86% <100.00%> (-0.87%) ⬇️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@vgvoleg vgvoleg changed the title Fix topic writer batching with message metadata Account for full protobuf size when batching topic writes Oct 27, 2026
@vgvoleg
vgvoleg requested a lite review from Copilot October 27, 2026 10:28

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Individually oversized messages must be rejected rather than sent as oversized RPCs.

1 open issue
What changed in this PR

This pull request updates topic-writer batching to enforce the complete serialized protobuf request size, including metadata, transaction identity, and framing overhead.

Changes:

  • Replaces fixed-overhead estimation with protobuf-aware size calculations.
  • Adds boundary tests for metadata, transactions, framing, codecs, and oversized messages.
  • Documents the serialized-size batching behavior and records the fix in the changelog.

A critical issue remains: individually oversized messages can still be sent instead of being rejected with a terminal TopicWriterError.

File Description
ydb/​_topic_writer/​topic_writer.py Implements protobuf-size-based request splitting.
ydb/​_topic_writer/​topic_writer_test.py Adds serialized-size and boundary-condition tests.
docs/​topic.rst Documents size-aware topic write batching.
CHANGELOG.md Records the topic-writer batching fix.

Review effort: Lite

Comment thread ydb/_topic_writer/topic_writer.py Resolved

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Oversized individual messages can still be appended and sent in requests exceeding the gRPC limit, potentially causing indefinite retries and blocking close().

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Medium severity Reject oversized messages before appending to requests

ydb/​_topic_writer/​topic_writer.py:286

When a single serialized message is larger than the limit, this branch first emits the current request and then appends that message to a new request anyway. The resulting RPC is guaranteed to exceed the gRPC send limit, so the writer can still retry the oversized request indefinitely and leave close() waiting, which is the failure mode this change is intended to prevent. Detect an oversized message before appending it and fail the write with a terminal TopicWriterError (the new oversized-message test currently codifies sending an over-limit request).

1 resolved since last review

Review effort: Lite

@vgvoleg
vgvoleg merged commit e26a1f3 into main Oct 28, 2026
35 checks passed
@vgvoleg
vgvoleg deleted the fix-topic-writer-message-size branch October 28, 2026 09:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants