Skip to content

feat(sdk): consume the request counter for partition operations - #3958

Merged
spetz merged 8 commits into
masterfrom
fix_sdk_request_id_for_partition
Aug 31, 2026
Merged

feat(sdk): consume the request counter for partition operations#3958
spetz merged 8 commits into
masterfrom
fix_sdk_request_id_for_partition

Conversation

@numinnex

@numinnex numinnex commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Our partition operations did not use the request counter which meant that partition ops did not participate in the deduplication process. This PR makes the partition operations bump the request counter for all of the SDKs

@github-actions github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Aug 24, 2026
@codecov

codecov Bot commented Aug 24, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.95775% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.92%. Comparing base (dc1cdf2) to head (f3075cc).

Files with missing lines Patch % Lines
core/consensus/src/client_table.rs 85.71% 1 Missing and 1 partial ⚠️
...he/iggy/client/async/tcp/vsr/ConsensusSession.java 0.00% 1 Missing and 1 partial ⚠️
foreign/go/internal/vsr/envelope.go 0.00% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff            @@
##             master    #3958   +/-   ##
=========================================
  Coverage     84.92%   84.92%           
+ Complexity     1405     1402    -3     
=========================================
  Files          1225     1225           
  Lines        179932   179943   +11     
  Branches     146219   146249   +30     
=========================================
+ Hits         152812   152823   +11     
+ Misses        23082    23063   -19     
- Partials       4038     4057   +19     
Components Coverage Δ
Rust Core 85.81% <96.42%> (+<0.01%) ⬆️
Java SDK 67.29% <0.00%> (-0.06%) ⬇️
C# SDK 75.41% <ø> (+0.10%) ⬆️
Python SDK 90.06% <ø> (ø)
PHP SDK 85.65% <ø> (ø)
Node SDK 96.15% <100.00%> (-0.10%) ⬇️
Go SDK 69.35% <0.00%> (+0.01%) ⬆️
Files with missing lines Coverage Δ
core/binary_protocol/src/consensus/header.rs 81.87% <ø> (ø)
core/sdk/src/quic/quic_client.rs 75.42% <ø> (ø)
core/sdk/src/vsr.rs 94.49% <100.00%> (+0.48%) ⬆️
core/server/src/dispatch.rs 89.54% <ø> (+0.06%) ⬆️
core/server/src/http/session.rs 95.67% <ø> (ø)
core/simulator/src/client.rs 96.34% <100.00%> (-0.01%) ⬇️
core/simulator/src/lib.rs 96.95% <ø> (ø)
foreign/csharp/Iggy_SDK/Vsr/ConsensusSession.cs 98.96% <ø> (-0.04%) ⬇️
foreign/csharp/Iggy_SDK/Vsr/VsrHeader.cs 94.73% <ø> (ø)
foreign/csharp/Iggy_SDK/Vsr/VsrOperation.cs 95.83% <ø> (-0.11%) ⬇️
... and 13 more

... and 37 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@numinnex

Copy link
Copy Markdown
Contributor Author

Merge after #3945

@hubcio hubcio left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the id half looks right and lands uniformly in all five sdks. the checksum half doesn't belong in the same pr. inline comments cover what's inside the diff - the rest sits outside it:

core/server/src/dispatch.rs:1594-1604 - the partition path overwrites new_header.client with transport_client_id and clamps request to max(1). so the client half of a (client, request) dedup key is already discarded before the partition plane sees it, and transport_client_id changes on the very reconnect a retry follows. minting distinct ids buys a future dedup layer nothing until that rewrite changes too. the comment sitting right there - "a zero request id (the SDK does not number data-plane ops) is normalized" - is falsified by this pr and should be updated in it.

core/sdk/src/tcp/tcp_client.rs:1266 - the new hash runs inside the consensus_session std mutex. the comment at :121-124 picks std over tokio precisely because that section is "pure CPU"; it's now a full-batch hash blocking a tokio worker. quic_client.rs:630 is the same shape. hash the payload before taking the lock.

foreign/java/java-sdk/src/main/java/org/apache/iggy/client/async/tcp/vsr/ConsensusSession.java:56 - beginRegister() resets requestCounter to 1 but leaves correlationCounter alone. partition ops now draw from the counter that resets, so a re-login with an in-flight send can re-mint a RequestKey already in pendingRequests; registerRequest throws and writeVsrFrame turns that into closeChannel. that window used to be metadata-only.

core/simulator/src/client.rs:121-144 - still models the pre-pr client: partition ids from a separate counter at 1 << 63, metadata ids contiguous, checksum zero. the one tool that could find this interaction can't generate the pattern any more. its doc at :64 and :124-125 also still claims a gap-free requirement and a RequestGap, which client_table.rs:525 says does not exist.

stale doc sites: core/binary_protocol/src/consensus/header.rs:290 says the wire sends zero, client_table.rs:581 and :2825 repeat it, and node header.ts:116, go header.go:119, csharp VsrHeader.cs:24, java VsrHeaders.java:29 all state the server does not read the field. six places, all false. fix header.rs first since the others quote it.

foreign/go/internal/vsr/envelope_test.go:164 - the unbound-partition test checks only the error; its metadata twin at :156 also pins that the rejection burns no id. that invariant is live now that partition ops draw ids.

core/server/src/dispatch.rs:1015 - stamping makes the checksum-mismatch deny path reachable for SendMessages for the first time. it's terminal, no retry, and nothing tests it.

and nothing anywhere sends n batches then retries a metadata op to prove the gapped sequence still dedups end to end. worth one case in core/integration/tests/sdk/.

Comment thread core/sdk/src/vsr.rs Outdated
Comment thread core/sdk/src/vsr.rs Outdated
Comment thread core/sdk/src/vsr.rs Outdated
Comment thread core/sdk/src/vsr.rs Outdated
Comment thread core/sdk/src/vsr.rs
Comment thread foreign/node/src/wire/vsr/index.ts
@github-actions github-actions Bot added S-waiting-on-author PR is waiting on author response and removed S-waiting-on-review PR is waiting on a reviewer labels Aug 27, 2026
@numinnex
numinnex force-pushed the fix_sdk_request_id_for_partition branch from a045ec0 to 92c9ce8 Compare August 27, 2026 13:53
@numinnex

Copy link
Copy Markdown
Contributor Author

/ready

@github-actions github-actions Bot added S-waiting-on-review PR is waiting on a reviewer and removed S-waiting-on-author PR is waiting on author response labels Aug 27, 2026

@spetz spetz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some extra findings

Simulator

core/simulator/src/client.rs still models partition operations with a separate counter starting at 1 << 63, and its test asserts that partition operations do not consume the shared counter. Please update the simulator to use one counter for all replicated operations so it can generate the same sequence as the SDKs after this PR.

Server comment and dependency

core/server/src/dispatch.rs:1602 still says the SDK does not number data-plane operations, which becomes false after this PR. The compatibility normalization can remain, but the comment should distinguish current SDKs from older or internal callers. The stable VSR client-ID rewrite is handled by PR #3959, so that dependency should also be made explicit.

Go rejection invariant

foreign/go/internal/vsr/envelope_test.go verifies that an unbound metadata request does not consume an ID, but the equivalent partition test only checks the error. Since partition operations now consume the same counter, please also assert that CurrentRequestID() remains 1 after the rejection.

Existing checksum documentation

Several comments still claim request checksums are always zero or are not read by the server, including header.rs, client_table.rs, and the Go, Node, C#, and Java header mirrors. That was already stale before this final diff, so it does not need to block this PR, but it should be cleaned up separately.

Comment thread core/consensus/src/client_table.rs
@github-actions github-actions Bot added S-waiting-on-author PR is waiting on author response and removed S-waiting-on-review PR is waiting on a reviewer labels Aug 31, 2026
@numinnex
numinnex force-pushed the fix_sdk_request_id_for_partition branch from aa4c203 to 13c671c Compare August 31, 2026 15:04
@numinnex

Copy link
Copy Markdown
Contributor Author

/ready

@github-actions github-actions Bot added S-waiting-on-review PR is waiting on a reviewer and removed S-waiting-on-author PR is waiting on author response labels Aug 31, 2026
@spetz

spetz commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

LGTM after deep review.

spetz
spetz previously approved these changes Aug 31, 2026
hubcio
hubcio previously approved these changes Aug 31, 2026
@spetz
spetz merged commit 6ef0c16 into master Aug 31, 2026
98 checks passed
@spetz
spetz deleted the fix_sdk_request_id_for_partition branch August 31, 2026 19:01
@github-actions github-actions Bot removed the S-waiting-on-review PR is waiting on a reviewer label Aug 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants