Skip to content

fix(connectors): propagate sink plugin consume failure status - #3951

Closed
minleejae wants to merge 2 commits into
apache:masterfrom
minleejae:fix-sink-consume-status
Closed

fix(connectors): propagate sink plugin consume failure status#3951
minleejae wants to merge 2 commits into
apache:masterfrom
minleejae:fix-sink-consume-status

Conversation

@minleejae

Copy link
Copy Markdown

Binds the iggy_sink_consume FFI status and returns RuntimeError::SinkConsumeFailed { plugin_id, status } on non-zero, matching the existing iggy_sink_open check in the same file. The caller's existing Err path handles logging and skips the processed counter, so a failed batch is no longer counted as processed.

Includes a regression test with stub extern "C" callbacks for both paths — the failure-path test fails on master with expected SinkConsumeFailed, got success and passes with this change.

Closes #3950

@github-actions

Copy link
Copy Markdown

Thanks for the PR. It is labeled S-waiting-on-review and queued for review.

Slash commands (own line, regular comment) move it around the queue:

  • /ready - back to S-waiting-on-review after addressing feedback
  • /author - flip to S-waiting-on-author while you finish changes
  • /request-review @user-or-team - request a reviewer

See CONTRIBUTING.md for details.

@github-actions github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Aug 22, 2026
Comment thread core/connectors/runtime/src/sink.rs Outdated
// the SDK returns non-zero when the sink's consume() errors or the batch cannot
// be deserialized. Ignoring it would count the batch as processed and advance
// consumer offsets over messages the sink never stored — the same silent-loss
// class that the iggy_sink_open status check prevents at startup.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

while I agree that the plugin should exit when plugin returns non-zero status, simply exitting the plugin does not solve the problem highlighted in the issue.

I think we should fix the comment and update the issue description to reflect this.

Also see discussion: #2927

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right — reworded the comment and updated the issue description: this change is fail-fast (surfaces the failure, stops counting failed batches as processed), not redelivery, since offsets are already committed at poll time. Happy to adjust this PR to whatever design lands in #2927.

Signed-off-by: minleejae <55116920+minleejae@users.noreply.github.com>
@minleejae
minleejae requested a review from kriti-sc August 23, 2026 05:13
@github-actions

Copy link
Copy Markdown

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs.

If you need a review, please ensure CI is green and the PR is rebased on the latest master. Don't hesitate to ping the maintainers - either @core on Discord or by mentioning them directly here on the PR.

Thank you for your contribution!

@github-actions github-actions Bot added the S-stale Inactive issue or pull request label Aug 31, 2026
@hubcio

hubcio commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

closing in favour of #3954, which does the same status bind at sink.rs:740 plus the offset-commit half you scoped out of here - so a rejected batch isn't just reported, it's redelivered.

nothing wrong with this patch; #3950's repro is what got the runtime fix prioritized. flagged on #3954 that your process_messages unit tests should carry over - it only has integration coverage of that path.

#3950 stays open until #3954 merges. thanks for the repro. @kriti-sc

@github-actions github-actions Bot removed the S-stale Inactive issue or pull request label Sep 3, 2026
@hubcio hubcio closed this Sep 3, 2026
@github-actions github-actions Bot removed the S-waiting-on-review PR is waiting on a reviewer label Sep 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

connectors: runtime discards the sink consume FFI status, so failed batches are counted as processed

3 participants