KAFKA-20772 (1/n): Add DLQ system tests for single-partition share group scenarios by sjhajharia · Pull Request #22786 · apache/kafka

sjhajharia · 2026-07-08T05:20:55Z

This adds the first system-test coverage for share group dead-letter
queues (KIP-1191, part of KAFKA-19469), plus the client-side tooling
changes needed to drive it — there was previously no way for any
ducktape test client to explicitly REJECT or RELEASE a share-consumed
record; VerifiableShareConsumer only ever implicitly accepted.

VerifiableShareConsumer.java

New --ack-pattern CLI flag: a comma-separated cycle of
accept|release|reject|renew, applied per record via (offset % pattern.length) — keyed off the immutable record offset so the
assignment is stable across redeliveries.
When set, the consumer is switched to
share.acknowledgement.mode=explicit and requires
--acknowledgement-mode sync|async (rejects auto at arg-parse time,
since explicit per-record acks still need a commit to ship).
offsets_acknowledged events now optionally include an
ackTypeCounts breakdown ({"ACCEPT": n, "RELEASE": n, ...}), omitted
entirely when --ack-pattern isn't used — no change to the existing
JSON schema for any current caller.

verifiable_share_consumer.py (ducktape service)

Plumbs ack_pattern through to the CLI, and adds
total_accepted()/total_released()/total_rejected()/total_renewed()
accessors backed by the new ackTypeCounts data.

kafka.py

set_share_group_dlq_config(group, topic_name, copy_record_enable) —
sets the GroupConfig DLQ configs via kafka-configs.sh.
set_share_group_delivery_count_limit(group, limit) — sets
share.delivery.count.limit (the per-group override).
New share_version constructor param to bootstrap a cluster with
--feature share.version=<n>. Since DLQ (share.version=2) is above
ShareVersion.LATEST_PRODUCTION, this also flips on
unstable.feature.versions.enable in the broker properties.

New: tests/kafkatest/tests/client/share_consumer_dlq_test.py
Three tests on a real 3-broker isolated-KRaft cluster, verifying actual
DLQ topic content (not just consumer-side counters):

test_single_partition_dlq_reject — every record REJECTed, DLQ'd
exactly once, values null (copy-record disabled, the default).
test_single_partition_dlq_release — every record RELEASEd
repeatedly; once delivery count exceeds a lowered
share.delivery.count.limit, it lands in the DLQ.
test_single_partition_dlq_mixed — records cycled
reject/release/accept by offset % 3, copy-record enabled; asserts the
DLQ'd record set and values exactly match the subset of produced values
that should have been rejected/released (cross-checked against
VerifiableProducer.acked_values, not just "value is non-null").

All three verified passing end-to-end against a live ducker cluster
(tests/docker/run_tests.sh).

> TC_PATHS="tests/kafkatest/tests/client/share_consumer_dlq_test.py"
bash tests/docker/run_tests.sh

================================================================================
SESSION REPORT (ALL TESTS)
ducktape version: 0.14.0
session_id:       2026-07-08--005
run time:         5 minutes 27.339 seconds
tests run:        6
passed:           6
flaky:            0
failed:           0
ignored:          0
================================================================================
test_id:
kafkatest.tests.client.share_consumer_dlq_test.ShareConsumerDLQTest.test_single_partition_dlq_mixed.metadata_quorum=COMBINED_KRAFT
status:     PASS
run time:   45.786 seconds
--------------------------------------------------------------------------------
test_id:
kafkatest.tests.client.share_consumer_dlq_test.ShareConsumerDLQTest.test_single_partition_dlq_mixed.metadata_quorum=ISOLATED_KRAFT
status:     PASS
run time:   1 minute 5.269 seconds
--------------------------------------------------------------------------------
test_id:
kafkatest.tests.client.share_consumer_dlq_test.ShareConsumerDLQTest.test_single_partition_dlq_reject.metadata_quorum=COMBINED_KRAFT
status:     PASS
run time:   43.421 seconds
--------------------------------------------------------------------------------
test_id:
kafkatest.tests.client.share_consumer_dlq_test.ShareConsumerDLQTest.test_single_partition_dlq_reject.metadata_quorum=ISOLATED_KRAFT
status:     PASS
run time:   1 minute 2.872 seconds
--------------------------------------------------------------------------------
test_id:
kafkatest.tests.client.share_consumer_dlq_test.ShareConsumerDLQTest.test_single_partition_dlq_release.metadata_quorum=COMBINED_KRAFT
status:     PASS
run time:   43.429 seconds
--------------------------------------------------------------------------------
test_id:
kafkatest.tests.client.share_consumer_dlq_test.ShareConsumerDLQTest.test_single_partition_dlq_release.metadata_quorum=ISOLATED_KRAFT
status:     PASS
run time:   1 minute 5.794 seconds
--------------------------------------------------------------------------------

Reviewers: Sushant Mahajan smahajan@confluent.io, Chia-Ping Tsai
chia7712@gmail.com

smjn

Please add combined_kraft as well in the matrix

Copilot

Pull request overview

This PR adds end-to-end ducktape system test coverage for share group dead-letter queues (DLQ, KIP-1191) in a single-partition scenario, along with the supporting test harness/tooling needed to drive explicit per-record share acknowledgements and to bootstrap clusters with share.version=2.

Changes:

Extend VerifiableShareConsumer with an --ack-pattern flag for per-record explicit ack cycling and optional ackTypeCounts reporting in offsets_acknowledged events.
Enhance ducktape’s verifiable_share_consumer.py service to pass ack_pattern through and expose per-ack-type counters.
Add KafkaService helpers to configure share group DLQ settings and to bootstrap share.version, plus new DLQ-focused system tests.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File	Description
tools/src/main/java/org/apache/kafka/tools/VerifiableShareConsumer.java	Adds `--ack-pattern` explicit-ack behavior and optional ack-type breakdown in emitted events.
tests/kafkatest/services/verifiable_share_consumer.py	Plumbs `ack_pattern` into the CLI and aggregates `ackTypeCounts` into helper counters.
tests/kafkatest/services/kafka/kafka.py	Adds share-version bootstrapping and new helpers to configure share group DLQ + delivery-count-limit.
tests/kafkatest/tests/client/share_consumer_dlq_test.py	New ducktape system tests validating DLQ topic contents for reject/release/mixed patterns.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+        List<AcknowledgeType> ackPattern = parseAckPattern(res.getString("ackPattern"));
+        if (!ackPattern.isEmpty()) {
+            if (acknowledgementMode == AcknowledgementMode.AUTO) {
+                throw new ArgumentParserException(
+                    "--ack-pattern requires --acknowledgement-mode to be 'sync' or 'async', not 'auto'", parser);
+            }
+        }


smjn · 2026-07-09T05:50:20Z

Mostly good, one AI comment seems relevant and could result in flakiness. Please look into it.

smjn

AI comment regarding unchecked map creation overriding previous values could result in flakiness. Please address

chia7712

@sjhajharia thanks for this patch!

chia7712 · 2026-07-09T18:20:26Z

                partitionOffsets.add(record.offset());
+
+                if (!ackPattern.isEmpty()) {
+                    AcknowledgeType type = ackPattern.get((int) (record.offset() % ackPattern.size()));


What happens if ackPattern contains RENEW?

Thanks for the review. I have added the support for RENEW as well.

chia7712 · 2026-07-09T18:23:13Z

            .metavar("ACKNOWLEDGEMENT-MODE")
            .help("Acknowledgement mode for the share consumers (must be either 'auto', 'sync' or 'async')");

+        parser.addArgument("--ack-pattern")


Should we add those new options to KIP-1191? Or is it fine since this is a test class?

I think given its an internal class which we use only for our system tests and do not expose it to customers, we shall be good to go ahead in this state. Still, I am open to the other way round too if you think necessary.

Add DLQ Single Partition Tests

ef51e45

github-actions Bot added triage PRs from the community tools labels Jul 8, 2026

smjn requested a review from Copilot July 8, 2026 06:45

Copilot started reviewing on behalf of smjn July 8, 2026 06:46 View session

smjn added KIP-932 Queues for Kafka ci-approved and removed triage PRs from the community labels Jul 8, 2026

smjn suggested changes Jul 8, 2026

View reviewed changes

Copilot AI reviewed Jul 8, 2026

View reviewed changes

Add combined_kraft mode

e5927fb

sjhajharia requested a review from smjn July 8, 2026 12:17

smjn suggested changes Jul 9, 2026

View reviewed changes

Address comment

be8b985

sjhajharia requested a review from smjn July 9, 2026 06:14

Empty Commit

74361bd

chia7712 reviewed Jul 9, 2026

View reviewed changes

Address comment-1

4399414

sjhajharia requested a review from chia7712 July 10, 2026 04:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

KAFKA-20772 (1/n): Add DLQ system tests for single-partition share group scenarios#22786

KAFKA-20772 (1/n): Add DLQ system tests for single-partition share group scenarios#22786
sjhajharia wants to merge 5 commits into
apache:trunkfrom
sjhajharia:KAFKA-20772

sjhajharia commented Jul 8, 2026 •

edited by github-actions Bot

Loading

Uh oh!

smjn left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

smjn commented Jul 9, 2026

Uh oh!

smjn left a comment

Uh oh!

chia7712 left a comment

Uh oh!

chia7712 Jul 9, 2026

Uh oh!

sjhajharia Jul 10, 2026

Uh oh!

Uh oh!

chia7712 Jul 9, 2026

Uh oh!

sjhajharia Jul 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

sjhajharia commented Jul 8, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

smjn left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

smjn commented Jul 9, 2026

Uh oh!

smjn left a comment

Choose a reason for hiding this comment

Uh oh!

chia7712 left a comment

Choose a reason for hiding this comment

Uh oh!

chia7712 Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

sjhajharia Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

chia7712 Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

sjhajharia Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

sjhajharia commented Jul 8, 2026 •

edited by github-actions Bot

Loading