Conversation
dajac
force-pushed
the
uniform2-util
branch
from
September 18, 2026 07:57
12cfd9a to
cde85d8
Compare
dajac
force-pushed
the
uniform2-core
branch
from
September 18, 2026 07:57
599a446 to
dd32006
Compare
dajac
force-pushed
the
uniform2-util
branch
from
September 18, 2026 13:10
cde85d8 to
f5d91e8
Compare
dajac
force-pushed
the
uniform2-core
branch
from
September 18, 2026 13:11
3b6ac06 to
fc75a8e
Compare
The uniform assignor balances the number of partitions per member but does not look at topics: a member typically receives whole topics, so the partitions of a busy topic can end up on a single member. The uniform2 assignor spreads the partitions of every topic evenly across its subscribers, and keeps the other properties: the total number of partitions per member is balanced, within one for a single subscription and up to a local optimum otherwise, partitions only move when required, a valid assignment is returned as is down to the same partition set instances, and the result does not depend on iteration orders. The same algorithm serves homogeneous and heterogeneous subscriptions. The algorithm decides allocations before partition ids. Every subscriber of a topic gets its base partitions, p / n, and p % n subscribers get one extra partition, which settles the spread; the balance only depends on who gets the extra partitions, which three phases decide, claims, fill and even out, over cohorts of members sharing a subscription; the partition ids are then chosen per topic, members keeping what they own up to their allocation. The class Javadoc of AssignmentBuilder documents the algorithm and its vocabulary with worked examples. The code follows the phases: GroupModel normalizes the input, AllocationBuilder runs the three phases on Allocations and Loads, PartitionAssigner assigns the partition ids and AssignmentResult builds the group assignment, with the primitive collections of the util package. The assignor is not registered as a built-in yet: a coordinator enables it by naming its class in group.consumer.assignors. Rack awareness follows in its own change. The consumer assignor benchmark gains the uniform2 assignor, and AssignorHelpers.newHashMap becomes public for the assignment result. Unit tests cover every class with hand verified expectations, and the end to end scenarios: fresh groups, members joining and leaving one at a time or in bulk, partitions added, subscriptions changing, stale partitions, fixed points, and members subscribed through TopicIds as the coordinator does. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…phase At the local optimum of a group with different subscriptions, the members of the richer cohorts stay two or more above the lightest members of the poorest cohort, so every round of the even out phase pops thousands of them as givers and checks each of their topics for an eligible receiver, although nothing moves. Every check scanned the load orders of the cohorts of the topic from the start, past every member already getting an extra partition of it, up to thousands of them per topic. The best receiver of a topic only depends on the loads of the members of its cohorts and on its receivers, so it is now memoized until one of them changes. A version counts the changes: a load change stamps the cohort of the member and a change of receivers stamps the topic, and a memoized receiver is current when its version is at least those of the topic and of its cohorts. A check costs the cohorts of the topic instead of its receivers, and gives exactly the receiver the scan would find. Topics with fewer than 64 subscribers are not memoized: their scans cost no more than the check, and on a group of 20 members joining over 10000 topics, where every move stales most memos, the checks alone made the assignment 10 to 20% slower. On a stable group of 10000 members nested over 10 topics of 100000 partitions, where nothing moves, the assignment takes 43 ms before and 5.2 ms after; a member joining it goes from 80 to 7.4 ms and a fresh assignment of it from 83 to 5.8 ms. A join of 20 members over 10000 topics is unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
dajac
force-pushed
the
uniform2-util
branch
from
September 18, 2026 14:26
f5d91e8 to
c23bdcf
Compare
dajac
force-pushed
the
uniform2-core
branch
from
September 18, 2026 14:27
fc75a8e to
a2f86c9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is the second of three changes adding the uniform2 consumer group assignor: the assignor itself, without rack awareness, which follows in the third change.
The uniform assignor balances the number of partitions per member but does not look at topics: a member typically receives whole topics, so the partitions of a busy topic can end up on a single member. The uniform2 assignor spreads the partitions of every topic evenly across its subscribers and keeps the other properties: the total number of partitions per member is balanced, within one for a single subscription and up to a local optimum otherwise; partitions only move when required; a valid assignment is returned as is, down to the same partition set instances; and the result does not depend on iteration orders. The same algorithm serves homogeneous and heterogeneous subscriptions.
The algorithm decides allocations before partition ids. Every subscriber of a topic gets its base partitions,
p / n, andp % nsubscribers get one extra partition, which settles the spread. The balance only depends on who gets the extra partitions, which three phases decide over cohorts of members sharing a subscription: claims, fill and even out. The partition ids are then chosen one topic at a time, members keeping what they own up to their allocation. The class Javadoc ofAssignmentBuilderdocuments the algorithm and its vocabulary with worked examples, and the code follows the phases:GroupModelnormalizes the input,AllocationBuilderruns the three phases onAllocationsandLoads,PartitionAssignerassigns the partition ids andAssignmentResultbuilds the group assignment.The assignor is not registered as a built-in yet: a coordinator enables it by naming its class,
org.apache.kafka.coordinator.group.assignor.Uniform2Assignor, ingroup.consumer.assignors. The consumer assignor benchmark gains the uniform2 assignor, andAssignorHelpers.newHashMapbecomes public for the assignment result.Unit tests cover every class with hand verified expectations, and the end to end scenarios: fresh groups, members joining and leaving one at a time or in bulk, partitions added, subscriptions changing, stale partitions, fixed points, and members subscribed through
TopicIdsas the coordinator does.