Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
bdec557
Add topic multi-partition writer (write-by-key)
vgvoleg Jul 21, 2026
69879f6
Harden topic multiwriter repartition handling
vgvoleg Sep 2, 2026
048d08a
Add topic multiwriter SLO workload and select scenarios by diff
vgvoleg Sep 4, 2026
a786ed0
Cover the multiwriter patch and address review comments
vgvoleg Sep 4, 2026
2e28d60
Cover the remaining branches and narrow SLO scenario selection
vgvoleg Sep 4, 2026
030d2c8
Stop gating topic retry attempts on relative change
vgvoleg Sep 4, 2026
4f92461
Take the dedup cut from the retiring partition only
vgvoleg Sep 7, 2026
907b962
Fix multiwriter error propagation and split recovery
vgvoleg Sep 9, 2026
119d6f6
Fix Kafka-compatible partition routing
vgvoleg Sep 9, 2026
ee7de6e
Fix multiwriter recovery across partition merges
vgvoleg Sep 11, 2026
1843bc4
Use an available multiwriter implementation as SLO baseline
vgvoleg Sep 11, 2026
4ec8e80
Keep topic writer recovery active while flushing on close
vgvoleg Sep 11, 2026
a7df38b
Record SLO workload container exit status
vgvoleg Sep 11, 2026
b86d86a
Keep garbage collection enabled in SLO workloads
vgvoleg Sep 11, 2026
2656e84
Test multiwriter across explicit partition splits
vgvoleg Sep 11, 2026
b728532
Disable retry attempt metrics for topic SLO
vgvoleg Sep 14, 2026
c2ffe55
Fix topic writer shutdown after acknowledgement and close timeouts
vgvoleg Nov 16, 2026
7c83c11
Fix multiwriter admission during shutdown and repartition
vgvoleg Nov 18, 2026
a28465d
Document and test partial batch admission in topic multiwriter
vgvoleg Nov 18, 2026
e9e4d78
Run all six SLO scenarios with cyclic GC disabled
vgvoleg Nov 18, 2026
4d0ee61
Resolve SLO OpenTelemetry dependencies through the OTLP exporter
vgvoleg Nov 18, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 29 additions & 2 deletions .github/workflows/slo.yml
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,10 @@ jobs:
command: "--write-rps 200 --write-threads 8 --read-threads 8"
metrics_yaml_path: sdk-current/tests/slo/metrics-topic.yaml
thresholds_yaml_path: sdk-current/tests/slo/thresholds-topic.yaml
- name: sync-topic-multiwriter
command: "--write-rps 200 --write-threads 2 --keys-per-writer 16 --read-threads 8"
metrics_yaml_path: sdk-current/tests/slo/metrics-topic.yaml
thresholds_yaml_path: sdk-current/tests/slo/thresholds-topic.yaml

concurrency:
group: slo-${{ github.ref }}-${{ matrix.sdk.name }}
Expand Down Expand Up @@ -82,12 +86,26 @@ jobs:
- name: Determine baseline commit
id: baseline
working-directory: sdk-current
env:
SLO_SCENARIO: ${{ matrix.sdk.name }}
PR_HEAD_SHA: ${{ github.event.pull_request.head.sha }}
run: |
set -euo pipefail
BASELINE=$(git merge-base HEAD origin/main)
echo "sha=${BASELINE}" >> "$GITHUB_OUTPUT"

if git merge-base --is-ancestor "${BASELINE}" origin/main && \
if [ "${SLO_SCENARIO}" = "sync-topic-multiwriter" ] && \
! git grep -q -F 'def multiwriter(' "${BASELINE}" -- ydb/topic.py; then
# Before the API reaches main, compare against the preceding SDK implementation.
# Harness-only commits must not turn this into a comparison with identical SDK code.
LAST_IMPLEMENTATION=$(git log -1 --format=%H "${PR_HEAD_SHA}" -- \
ydb/_topic_writer ydb/topic.py ':(exclude)**/*_test.py')
BASELINE=$(git rev-parse "${LAST_IMPLEMENTATION}^")
if ! git grep -q -F 'def multiwriter(' "${BASELINE}" -- ydb/topic.py; then
echo "No earlier multiwriter implementation is available for comparison." >&2
exit 1
fi
BASELINE_REF="previous-multiwriter@${BASELINE:0:7}"
elif git merge-base --is-ancestor "${BASELINE}" origin/main && \
[ "$(git rev-parse origin/main)" = "${BASELINE}" ]; then
BASELINE_REF="main"
else
Expand All @@ -98,6 +116,7 @@ jobs:
BASELINE_REF="${BASELINE:0:7}"
fi
fi
echo "sha=${BASELINE}" >> "$GITHUB_OUTPUT"
echo "ref=${BASELINE_REF}" >> "$GITHUB_OUTPUT"

- name: Checkout baseline SDK version
Expand Down Expand Up @@ -136,6 +155,7 @@ jobs:
github_token: ${{ secrets.GITHUB_TOKEN }}
workload_name: ${{ matrix.sdk.name }}
workload_duration: "600"
fail_on_workload_error: "true"
workload_current_ref: ${{ github.head_ref || github.ref_name }}
workload_current_image: ydb-app-current
workload_current_command: ${{ matrix.sdk.command }}
Expand All @@ -149,3 +169,10 @@ jobs:
# Ignored by action versions without ydb-slo-action#57; self-activates
# once per-scenario thresholds land in the consumed tag.
thresholds_yaml_path: ${{ matrix.sdk.thresholds_yaml_path }}

- name: Inspect workload exit status
if: always()
run: |
docker inspect --format \
'{{.Name}} status={{.State.Status}} exit={{.State.ExitCode}} oom_killed={{.State.OOMKilled}} memory_limit={{.HostConfig.Memory}} error={{.State.Error}}' \
ydb-workload-current ydb-workload-baseline
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
* Fix multiwriter admission during close and partition changes, and allow retrying a single write with its original sequence number after a buffer timeout
* Fix topic writer shutdown after a close timeout and make multiwriter flush and close wait for delivery after an acknowledgement waiter is cancelled
* Add a topic multi-partition writer (`topic_client.multiwriter(...)`) that routes messages across partitions by their `key`, with Kafka-hash and key-range partition choosers, and transparently resends in-flight messages to child partitions on an auto-partition split (no loss, no duplicates); expose partition `key_range` on `describe_topic` results
* Fix automatic refresh of short-lived login tokens by using their JWT expiration time.

## 3.35.0 ##
Expand Down
3 changes: 2 additions & 1 deletion docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ version: "3.9"
services:
ydb:
image: ydbplatform/local-ydb:trunk
command: ["--enable-feature-flag", "enable_topic_split_merge"]
restart: always
ports:
- 2136:2136
Expand All @@ -11,4 +12,4 @@ services:
- ./ydb_certs:/ydb_certs
environment:
- YDB_ENABLE_COLUMN_TABLES=true
- YDB_USE_IN_MEMORY_PDISKS=true
- YDB_USE_IN_MEMORY_PDISKS=true
131 changes: 127 additions & 4 deletions docs/topic.rst
Original file line number Diff line number Diff line change
Expand Up @@ -170,6 +170,11 @@ Use ``topic_client.writer()`` as a context manager (recommended) or create it ma
finally:
writer.close()

With ``close(flush=True)``, the writer keeps reconnecting as needed until pending writes
finish. A transport disconnect during this wait does not stop recovery. If the close call
itself times out or is cancelled while waiting, the writer stops its background tasks and
closes the stream. A message whose acknowledgement was lost may already be on the server.


Writer Parameters
^^^^^^^^^^^^^^^^^
Expand Down Expand Up @@ -262,6 +267,125 @@ For high-throughput pipelines, buffer writes and gather futures:
raise f.exception()


Writing by Key (Multiple Partitions)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

A regular writer targets a single partition for its whole lifetime. To spread load across
all partitions of a topic from one logical writer — while keeping every message with the same
key on the same partition (so per-key ordering is preserved) — use ``topic_client.multiwriter()``.

Each message carries a ``key``; the writer hashes it and routes the message to the owning
partition, maintaining a separate underlying writer per partition. This is the client-side
companion to auto-partitioning.

**Synchronous:**

.. code-block:: python

with driver.topic_client.multiwriter("/local/my-topic") as writer:
writer.write(ydb.TopicWriterMessage(data="event", key="user-42"))
writer.write(ydb.TopicWriterMessage(data="event", key="user-7"))

**Asynchronous:**

.. code-block:: python

async with driver.topic_client.multiwriter("/local/my-topic") as writer:
await writer.write(ydb.TopicWriterMessage(data="event", key="user-42"))

**Partition choosers** decide how a key maps to a partition. By default the writer picks one
automatically after describing the topic — the key-range chooser for auto-partitioned topics,
the Kafka-hash chooser otherwise — so ``multiwriter(topic)`` works out of the box. You can also
set one explicitly:

* :class:`~ydb.TopicWriterPartitionByKeyBound` — hashes the key and selects the partition whose
server-side key range owns it. Mirrors YDB auto-partitioning, so a key lands where the server
expects it. Used automatically for auto-partitioned topics.
* :class:`~ydb.TopicWriterPartitionByKeyKafka` — Kafka-compatible routing for topics
with a fixed partition count. Hashes the UTF-8 key bytes with 32-bit MurmurHash2
and seed ``0x9747b28c``, then computes ``(hash & 0x7fffffff) % partitions_count``.
The result is an index into partition IDs sorted in ascending order. Compatibility
requires identical key bytes and partition IDs; changing the partition set can
route an existing key to a different partition. Missing and empty keys are hashed
as empty bytes; this differs from Kafka's handling of a missing key.

.. code-block:: python

writer = driver.topic_client.multiwriter(
"/local/my-topic",
partition_chooser=ydb.TopicWriterPartitionByKeyBound(),
producer_id_prefix="my-app", # each partition writer uses "<prefix>-<partition_id>"
)

The multi-writer accepts the same ``codec``, ``encoders``, ``auto_seqno``, ``auto_created_at``
and buffer-limit parameters as :meth:`writer`, and exposes ``write``, ``write_with_ack``,
``flush`` and ``close``. Its ``wait_init()`` method returns ``None`` once the topic has been
described and the partition set is known.

When passed a list, the multi-writer accepts messages one at a time in input order.
Each message's key selects its partition, and the buffer limits apply per partition.
If admission fails for one message, the call raises and skips the rest of the list.
For example, a timeout waiting for buffer space raises ``TopicWriterBufferFullError``.
The writer continues delivering messages it has already accepted. The exception does not
report how many were accepted. Retrying the whole list with automatic sequence numbers
can duplicate those messages.

With ``auto_seqno=False``, supply a positive ``seqno`` for every message. Numbers must strictly
increase across the whole multi-writer, including messages routed to different partitions.
Both automatic and manual sequence numbers are preserved on resend, including split and merge.
A single write rejected with ``TopicWriterBufferFullError`` can be retried with the same number
as long as it still exceeds the last accepted number.

Cancelling or timing out a call that is waiting for initialization does not cancel the shared
initialization. Other waiters and subsequent writes can still complete. Closing the writer
cancels unfinished initialization and waits for its task to stop. Each ``DescribeTopic`` RPC
has a separate 30-second timeout; a caller's shorter wait timeout does not shorten that RPC.

The multi-writer copies each message and its metadata when accepting a write. Later changes to
the original message do not change an accepted write or its retries. An automatically assigned
creation timestamp is also preserved across retries.

If an accepted write fails permanently, its acknowledgement future, ``flush()`` and
``close(flush=True)`` report the error, including when it happened before the flush or close
call. Closing still releases the underlying writers before raising the error. Use
``close(flush=False)`` to abandon pending writes without waiting for delivery.
Once closing begins, new writes raise ``TopicWriterClosedError``. Calling ``close(flush=False)``
also wakes a write waiting for buffer space. If the partition splits during that wait, an
open writer refreshes the route and retries the message.
Cancelling an individual acknowledgement future only ends that wait. Both ``flush()``
and ``close(flush=True)`` still wait for the message if it moves to a new partition.

.. note::

When an auto-partitioned partition is split (one into two) or merged (two into one), the
multi-writer re-describes the topic, routes subsequent keys to the new child partition(s), and
transparently resends the messages that were still in flight to the retired partition(s).

Which of those messages to resend is decided by asking the server how far the retired
partition's producer actually got, not by looking at the acknowledgements the client happened
to receive: a message can be persisted and its acknowledgement lost together with the session
that a split tears down. Anything at or below that point is reported as written instead of
being sent again, so a split produces neither loss nor duplicates, and per-key ordering is
preserved throughout.

Recovery also refreshes routing when a selected partition cannot initialize and follows
cascaded splits to their active leaf partitions. An incomplete set of child key ranges is
retried before updating routing. If recovery cannot find a complete topology or deliver an
accepted message within its retry limits, the affected messages fail explicitly and the
error is reported by ``flush()`` and ``close(flush=True)``.

When multiple parents share a descendant, recovery stops all affected parents and reads
each producer's cut before sending any of their messages. It filters each parent's tail
against that parent's cut, then sends the combined remainder in increasing sequence-number
order. New writes cannot overtake this transfer. If a destination producer's existing history
conflicts with a carried sequence number, the affected message fails explicitly instead of
being renumbered or reported as already written.

This client-side handling of merge does not enable server-side automatic merge. Its
availability depends on the target server. Recovery also follows intermediate inactive
partitions, including a merge followed by a split.


Writer Backpressure
^^^^^^^^^^^^^^^^^^^

Expand All @@ -281,10 +405,9 @@ A message is counted as occupying the buffer from the moment it is passed to ``w
until the server acknowledges it. Backpressure is active when **at least one** limit is set;
setting both means either limit can trigger a wait (OR semantics).

The limits are **soft**: ``write()`` blocks only if the buffer is *already* at or above the
limit when the call starts. Once unblocked, the entire batch is admitted regardless of its
size. This means callers that batch multiple messages in a single ``write()`` call will never
deadlock even when the batch is larger than the limit.
The single-partition writer uses **soft limits**. Its ``write()`` blocks only if the buffer is
*already* at or above the limit when the call starts. Once unblocked, the writer admits the
entire batch, even if the batch itself exceeds the limit.

**Blocking behavior (default)**

Expand Down
59 changes: 59 additions & 0 deletions examples/topic/multiwriter_example.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
"""Examples for the topic multi-partition writer (write-by-key).

The multi-writer routes each message to a partition by its ``key``, keeping every
message with the same key on the same partition (per-key ordering). By default it
picks a partition chooser automatically: the key-range chooser for
auto-partitioned topics, the Kafka-hash chooser otherwise.
"""

import asyncio

import ydb


def write_by_key_sync(db: ydb.Driver, topic_path: str):
with db.topic_client.multiwriter(topic_path, producer_id_prefix="my-app") as writer:
writer.write(ydb.TopicWriterMessage(data="event-a", key="user-42"))
writer.write(ydb.TopicWriterMessage(data="event-b", key="user-7"))
# messages with the same key always land on the same partition, in order
writer.write(ydb.TopicWriterMessage(data="event-c", key="user-42"))
writer.flush()


async def write_by_key_async(db: ydb.aio.Driver, topic_path: str):
async with db.topic_client.multiwriter(topic_path, producer_id_prefix="my-app") as writer:
await writer.write(ydb.TopicWriterMessage(data="event-a", key="user-42"))
# wait for the server ack of a specific message
await writer.write_with_ack(ydb.TopicWriterMessage(data="event-b", key="user-7"))


def write_by_key_with_explicit_chooser(db: ydb.Driver, topic_path: str):
# Force the Kafka-compatible hash chooser (murmur2(key) % partitions_count).
with db.topic_client.multiwriter(
topic_path,
partition_chooser=ydb.TopicWriterPartitionByKeyKafka(),
) as writer:
writer.write(ydb.TopicWriterMessage(data="event", key="user-42"))


def run_sync():
with ydb.Driver(
connection_string="grpc://localhost:2135?database=/local",
credentials=ydb.credentials.AnonymousCredentials(),
) as db:
db.wait(timeout=5, fail_fast=True)
write_by_key_sync(db, "/local/topic")


async def run_async():
async with ydb.aio.Driver(
connection_string="grpc://localhost:2135?database=/local",
credentials=ydb.credentials.AnonymousCredentials(),
) as db:
await db.wait(timeout=5, fail_fast=True)
await write_by_key_async(db, "/local/topic")


if __name__ == "__main__":
run_sync()
asyncio.run(run_async())
4 changes: 1 addition & 3 deletions tests/slo/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,7 @@
# `workload_current_command` from the action is appended to the `*-run` call
# as tuning flags (--read-rps, --write-rps, ...).
#
# Notes:
# - OpenTelemetry 1.39.x requires Python >= 3.9.
# - gcc/libc6-dev are needed to build hdrhistogram from sdist.
# gcc/libc6-dev are needed to build hdrhistogram from sdist.

FROM python:3.11-slim

Expand Down
22 changes: 21 additions & 1 deletion tests/slo/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,10 +24,13 @@ async (`ydb.aio`) path:
| `async-query` | Query service | async |
| `sync-topic` | Topic service | sync |
| `async-topic` | Topic service | async |
| `sync-topic-multiwriter` | Topic service, writes by key | sync |

> The `--async` CLI flag is kept as a manual override for `*-run` commands.
> The bare `topic` label is still accepted as an alias for `sync-topic`.

Each SLO workflow run executes all six scenarios listed above.

### Usage:

Each workload type has 3 commands:
Expand Down Expand Up @@ -267,7 +270,24 @@ When running `topic-run` (`sync-topic` / `async-topic`), the program creates `re
- a **backward** seqno (already seen) is a **duplicate** — reconnect redelivery; with producer-id dedup it should stay near zero (informational);
- **end-to-end latency** is `read_ts − write_ts` for the first delivery of each message (writer and reader share the process, so the timestamps are comparable).

Each message carries `writer_id:seqno:write_ts_ns:` followed by padding to the configured size. Topics are scoped per ref so the current and baseline containers (same cluster, run in parallel) don't share a topic.
Each message carries `writer_id:seqno:write_ts_ns:` followed by padding to the configured size. Topics are scoped per workload and per ref, so neither two workloads nor the current and baseline containers (same cluster, run in parallel) share a topic — sharing one would mix their producers into each other's delivery and ordering accounting.

### Topic multi-writer workload

`sync-topic-multiwriter` runs the same read side and the same accounting, but writes through `topic_client.multiwriter(...)`: each message carries a `key` and the writer picks the partition itself. That covers what the pinned-writer workload cannot — key routing, the per-partition sub-writer pool, and the recovery of both when nodes disappear under chaos.

- `writeJob` — each thread owns one multi-writer and cycles through `--keys-per-writer` distinct keys. One payload stream per **key** (not per partition), because a key is what the writer keeps ordered, and two keys may share a partition. Use fewer write threads than the pinned workload: every thread opens a sub-writer per partition it touches.
- `readJob` — unchanged. A forward gap is still loss and a backward seqno is still a duplicate; the streams it validates are keys rather than partition-pinned producers.

Enabled by `--use-multiwriter` on `topic-run`; the label sets it automatically. There is no async variant yet — the sync facade drives the same async implementation underneath, so the code under test is the same.

When the merge-base SDK does not provide `multiwriter`, this scenario compares against the
parent of the latest SDK implementation commit in the PR. Changes limited to the workload or
its tests do not advance that baseline. The report labels it `previous-multiwriter@<sha>`.
If no earlier implementation exists, baseline selection fails explicitly. An SDK without the
API also fails workload initialization instead of running reader threads with no writers.

> Local re-runs against an existing topic will report duplicates: per-key seqno counters restart at 1 with the process, while the reader still expects the sequence left by the previous run. Recreate the topic (`topic-cleanup` + `topic-create`) between local runs. CI is unaffected — every run gets its own topic.

## Collected metrics
- `oks` - amount of OK requests
Expand Down
Loading
Loading