Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
99 changes: 99 additions & 0 deletions .github/workflows/slo-2dc.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
name: Hosted 2DC SLO

on:
pull_request:
paths:
- .github/workflows/slo-2dc.yml
- tests/slo/**
- ydb/driver.py
- ydb/resolver.py
- ydb/pool.py
- ydb/aio/driver.py
- ydb/aio/pool.py
workflow_dispatch:
inputs:
server_image:
description: YDB server image to qualify (resolved to a digest)
default: cr.yandex/crpb3d0up7s6a7iuofo9/ydb-appteam-ci/local-ydb:trunk
required: true

permissions:
contents: read

jobs:
bridge:
runs-on: ubuntu-24.04
timeout-minutes: 30
concurrency:
group: hosted-2dc-${{ github.ref }}
cancel-in-progress: true
env:
WORKLOAD_MEMORY_LIMIT: 1G
WORKLOAD_CPU_LIMIT: '1.00'
WORKLOAD_CURRENT_IMAGE: py-slo-bridge:current
REQUESTED_IMAGE: ${{ inputs.server_image || 'cr.yandex/crpb3d0up7s6a7iuofo9/ydb-appteam-ci/local-ydb@sha256:bce9caa6a1a6aac9b513510b46f61c4d4b58c1f88d937bcb54c20dc1bda6c8d4' }}
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v4
with:
repository: ydb-platform/ydb-slo-action
ref: 54c3c03de1ce8c6a2d8b0828b328bd5db82ef2bb
path: .slo-action
- uses: actions/setup-python@v5
with:
python-version: '3.12'
- name: Install the Compose version required by the action
run: |
set -euo pipefail
mkdir -p "$HOME/.docker/cli-plugins"
curl -fsSL --retry 3 \
https://github.com/docker/compose/releases/download/v2.40.3/docker-compose-linux-x86_64 \
-o "$HOME/.docker/cli-plugins/docker-compose"
chmod +x "$HOME/.docker/cli-plugins/docker-compose"
docker compose version
docker compose up --help | grep -F -- --quiet-build
- name: Record runner and resolve server image
run: |
set -euo pipefail
mkdir -p bridge-artifacts
free -h
df -h
docker version
docker compose version
docker pull --platform linux/amd64 "$REQUESTED_IMAGE"
docker image inspect "$REQUESTED_IMAGE" > bridge-artifacts/server-image.json
SERVER_IMAGE=$(docker image inspect --format '{{index .RepoDigests 0}}' "$REQUESTED_IMAGE")
printf '%s\n' "$SERVER_IMAGE" > bridge-artifacts/server-image.txt
python tests/slo/bridge/prepare.py .slo-action "$SERVER_IMAGE"
- name: Build current Python SDK workload
run: docker build -f tests/slo/Dockerfile -t py-slo-bridge:current .
- name: Run all five Bridge scenarios
id: slo
uses: ./.slo-action/init
timeout-minutes: 20
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
workload_name: sync-query
workload_duration: '600'
workload_current_ref: current
workload_current_image: py-slo-bridge:current
workload_current_command: >-
--read-rps 50 --write-rps 10 --read-threads 2 --write-threads 2
--read-timeout 30000 --write-timeout 30000
--prefer-primary-pile
bridge_mode: true
disable_compose_profiles: extra-nodes
fail_on_workload_error: true
- name: Verify scenario coverage and workload recovery
if: always()
timeout-minutes: 3
env:
ACTION_OUTCOME: ${{ steps.slo.outcome }}
run: python tests/slo/bridge/verify.py
- name: Upload qualification evidence
if: always()
uses: actions/upload-artifact@v4
with:
name: python-hosted-2dc
path: bridge-artifacts/
retention-days: 14
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
* Add opt-in PRIMARY/PROMOTED Bridge pile preference with synchronized-pile fallback for sync and async drivers
* Fix automatic refresh of short-lived login tokens by using their JWT expiration time.

## 3.35.0 ##
Expand Down
34 changes: 34 additions & 0 deletions docs/driver.rst
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,9 @@ DriverConfig
* - ``discovery_request_timeout``
- ``10``
- Timeout in seconds for each discovery request.
* - ``prefer_primary_pile``
- ``False``
- Prefer PRIMARY/PROMOTED Bridge endpoints with synchronized-pile fallback; no effect without Bridge metadata.
* - ``table_client_settings``
- ``None``
- :class:`~ydb.TableClientSettings` instance to configure the Table service client.
Expand All @@ -62,6 +65,37 @@ DriverConfig
- gRPC-level compression (e.g. ``grpc.Compression.Gzip``).


Preferring the primary Bridge pile
----------------------------------------

Set ``prefer_primary_pile`` on the ``DriverConfig`` passed to the sync or async driver.

.. code-block:: python

config = ydb.DriverConfig(
endpoint="grpc://discovery.example:2135",
database="/Root/mydb",
prefer_primary_pile=True,
)
driver = ydb.Driver(config)

While discovery reports Bridge pile states, the driver tries PRIMARY and PROMOTED
endpoints first, followed by SYNCHRONIZED endpoints, then endpoints in other or
unknown states. This order applies to connection attempts and to unbound requests
for either value of ``use_all_nodes``. Endpoints within a tier have equal priority
regardless of their datacenter. The driver updates priorities when discovery reports a role change.
When a discovery response contains no pile states, the driver resumes its
configured balancing policy for both existing and new connections.
The option controls client-side routing and is disabled by default.

If the connection cache is full and none of its connections lead to a PRIMARY
or PROMOTED pile, discovery opens a connection to a preferred endpoint before
closing one lower-priority connection. Sessions using that connection can be
interrupted. Once the cache has a preferred connection, additional endpoints
can join as slots become available. Existing sessions keep their node affinity
while their connection is available.


Creating a Driver
-----------------

Expand Down
206 changes: 206 additions & 0 deletions tests/aio/test_discovery_detect_local_dc.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
# -*- coding: utf-8 -*-
import asyncio
import pytest
from unittest.mock import MagicMock, patch, AsyncMock
from ydb import driver, connection
from ydb import _apis, resolver
from ydb.aio import pool, _utilities


Expand Down Expand Up @@ -141,3 +143,207 @@ def mock_init(self, endpoint, driver_config, endpoint_options=None):
await discovery.execute_discovery()

assert detect_mock.call_count == 0, "detect_local_dc should NOT be called when use_all_nodes=True"


class PrimaryConnection:
failed = set()
created = []

def __init__(self, endpoint, driver_config, endpoint_options=None):
self.endpoint = endpoint
self.node_id = endpoint_options.node_id
self.callbacks = []
self.closed = False
self.created.append(self)

async def connection_ready(self, ready_timeout):
if self.endpoint in self.failed:
raise asyncio.TimeoutError()

def add_cleanup_callback(self, callback):
self.callbacks.append(callback)

async def close(self, grace=30):
self.closed = True
for callback in self.callbacks:
callback(self)


def primary_result(states, nodes=((1, "A"), (2, "B"))):
message = _apis.ydb_discovery.ListEndpointsResult(self_location="local")
for node_id, pile in nodes:
message.endpoints.add(
address=f"node{node_id}",
port=2135,
node_id=node_id,
location="local" if pile == "A" else "remote",
bridge_pile_name=pile,
)
for name, state in states.items():
message.pile_states.add(pile_name=name, state=state)
response = _apis.ydb_discovery.ListEndpointsResponse()
response.operation.ready = True
response.operation.status = 400000
response.operation.result.Pack(message)
return resolver.DiscoveryResult.from_response(None, response, use_all_nodes=True)


def primary_discovery(monkeypatch, result, enabled=True, use_all_nodes=True, failed=()):
PrimaryConnection.failed, PrimaryConnection.created = set(failed), []
monkeypatch.setattr(pool, "Connection", PrimaryConnection)
config = driver.DriverConfig("grpc://test:2135", "/local", use_all_nodes=use_all_nodes, prefer_primary_pile=enabled)
discovery = pool.Discovery(pool.ConnectionsCache(use_all_nodes), config)
discovery._resolver.resolve = AsyncMock(return_value=result)
return discovery


@pytest.mark.asyncio
async def test_primary_role_change_and_affinity(monkeypatch):
discovery = primary_discovery(monkeypatch, primary_result({"A": 1, "B": 3}))
assert await discovery.execute_discovery()
assert {(await discovery._cache.get()).node_id for _ in range(10)} == {1}
assert (await discovery._cache.get(connection.EndpointKey("node2:2135", 2))).node_id == 2
discovery._resolver.resolve.return_value = primary_result({"A": 3, "B": 1})
assert await discovery.execute_discovery()
assert {(await discovery._cache.get()).node_id for _ in range(10)} == {2}
assert len(PrimaryConnection.created) == 2
assert not any(conn.closed for conn in PrimaryConnection.created)
assert (await discovery._cache.get(connection.EndpointKey("node1:2135", 1))).node_id == 1
await (await discovery._cache.get()).close()
assert (await discovery._cache.get()).node_id == 1


@pytest.mark.asyncio
async def test_primary_off_and_ordinary_cluster_locality(monkeypatch):
discovery = primary_discovery(monkeypatch, primary_result({"A": 1, "B": 3}), enabled=False)
assert await discovery.execute_discovery()
assert {(await discovery._cache.get()).node_id for _ in range(10)} == {1, 2}
discovery = primary_discovery(monkeypatch, primary_result({}), use_all_nodes=False)
assert await discovery.execute_discovery()
assert {(await discovery._cache.get()).node_id for _ in range(10)} == {1}
discovery._resolver.resolve.return_value = primary_result({"A": 3, "B": 1})
assert await discovery.execute_discovery()
assert (await discovery._cache.get()).node_id == 2
discovery._resolver.resolve.return_value = primary_result({})
assert await discovery.execute_discovery()
assert (await discovery._cache.get()).node_id == 1


@pytest.mark.asyncio
@pytest.mark.parametrize("use_all_nodes,detect_local_dc", [(True, True), (False, False), (False, True)])
async def test_empty_pile_states_restore_configured_balancing(monkeypatch, use_all_nodes, detect_local_dc):
discovery = primary_discovery(monkeypatch, primary_result({"A": 1, "B": 3}), use_all_nodes=use_all_nodes)
discovery._driver_config.detect_local_dc = detect_local_dc
detector = AsyncMock(return_value="remote")
monkeypatch.setattr(_utilities, "detect_local_dc", detector)
assert await discovery.execute_discovery()
assert {(await discovery._cache.get()).node_id for _ in range(2)} == {1}
detector.assert_not_awaited()

ordinary = primary_result({})
ordinary.self_location = "local" if detect_local_dc else "remote"
discovery._resolver.resolve.return_value = ordinary
assert await discovery.execute_discovery()
assert {(await discovery._cache.get()).node_id for _ in range(2)} == ({1, 2} if use_all_nodes else {2})
assert len(PrimaryConnection.created) == 2 and not any(conn.closed for conn in PrimaryConnection.created)


@pytest.mark.asyncio
async def test_primary_promoted_and_unknown_states(monkeypatch):
discovery = primary_discovery(
monkeypatch,
primary_result({"A": 1, "B": 2, "C": 3, "D": 99}, ((1, "A"), (2, "B"), (3, "C"), (4, "D"), (5, "missing"))),
)
assert await discovery.execute_discovery()
assert {(await discovery._cache.get()).node_id for _ in range(10)} == {1, 2}
await (await discovery._cache.get(connection.EndpointKey("node1:2135", 1))).close()
await (await discovery._cache.get(connection.EndpointKey("node2:2135", 2))).close()
assert (await discovery._cache.get()).node_id == 3
await (await discovery._cache.get()).close()
assert {(await discovery._cache.get()).node_id for _ in range(10)} == {4, 5}


@pytest.mark.asyncio
async def test_primary_full_cache_admission_and_failed_connection(monkeypatch):
old = tuple((i, "A") for i in range(1, 10))
discovery = primary_discovery(monkeypatch, primary_result({"A": 1}, old))
assert await discovery.execute_discovery() and discovery._cache.size == 9
discovery._resolver.resolve.return_value = primary_result({"A": 3, "B": 1}, old + ((10, "B"),))
PrimaryConnection.failed = {"node10:2135"}
assert await discovery.execute_discovery()
assert discovery._cache.size == 9
assert not any(conn.closed for conn in PrimaryConnection.created if conn.node_id != 10)
PrimaryConnection.failed.clear()
assert await discovery.execute_discovery()
assert (await discovery._cache.get()).node_id == 10
assert discovery._cache.size == 9
created_count = len(PrimaryConnection.created)
assert await discovery.execute_discovery()
assert len(PrimaryConnection.created) == created_count


@pytest.mark.asyncio
async def test_full_cache_evicts_only_for_first_preferred_connection(monkeypatch):
old = tuple((i, "A") for i in range(1, 10))
nodes = old + ((10, "B"), (11, "B"))
discovery = primary_discovery(monkeypatch, primary_result({"A": 1}, old))
assert await discovery.execute_discovery()

discovery._resolver.resolve.return_value = primary_result({"A": 1, "B": 3}, nodes)
assert await discovery.execute_discovery()
assert len(PrimaryConnection.created) == 9 and not any(conn.closed for conn in PrimaryConnection.created)

discovery._resolver.resolve.return_value = primary_result({"A": 3, "B": 2}, nodes)
assert await discovery.execute_discovery()
preferred = await discovery._cache.get()
assert preferred.node_id in (10, 11)
assert await discovery.execute_discovery()
assert await discovery._cache.get() is preferred
assert len(PrimaryConnection.created) == 10
assert sum(conn.closed for conn in PrimaryConnection.created) == 1


@pytest.mark.asyncio
async def test_primary_alias_retirement_keeps_node_affinity():
cache = pool.ConnectionsCache()
options = connection.EndpointOptions(node_id=1)
old = PrimaryConnection("old:2135", None, options)
new = PrimaryConnection("new:2135", None, options)
cache.add(old)
cache.add(new)
await old.close()
assert await cache.get(connection.EndpointKey(None, 1)) is new


@pytest.mark.asyncio
async def test_primary_pile_preference_is_config_only(monkeypatch):
from ydb import topic
from ydb.aio import driver as aio_driver, table, scheme, coordination

seen = []
monkeypatch.setattr(pool.ConnectionPool, "__init__", lambda self, config: seen.append(config))
for module, name in (
(topic, "TopicClientAsyncIO"),
(table, "TableClient"),
(scheme, "SchemeClient"),
(coordination, "CoordinationClient"),
):
monkeypatch.setattr(module, name, MagicMock())
config = aio_driver.DriverConfig("grpc://test:2135", "/local", prefer_primary_pile=True)
aio_driver.Driver(config)
assert seen[-1].prefer_primary_pile
with pytest.raises(TypeError, match="DriverConfig"):
aio_driver.Driver(config, prefer_primary_pile=False)
with pytest.raises(TypeError, match="DriverConfig"):
aio_driver.Driver(endpoint="grpc://test:2135", database="/local", prefer_primary_pile=True)
assert config.prefer_primary_pile


@pytest.mark.asyncio
async def test_primary_discovery_cancellation_closes_candidate(monkeypatch):
discovery = primary_discovery(monkeypatch, primary_result({"A": 1, "B": 3}))
monkeypatch.setattr(PrimaryConnection, "connection_ready", AsyncMock(side_effect=asyncio.CancelledError))
with pytest.raises(asyncio.CancelledError):
await discovery.execute_discovery()
assert len(PrimaryConnection.created) == 1
assert PrimaryConnection.created[0].closed
16 changes: 16 additions & 0 deletions tests/slo/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,21 @@
# SLO workload

## GitHub-hosted Bridge SLO

`.github/workflows/slo-2dc.yml` runs the standard five Bridge scenarios from the
pinned SLO Action v2 on `ubuntu-24.04`. The ordinary `sync-query` workload runs for
10 minutes with `--prefer-primary-pile` enabled.

The runner uses two storage and two database nodes, `erasure=none` and SectorMap disks.
The adapter bounds resources and cold-start readiness time, restricts bootstrap DNS
to active compute nodes, and sets a 30-second gap between fault scenarios to fit the
10-minute workload. The upstream scenario scripts and SDK discovery intervals are unchanged.

The job requires all five scenarios to complete, successful recent reads and writes,
and no workload failure, OOM or unexpected container restart. Transient operation
errors are recorded. Server image metadata, workload and chaos logs, final container
states and counters are retained in the `python-hosted-2dc` artifact for 14 days.

SLO is the type of test where app based on ydb-sdk is tested against falling YDB cluster nodes, tablets, network
(that is possible situations for distributed DBs with hundreds of nodes)

Expand Down
1 change: 1 addition & 0 deletions tests/slo/bridge/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
__pycache__/
Loading
Loading