Skip to content

refactor: automate SchemeShard path reference ownership - #46343

Closed
Enjection wants to merge 23 commits into
ydb-platform:mainfrom
Enjection:draft/schemeshard-tpathref
Closed

Enjection wants to merge 23 commits into
ydb-platform:mainfrom
Enjection:draft/schemeshard-tpathref

Conversation

@Enjection

@Enjection Enjection commented Jul 13, 2026 •

Copy link
Copy Markdown
Member

YDBBUGS-452 · YDBBUGS-585

Changelog entry

Internal SchemeShard path-reference ownership and cleanup refactoring.

Description for reviewers

Automate path database-reference ownership for transactions, publications, and type-info map entries; retain restart checks and cleanup fixes. Shard reference accounting remains unchanged.

Keep cheap map-membership undo, explicit field undo, and staged table AlterData. Remove automatic whole-object snapshots and the associated partitioning CoW prototype: Update() no longer copies a table's partitioning or per-shard statistics. Existing operations that prohibit proposal rollback keep that contract; this is not an MVCC redesign.

Deferred snapshot prototype: Enjection/ydb#45 (draft).

Pre-merge verification (fe39a0b): 53 related SchemeShard suites, 1,870/1,870 tests passed after reruns; three initial timeout cases each passed three serial repeats and both affected complete suites passed with lower test concurrency. Includes the 200,000-shard snapshot-free Update() regression and reference-count, undo-ordering, and stable-alias tests.

Merged fresh upstream main (bd9080b) without rebasing; resolved the generated-column helper conflict and adapted new metrics/topic lookups to the ownership API. Post-merge verification (96311a4): 15 affected SchemeShard suites, 1,033/1,033 tests passed with hya make -T --build=relwithdebinfo -ttt --test-threads=4; no failures, timeouts, filters, or increased timeout limits.

@ydbot

ydbot commented Jul 13, 2026

Copy link
Copy Markdown
Collaborator

Run Extra Tests

Run additional tests for this PR. You can customize:

  • Test Size: small, medium, large (default: small, medium)
  • Test Targets: any directory path (default: ydb/)
  • Sanitizers: ASAN, MSAN, TSAN
  • Coredumps: enable for debugging (default: off)
  • Additional args: custom ya make arguments

▶  Run tests

@github-actions

This comment was marked as duplicate.

@github-actions

This comment was marked as outdated.

@github-actions

This comment was marked as duplicate.

@github-actions

This comment was marked as outdated.

@Enjection
Enjection force-pushed the draft/schemeshard-tpathref branch from 8fbb967 to 16802af Compare July 14, 2026 10:28
@github-actions

This comment was marked as duplicate.

@github-actions

This comment was marked as outdated.

@github-actions

This comment was marked as duplicate.

@github-actions

This comment was marked as outdated.

@Enjection
Enjection force-pushed the draft/schemeshard-tpathref branch from ec51957 to 58013c6 Compare July 14, 2026 23:47
@github-actions

This comment was marked as duplicate.

@github-actions

This comment was marked as outdated.

@Enjection
Enjection force-pushed the draft/schemeshard-tpathref branch from 58013c6 to 70e06c7 Compare July 15, 2026 05:17
@github-actions

This comment was marked as outdated.

@github-actions

This comment was marked as duplicate.

@Enjection
Enjection force-pushed the draft/schemeshard-tpathref branch from 70e06c7 to 2d63886 Compare July 15, 2026 07:42
@github-actions

This comment was marked as duplicate.

@github-actions

This comment was marked as outdated.

@Enjection
Enjection force-pushed the draft/schemeshard-tpathref branch from 2d63886 to 22d1d51 Compare July 15, 2026 11:13
@github-actions

This comment was marked as duplicate.

@github-actions

This comment was marked as outdated.

@Enjection
Enjection force-pushed the draft/schemeshard-tpathref branch from 22d1d51 to 6753296 Compare July 15, 2026 14:21
- Set() now debug-asserts the acquired path was grabbed (GrabPath/GrabNewPath)
  in this tx, so a create op that forgets to snapshot before acquiring a ref
  crashes in debug instead of drifting. Backstops the create-op grab fixes.
- TMemoryChanges::IsPathTracked scans the (small, per-tx) Paths stack; the stack
  is subclassed to expose its container, no shadow index or release-build cost.
- Document that erase() records no undo (execute-phase only).

Untracked-in-armed guard intentionally omitted: legitimate untracked-in-armed
flows exist (ResolveDomainInfo, extsubdomain dual-tracking), so a blanket assert
would false-fire.

Validated: ut_base + ut_ttl 298/298 GOOD (relwithdebinfo, asserts active).
…tract

- Add TRefLabel: a DbRefCount log label that only binds char arrays (string
  literals). A raw const char* (e.g. TString::c_str()) has no viable ctor and
  fails to compile, so a stored label can never dangle. Thread it through
  TPathRef, AcquirePathDbRef/ReleasePathDbRef, TSelfRefMap, and AcquireSelfDbRef;
  convert to const char* only at the Increment/DecrementPathDbRefCount boundary.
  Direct Increment/Decrement callers keep the raw TStringBuf API by design.
- Document the tracked/untracked-by-phase contract on TSelfRefMap and note the
  extsubdomain untracked-in-armed SubDomains flow.

Validated: ut_base + ut_ttl 298/298 GOOD.
The LOG_ERROR after the Y_VERIFY_S was unreachable: an orphaned source is
skipped earlier when TolerateOrphanedPaths is on, so reaching that arm means the
control is off and the assert always fires first. Remove the dead log; fail-fast
and ICB-gating semantics are unchanged.
States the contract (intentionally drop the count; a snapshot owns it) instead
of a metaphor, and can't be confused with Reset(). Mechanical rename plus a
shorter class comment; no behavior change.
…heir alters

Both held a THolder<...Config> AlterConfig, making them non-copyable, so their
armed-propose alters had to use UpdateUntracked (no rollback) while every sibling
alter uses tracked Update. Add a copy ctor that deep-copies AlterConfig (delete
copy-assign; the undo path pointer-swaps, never assigns), then switch alter_kesus
and alter_fs to tracked Update. Removes the untracked-in-armed exception outside
SubDomains.

Validated: ut_base + ut_filestore + ut_ttl 306/306 GOOD.
Semantics-free rename so the type names state their contract:
- TPathRef            -> TPathDbRef        (owns a DbRefCount unit, not a TPath)
- TSelfRefMap/I...    -> TDbRefMap/IDbRefMap
- SelfRefMaps         -> DbRefMaps
- AcquireSelfDbRef / ReleaseSelfDbRef / SelfDbRefs -> Acquire/ReleaseOwnDbRef / OwnDbRefs
- RecordSelfRefUndo / SelfRefUndos / SelfRefUndoClone / SelfRefUndoRestoreSlot
  -> RecordDbRefUndo / DbRefUndos / DbRefUndoClone / DbRefUndoRestoreSlot
- DebugCheckSelfRefIntegrity -> DebugCheckDbRefIntegrity
- NSelfRefDetail -> NDbRefDetail

Filenames and the TargetPathRef/SourcePathRef member names kept to limit churn.
No behavior change; ut_base + ut_ttl + ut_filestore 306/306 GOOD.
Follow-up to the type rename:
- schemeshard_self_ref_map.h -> schemeshard_db_ref_map.h   (TDbRefMap)
- schemeshard_path_ref.{h,cpp} -> schemeshard_path_db_ref.{h,cpp}  (TPathDbRef)
- ut_base/ut_self_ref_map.cpp -> ut_base/ut_db_ref_map.cpp

Include sites and ya.make updated. No behavior change; ut_base + ut_ttl 298/298 GOOD.
CreateTx returns a TTxState with its target ref already acquired; the
copy-assign in split_merge released it and substituted a hand-built
local's empty handles, so an in-flight split held no DbRefCount ref on
its target path. Introduced by the TPathRef conversion - pre-PR CreateTx
bumped DbRefCount directly, so the same copy was harmless.

TPathDbRef is now move-only, making the class of bug a compile error; it
was the only copy site in the tree.

Tests: ut_split_merge, ut_split_merge_reboots 212/212.
ResolveDomainInfo and GetPathVersion each did contains() followed by a
value-returning accessor. TDbRefMap::FindPtr is already const and returns
const V*, so both collapse to a single lookup: ResolveDomainInfo no longer
needs the const_cast, and GetPathVersion no longer copies a std::shared_ptr
(atomic refcount) per describe of a schema-preset column table.

Also note in TTableInfo::DeepCopy that the copy stays O(partitions) via
Stats.PartitionStats and VerifyConsistency, despite sharing Partitioning.

Tests: ut_base, ut_olap 286/286.
erase records no undo, so an armed-propose erase cannot roll back. That
contract was documented but unenforced: Set and Update have IsArmed
tripwires, erase had none because it takes no TMemoryChanges.

Arm now publishes the active TMemoryChanges on the shard, so erase can
assert against it via a free function - db_ref_map.h holds only a forward
declaration of TSchemeShard. The destructor withdraws only its own
registration, since several TMemoryChanges coexist and only the propose
one arms.

All 31 current erase sites are plan-step or later, so this locks in
behaviour rather than changing it.

Tests: ut_base, ut_olap, ut_index, ut_move, ut_cdc_stream, ut_sequence,
ut_replication, ut_external_table, ut_resource_pool, ut_backup_collection,
ut_consistent_copy_tables, ut_extsubdomain - 645 GOOD, guard did not fire
(6 TAsyncIndexTests reboots timeouts are pre-existing, same on main).
Arm() set a flag cleared only when TMemoryChanges died - a member of the
propose tx - so the window ran past Complete(). Any erase() legitimately
running later in that tx would have tripped the new guard.

Disarm() via Y_DEFER closes the window on every exit, including the early
returns for quota, rewrite and split failures. AbortOperationPropose runs
from inside ProcessOperationParts, so rollback is still armed; the Set and
Update tripwires, NeedsUpdateSnapshot and RecordDbRefUndo all fire during
part-propose and are unaffected. The destructor keeps Disarm as a backstop.

Tests: same 12 suites, 645 GOOD, guard did not fire (6 TAsyncIndexTests
reboots timeouts are pre-existing, same on main).
@Enjection
Enjection dismissed stale reviews from flown4qqqq and dorooleg via 6b1b4ba August 20, 2026 08:17
@Enjection
Enjection force-pushed the draft/schemeshard-tpathref branch from 26f764e to 6b1b4ba Compare August 20, 2026 08:17
@github-actions

This comment was marked as duplicate.

@github-actions

This comment has been minimized.

dorooleg
dorooleg previously approved these changes Aug 20, 2026
@github-actions

github-actions Bot commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

🔴 Unable to merge your PR into the base branch. Please rebase or merge it with the base branch.

@Enjection Enjection changed the title schemeshard: TPathRef, owning handles for path DbRefCount references (design pilot) refactor: automate SchemeShard path reference ownership Sep 6, 2026
Merge upstream main bd9080b without rewriting the PR history. Adapt new generated-column, metrics, and topic code to read-only ownership-map lookups.
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@ijon

ijon commented Sep 10, 2026 •

Copy link
Copy Markdown
Contributor

Main concerns:

  1. Do we even need this type of reference counting? Or perhaps the same goal can be achieved with something simpler?
  2. Eliminating the need for explicit Grab* calls for operations appears to be a supplementary goal, not strictly required to achieve reference counting accuracy.
  3. The performance implications are unknown. -- should be no performance hits from new ref counting (versioning aside)
  4. The column tables case is complicated, as they still employ manual reference counting -- when would they be universally supported?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants