Skip to content

[Filestore] [2/N] Add compressed bitmap serialization helpers and tests - #7247

Open
Maximus5 wants to merge 1 commit into
mainfrom
users/maximus5/filestore-resize-persist-state-bitmap
Open

[Filestore] [2/N] Add compressed bitmap serialization helpers and tests#7247
Maximus5 wants to merge 1 commit into
mainfrom
users/maximus5/filestore-resize-persist-state-bitmap

Conversation

@Maximus5

@Maximus5 Maximus5 commented Sep 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Notes

Add filestore storage-core helpers for serializing and deserializing TCompressedBitmap through TCompressedBitmapData.

The PR also adds validation for shard creation state bitmap metadata, including max bit count, chunk count, chunk indexes, and empty chunk data. This is needed as a foundation for persistent resize shard creation state, where bitmap data is stored in tablet state and later loaded by resize/create flows.

Unit tests cover bitmap round-trips, empty bitmaps, zero-chunk skipping, minBitCount handling, out-of-range chunks on load, save bounds, and invalid shard creation state metadata.

Stack

  1. [Filestore] [1/N] Document the filestore resize flow in legacy and throttled modes #6376
  2. [Filestore] [2/N] Add compressed bitmap serialization helpers and tests #7247
  3. [Filestore] [3/N] Add shard creation state companion for persistent resize progress #7249
  4. [Filestore] [4/N] Persist shard creation progress across throttled create and resize retries #7253
  5. [Filestore] [5/N] Use persistent shard creation state during throttled create and resize #7255
  6. [Filestore] [6/N] Add resize retry and tablet restart coverage for sharded filestores #7256

@github-actions

github-actions Bot commented Sep 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Note

This is an automated comment that will be appended during run.

Note

All workloads for linux-x86_64-relwithdebinfo have completed.

Tip

Planned checks for linux-x86_64-relwithdebinfo.

🟢 linux-x86_64-relwithdebinfo target: cloud/tasks/,cloud/storage/ (test time: 273s): all tests PASSED for commit 9acbf9f.

TESTS PASSED ERRORS FAILED FAILED BUILD SKIPPED MUTED?
1580 1580 0 0 0 0 0

🔴 linux-x86_64-relwithdebinfo target: cloud/blockstore/ (test time: 1242s): some tests FAILED for commit 9acbf9f.

TESTS PASSED ERRORS FAILED FAILED BUILD SKIPPED MUTED?
7573 7571 0 0 1 1 0

@Maximus5
Maximus5 force-pushed the users/maximus5/filestore-resize-persist-state-bitmap branch from 9acbf9f to 253ba33 Compare September 21, 2026 14:35
@github-actions

github-actions Bot commented Sep 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Note

This is an automated comment that will be appended during run.

Note

All workloads for linux-x86_64-relwithdebinfo have completed.

Tip

Planned checks for linux-x86_64-relwithdebinfo.

🟢 linux-x86_64-relwithdebinfo target: cloud/tasks/,cloud/storage/ (test time: 276s): all tests PASSED for commit 253ba33.

TESTS PASSED ERRORS FAILED FAILED BUILD SKIPPED MUTED?
1580 1580 0 0 0 0 0

🟢 linux-x86_64-relwithdebinfo target: cloud/disk_manager/ (test time: 304s): all tests PASSED for commit 253ba33.

TESTS PASSED ERRORS FAILED FAILED BUILD SKIPPED MUTED?
1746 1746 0 0 0 0 0

🟢 linux-x86_64-relwithdebinfo target: cloud/blockstore/ (test time: 1405s): all tests PASSED for commit 253ba33.

TESTS PASSED ERRORS FAILED FAILED BUILD SKIPPED MUTED?
7574 7573 0 0 0 1 0

🟢 linux-x86_64-relwithdebinfo target: cloud/filestore/ (test time: 4505s): all tests PASSED for commit 253ba33.

TESTS PASSED ERRORS FAILED FAILED BUILD SKIPPED MUTED?
5038 5037 0 0 0 1 0

@Maximus5
Maximus5 added this pull request to stack #7257 September 21, 2026 16:12
This was referenced Sep 21, 2026
Open
Open
Open
Open
Open
@Maximus5
Maximus5 force-pushed the users/maximus5/filestore-resize-persist-state-bitmap branch from 253ba33 to 5ea68ab Compare September 24, 2026 15:17
@github-actions

github-actions Bot commented Sep 24, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Note

This is an automated comment that will be appended during run.

Note

All workloads for linux-x86_64-relwithdebinfo have completed.

Tip

Planned checks for linux-x86_64-relwithdebinfo.

🟢 linux-x86_64-relwithdebinfo target: cloud/tasks/,cloud/storage/,cloud/fastshard/ (test time: 272s): all tests PASSED for commit 5ea68ab.

TESTS PASSED ERRORS FAILED FAILED BUILD SKIPPED MUTED?
1671 1671 0 0 0 0 0

🟢 linux-x86_64-relwithdebinfo target: cloud/disk_manager/ (test time: 332s): all tests PASSED for commit 5ea68ab.

TESTS PASSED ERRORS FAILED FAILED BUILD SKIPPED MUTED?
1746 1746 0 0 0 0 0

🟢 linux-x86_64-relwithdebinfo target: cloud/blockstore/ (test time: 1697s): all tests PASSED for commit 5ea68ab.

TESTS PASSED ERRORS FAILED FAILED BUILD SKIPPED MUTED?
7436 7435 0 0 0 1 0

🟢 linux-x86_64-relwithdebinfo target: cloud/filestore/ (test time: 4887s): all tests PASSED for commit 5ea68ab.

TESTS PASSED ERRORS FAILED FAILED BUILD SKIPPED MUTED?
4812 4810 0 0 0 2 0

@Maximus5
Maximus5 force-pushed the users/maximus5/filestore-resize-persist-state-bitmap branch from 5ea68ab to 41ee130 Compare September 30, 2026 16:44
@Maximus5 Maximus5 added the recheck Add this label to relaunch checks, it will be automatically removed label Sep 30, 2026
@github-actions github-actions Bot removed the recheck Add this label to relaunch checks, it will be automatically removed label Sep 30, 2026
@github-actions

github-actions Bot commented Sep 30, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Note

This is an automated comment that will be appended during run.

Note

All workloads for linux-x86_64-relwithdebinfo have completed.

Tip

Planned checks for linux-x86_64-relwithdebinfo.

🟢 linux-x86_64-relwithdebinfo target: cloud/tasks/,cloud/storage/,cloud/fastshard/ (test time: 284s): all tests PASSED for commit 41ee130.

TESTS PASSED ERRORS FAILED FAILED BUILD SKIPPED MUTED?
1718 1718 0 0 0 0 0

🟢 linux-x86_64-relwithdebinfo target: cloud/disk_manager/ (test time: 335s): all tests PASSED for commit 41ee130.

TESTS PASSED ERRORS FAILED FAILED BUILD SKIPPED MUTED?
1755 1755 0 0 0 0 0

🔴 linux-x86_64-relwithdebinfo target: cloud/blockstore/ (test time: 1611s): some tests FAILED for commit 41ee130.

TESTS PASSED ERRORS FAILED FAILED BUILD SKIPPED MUTED?
7487 7476 0 10 0 1 0

🟢 linux-x86_64-relwithdebinfo target: cloud/blockstore/ (test time: 1171s): all tests PASSED for commit 41ee130.

TESTS PASSED ERRORS FAILED FAILED BUILD SKIPPED MUTED?
7493 7492 0 0 0 1 0

🔴 linux-x86_64-relwithdebinfo target: cloud/filestore/ (test time: 5405s) Test run completed, no test results found for commit 41ee130.
Please check build logs.

🟢 linux-x86_64-relwithdebinfo target: cloud/filestore/ (test time: 4995s): all tests PASSED for commit 41ee130.

TESTS PASSED ERRORS FAILED FAILED BUILD SKIPPED MUTED?
4901 4900 0 0 0 1 0

@Maximus5
Maximus5 force-pushed the users/maximus5/filestore-resize-persist-state-bitmap branch from 41ee130 to 3196875 Compare October 1, 2026 08:22
@Maximus5 Maximus5 added the recheck Add this label to relaunch checks, it will be automatically removed label Oct 1, 2026
@github-actions github-actions Bot removed the recheck Add this label to relaunch checks, it will be automatically removed label Oct 1, 2026
@github-actions

github-actions Bot commented Oct 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Note

This is an automated comment that will be appended during run.

Note

All workloads for linux-x86_64-relwithdebinfo have completed.

Tip

Planned checks for linux-x86_64-relwithdebinfo.

🟢 linux-x86_64-relwithdebinfo target: cloud/tasks/,cloud/storage/,cloud/fastshard/ (test time: 271s): all tests PASSED for commit 3196875.

TESTS PASSED ERRORS FAILED FAILED BUILD SKIPPED MUTED?
1718 1718 0 0 0 0 0

🟢 linux-x86_64-relwithdebinfo target: cloud/disk_manager/ (test time: 313s): all tests PASSED for commit 3196875.

TESTS PASSED ERRORS FAILED FAILED BUILD SKIPPED MUTED?
1755 1755 0 0 0 0 0

🟢 linux-x86_64-relwithdebinfo target: cloud/blockstore/ (test time: 1469s): all tests PASSED for commit 3196875.

TESTS PASSED ERRORS FAILED FAILED BUILD SKIPPED MUTED?
7494 7493 0 0 0 1 0

🟢 linux-x86_64-relwithdebinfo target: cloud/filestore/ (test time: 5109s): all tests PASSED for commit 3196875.

TESTS PASSED ERRORS FAILED FAILED BUILD SKIPPED MUTED?
4901 4900 0 0 0 1 0

SvartMetal
SvartMetal reviewed Oct 2, 2026
View reviewed changes
Comment thread cloud/filestore/private/api/protos/tablet.proto Outdated
bytes Data = 2;
}

message TCompressedBitmapData

@SvartMetal SvartMetal Oct 2, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's try to move serialization helpers and protobufs to cloud/storage/core/libs/common?

@Maximus5
Maximus5 force-pushed the users/maximus5/filestore-resize-persist-state-bitmap branch from 3196875 to 50ebed0 Compare October 21, 2026 18:35
@Maximus5 Maximus5 added the recheck Add this label to relaunch checks, it will be automatically removed label Oct 21, 2026
@github-actions github-actions Bot removed the recheck Add this label to relaunch checks, it will be automatically removed label Oct 21, 2026
@github-actions

github-actions Bot commented Oct 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Note

This is an automated comment that will be appended during run.

Note

All workloads for linux-x86_64-relwithdebinfo have completed.

Tip

Planned checks for linux-x86_64-relwithdebinfo.

🟢 linux-x86_64-relwithdebinfo target: cloud/tasks/,cloud/storage/,cloud/fastshard/ (test time: 268s): all tests PASSED for commit 50ebed0.

TESTS PASSED ERRORS FAILED FAILED BUILD SKIPPED MUTED?
1805 1805 0 0 0 0 0

🟢 linux-x86_64-relwithdebinfo target: cloud/disk_manager/ (test time: 313s): all tests PASSED for commit 50ebed0.

TESTS PASSED ERRORS FAILED FAILED BUILD SKIPPED MUTED?
1849 1849 0 0 0 0 0

🟢 linux-x86_64-relwithdebinfo target: cloud/blockstore/ (test time: 1477s): all tests PASSED for commit 50ebed0.

TESTS PASSED ERRORS FAILED FAILED BUILD SKIPPED MUTED?
7688 7687 0 0 0 1 0

🔴 linux-x86_64-relwithdebinfo target: cloud/filestore/ (test time: 5460s) Test run completed, no test results found for commit 50ebed0.
Please check build logs.

🟢 linux-x86_64-relwithdebinfo target: cloud/filestore/ (test time: 5090s): all tests PASSED for commit 50ebed0.

TESTS PASSED ERRORS FAILED FAILED BUILD SKIPPED MUTED?
5057 5056 0 0 0 1 0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@SvartMetal SvartMetal SvartMetal left review comments
@qkrorlqr qkrorlqr Awaiting requested review from qkrorlqr
@ruslan-gar ruslan-gar Awaiting requested review from ruslan-gar
@debnatkh debnatkh Awaiting requested review from debnatkh

At least 2 approving reviews are required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@Maximus5 @SvartMetal