Skip to content

[Feature] Add logical backup dump, verify, inspect, pack and restore - #18623

Open
Caideyipi wants to merge 10 commits into
masterfrom
feat/pipe-logical-backup-sink
Open

Caideyipi wants to merge 10 commits into
masterfrom
feat/pipe-logical-backup-sink

Conversation

@Caideyipi

@Caideyipi Caideyipi commented Sep 11, 2026 •

Copy link
Copy Markdown
Collaborator

Logical backup now has separate capture, inspection, validation, packaging and restore operations. dump connects to the source and seals a portable directory; pack verifies an existing directory and creates a ZIP without accepting database connection options. Deprecated export/import aliases report their replacements and preserve the corresponding workflows.

  • Add versioned logical request streams for ConfigRegion, SchemaRegion and DataRegion, with framed records, event groups, CRCs, segment digests, rolling, fsync policies and crash recovery. Metadata snapshots are decoded into logical plans and data files into logical tablet requests. Each Region has one ordered writer; temporary parameter validation does not create backup files.
  • Make the root manifest.json the archive entry point, recording scope, models, filters, capture settings, source identity, stream/file paths and SHA-256 digests, counts, completeness and capture/seal times. Legacy stream-only inputs remain readable and gain a root manifest when packed.
  • Validate capture settings before task creation, exclude authorization objects by default, preserve path/time filtering for InsertNodes and deletion ranges, and wait for metadata as well as data completion before sealing. Bounded continuous capture drains accepted events after its --until boundary.
  • Share structural, checksum and semantic validation across inspection, verification, packaging and restore. Reject unsafe/duplicate/unlisted archive paths and corruption; --allow-incomplete only permits explicitly declared content gaps.
  • Restore config, schema and data in deterministic stream order. Local dry-run needs no connection, target dry-run checks version/precision/authentication/protocol support, and actual replay persists content/target-bound checkpoints with request-level progress. Only the uncertain request on resume may accept an idempotent conflict. Authorization restore requires explicit selection.
  • Document the canonical commands, shared-directory deployment, recovery steps and current capability limits in iotdb-client/cli/LOGICAL_BACKUP.md; keep English and Chinese help/error messages in sync.

Current capability limits are explicit in help and documentation: source nodes and the CLI need a shared backup filesystem; --stop-sequence is rejected because there is no stable global source sequence; filtered capture requires one model, with a single prefix/full Tree path for config/schema and independent database/table regexes for Table. Target object conflicts and object-level permissions are checked during replay, and only --conflict fail is supported. Streams do not represent a global source transaction.

Validation (Windows, JDK 17):

  • Full English reactor compilation and the 9 targeted unit-test classes: 96 tests passed.
  • IoTDBPipeLogicalBackupIT: 2 tests passed using two clusters with 1 ConfigNode and 1 DataNode each. Covers Tree snapshots, aligned/null data, scope/auth/path/time exclusion, ZIP verification, local/target dry-run, restore/checkpoint resume, and bounded Table capture with realtime inserts and deletion. Both cases assert that metadata is stored as logical plans rather than snapshot-file requests.
  • Spotless, English/Chinese message-key and format-placeholder parity, deterministic message names, new-file license headers, credential exclusion and git diff --check passed.

Chinese full-reactor test-compile -P with-zh-locale passed.

The verification build used -Drat.skip=true because the local, pre-existing Go submodule changes contain 5 unrelated files without license headers. Those changes were preserved and are excluded from this PR update; all new files in this change have Apache license headers.

@codecov

codecov Bot commented Sep 11, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 68.18750% with 1018 lines in your changes missing coverage. Please review.
✅ Project coverage is 46.59%. Comparing base (4c34a25) to head (7772f63).

Files with missing lines Patch % Lines
.../apache/iotdb/tool/pipe/PipeLogicalBackupTool.java 56.36% 415 Missing ⚠️
...sink/protocol/logicalbackup/LogicalBackupSink.java 29.43% 187 Missing ⚠️
...sink/protocol/logicalbackup/LogicalBackupSink.java 0.96% 103 Missing ⚠️
...s/pipe/sink/logicalbackup/LogicalBackupWriter.java 88.44% 84 Missing ⚠️
...sink/logicalbackup/LogicalBackupSegmentReader.java 76.27% 56 Missing ⚠️
...sink/logicalbackup/LogicalBackupArchiveReader.java 85.15% 49 Missing ⚠️
...ache/iotdb/tool/pipe/LogicalBackupDumpOptions.java 72.78% 43 Missing ⚠️
...s/pipe/sink/logicalbackup/LogicalBackupRecord.java 57.14% 18 Missing ⚠️
...dinator/runtime/heartbeat/PipeHeartbeatParser.java 57.89% 16 Missing ⚠️
.../pipe/sink/logicalbackup/LogicalBackupCatalog.java 89.36% 15 Missing ⚠️
... and 8 more
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #18623      +/-   ##
============================================
+ Coverage     46.40%   46.59%   +0.19%     
  Complexity      713      713              
============================================
  Files          5537     5551      +14     
  Lines        404321   407506    +3185     
  Branches      52850    53501     +651     
============================================
+ Hits         187615   189879    +2264     
- Misses       216706   217627     +921     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@Caideyipi Caideyipi changed the title [Feature] Add Pipe logical backup sink and restore tooling [Feature] Add Pipe logical backup sink and import tooling Sep 16, 2026
@Caideyipi Caideyipi changed the title [Feature] Add Pipe logical backup sink and import tooling [Feature] Add logical backup dump, verify, inspect, pack and restore Oct 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant