Skip to content

[Feature] Expose DROP TABLE mod file progress - #18855

Open
Caideyipi wants to merge 2 commits into
apache:masterfrom
Caideyipi:feature/drop-table-mod-progress
Open

Caideyipi wants to merge 2 commits into
apache:masterfrom
Caideyipi:feature/drop-table-mod-progress

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Description

Extends #18806 so information_schema.drop_table_procedures reports live mod file progress while an asynchronous DROP TABLE is in DELETE_DATA.

SELECT database, table_name, progress,
       written_mod_file_count, remaining_mod_file_count
FROM information_schema.drop_table_procedures;

written_mod_file_count increases after a distinct mod file has been written and closed successfully. remaining_mod_file_count counts the pending mod files once all participating regions have identified their totals. Shared mod files are counted once; TsFiles removed outright are excluded. A drop with no mod files reports 0, 0.

ConfigNode queries the DataNode executing each region and updates the targets when deletion retries select another replica, counting each region once. Unknown counters remain SQL NULL, including unavailable progress RPCs. A region that has not started reports zero written with an unknown remaining count. The final snapshot is retained with the procedure and serialized compatibly with the previous procedure format.

Validation

  • English and Chinese locale compilation passed.
  • Targeted ConfigNode and DataNode unit tests passed: 57 cases, 56 passed and 1 skipped. They cover aggregation, serialization compatibility, consensus propagation, shared mod files, live counters during a blocked close, failed closes, and zero-mod drops.
  • IoTDBDropTableModProgressIT and IoTDBTableIT#testAsyncDropTableProgress passed with TableSimpleIT and TableClusterIT (HighPerformanceMode, 1 ConfigNode + 3 DataNodes, data replication factor 2).
  • Spotless, Checkstyle, and git diff --check passed. Integration-test packaging used -Drat.skip=true because the existing Go submodule contains files rejected by RAT.

This PR has:

  • been self-reviewed for concurrent reads and writes.
  • added comments explaining the counter semantics and concurrency.
  • added unit tests and integration tests.
  • been tested in a test IoTDB cluster.

Comment on lines +103 to +113
/** Refreshes counters without allowing observability RPCs to block deletion indefinitely. */
public void updateModFileProgress() {
if (!modProgressLock.tryLock()) {
return;
}
try {
refreshModFileProgress();
} finally {
modProgressLock.unlock();
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When there are concurrent SHOW DROP TABLE requests, only one will wait for the latest result; the others will just return the stale result?
How about creating a future with the lock holder and letting the others wait for the result?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied in f434aa0. Concurrent SHOW DROP TABLE requests now wait on the same in-flight CompletableFuture before reading the refreshed counters. The final refresh uses the same coordination so an earlier query cannot overwrite the final snapshot. Added a regression test for the wait behavior; all 10 DropTableProcedureTest tests pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants