Guard against deleting a knowledge base while its documents are still indexing (pause/await before delete)

[shaba]

Self Checks

• I have searched for existing issues, including closed ones.
• I confirm that I am using English to submit this report.
• Please do not modify this template :) and fill in all the required fields.

1. Is this request related to a challenge you're experiencing?

Follow-up to #38518. That issue has two halves:

1. Non-robust cleanup — being fixed in fix(dataset): clean deleted dataset in committed batches to avoid orphaned data #38519 (delete in per-step-committed batches so a failure no longer rolls back the whole cleanup and orphans everything).
2. No guard against deleting a dataset while its documents are still indexing — this request.

DatasetService.delete_dataset (api/services/dataset_service.py, ~L1308) sends the dataset_was_deleted signal and does db.session.delete(dataset) immediately, with no check for or cancellation of in-flight indexing. When a dataset is deleted mid-indexing, the running pipeline keeps executing and fails repeatedly with KnowledgeIndexNodeError: Dataset <id> not found. (core/rag/index_processor/index_processor.py index_and_clean), and it can keep writing rows that race with the cleanup task — the root cause behind the orphaned data in #38518. Even with #38519 making cleanup robust, avoiding the race in the first place is the correct defensive posture.

2. Describe the feature you'd like to see

Before deleting a dataset, pause and await any in-flight indexing for its documents, then delete.

A concrete shape (suggested on #38518): reuse the existing per-document pause mechanism — Document.is_paused (set e.g. in controllers/console/datasets/datasets_document.py) which makes indexing_runner raise DocumentIsPausedError at its stage checkpoints. delete_dataset could:

1. Find all documents of the dataset in an active indexing state (IndexingStatus WAITING / PARSING / CLEANING / SPLITTING / INDEXING, see models/enums.py).
2. Set their pause flag (Redis + DB) and wait for the running tasks to acknowledge/stop at the next checkpoint (with a bounded timeout).
3. Proceed with deletion + cleanup once no document is actively indexing.

This closes the race without having to revoke Celery tasks directly. An alternative (simpler but less friendly) is to reject deletion with a clear error while indexing is active, letting the user stop/finish it first.

3. How will this feature improve your workflow?

Deleting a knowledge base while it is still indexing (easy to do on large ingests) no longer races the cleanup and leaves tens of GB of orphaned document_segments / child_chunks / embedding_vector_index_* behind — which is exactly what we hit in production (#38518).

Related

• Deleting a knowledge base while its documents are still indexing leaves orphaned segments, child_chunks and pgvector tables #38518 (parent report), fix(dataset): clean deleted dataset in committed batches to avoid orphaned data #38519 (cleanup-robustness fix)
• fix: clean orphan segments during dataset deletion #37773 (related document-vs-dataset segment race), [Bug] Missing retry limits in _delete_records causes infinite loop and disk exhaustion (Poison Pill) #38117 (infinite loop in _delete_records during dataset deletion)