Deleting a dataset leaves an orphaned vector collection when `doc_form` or `indexing_technique` is empty (vector DB never cleaned up)

[6mvp6]

Self Checks

• I have read the Contributing Guide and Language Policy.
• This is only for bug report, if you would like to ask a question, please head to Discussions.
• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report, otherwise it will be closed.
• 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：）
• Please do not modify this template :) and fill in all the required fields.

Dify version

v1.15.0

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

1. Deploy Dify with VECTOR_STORE=qdrant.

2. Use a dataset whose datasets.doc_form and/or datasets.indexing_technique column is empty/null. This is common for knowledge bases created in older Dify versions and carried forward via upgrade. Index some documents so a Qdrant collection such as Vector_index_<uuid>_Node exists on the vector store.

3. Delete the dataset from the UI (or via DELETE /datasets/{id}).

4. Inspect Qdrant afterwards:

   GET /collections
   

   The Vector_index_<uuid>_Node collection is still present. Checking the Qdrant HTTP access log shows that no DELETE /collections/... request was ever issued during the deletion.

5. The orphaned collection persists permanently. Repeated deletes keep accumulating more orphans; they are never reconciled.

✔️ Expected Behavior

When a dataset is deleted, the vector-store collection that holds its embeddings must be reliably deleted (or scheduled for deletion that is guaranteed to run), so that no orphaned collection is left behind — regardless of whether doc_form / indexing_technique are populated.

❌ Actual Behavior

The datasets row is removed, but the corresponding Qdrant collection is never deleted and becomes an orphan. The deletion handler short-circuits before dispatching the cleanup task, and there is no retry/reconcile path, so the orphan is permanent.

Root cause (code-level)

The signal handler that owns vector cleanup guards too aggressively and bails out for exactly the case the cleanup task is prepared to handle.

api/events/event_handlers/clean_when_dataset_deleted.py:

@dataset_was_deleted.connect
def handle(sender: Dataset, **kwargs):
    dataset = sender
    if not dataset.doc_form or not dataset.indexing_technique:
        return                       # ← skips vector cleanup entirely
    clean_dataset_task.delay(
        dataset.id,
        dataset.tenant_id,
        dataset.indexing_technique,
        dataset.index_struct,
        dataset.collection_binding_id,
        dataset.doc_form,
        dataset.pipeline_id,
    )

The early return short-circuits cleanup for any dataset whose doc_form or indexing_technique is empty. Datasets created in older Dify versions (and migrated forward) frequently have one of these fields empty, so deleting them never dispatches clean_dataset_task, and the Qdrant collection is never removed.

This is also internally inconsistent. The very task it guards already defends against an empty doc_form:

# api/tasks/clean_dataset_task.py
if doc_form is None or (isinstance(doc_form, str) and not doc_form.strip()):
    # Use default paragraph index type for empty/invalid datasets
    # to enable vector database cleanup
    doc_form = IndexStructureType.PARAGRAPH_INDEX

So the handler's guard is stricter than the task it protects — it blocks the precise case the task already handles. indexing_technique is likewise not required by the cleanup logic (it is only stored on the reconstructed Dataset object and not used to select the cleanup path), so guarding on it serves no purpose other than skipping cleanup.

A second, related weakness compounds the problem. Inside clean_dataset_task, the vector cleanup is wrapped in try/except; on failure it only logs and then proceeds to commit the rest of the DB cleanup:

# api/tasks/clean_dataset_task.py
try:
    index_processor = IndexProcessorFactory(doc_form).init_index_processor()
    index_processor.clean(dataset, None, with_keywords=True, delete_child_chunks=True)
except Exception:
    logger.exception(...)
    # Continue with document and segment deletion even if vector cleanup fails

Because delete_dataset has already committed the deletion of the datasets row (db.session.delete(dataset) + commit) before the async task runs, a failed cleanup can never be retried — the orphan is permanent and silent.

Impact

Orphans accumulate silently and invisibly (the API never reports them). On a qdrant container restart/recreate, all collections — including orphans — are loaded at once, spiking open file handles. If the container's nofile ulimit is modest, this triggers RocksDB IO error: Too many open files → Qdrant panic → restart: always loop → all knowledge-base retrieval fails. From the API side this surfaces as:

InactiveRpcError ... StatusCode.UNAVAILABLE
details: "DNS resolution failed for qdrant:6334: ... Domain name not found"

(The DNS error is a symptom: the qdrant container is crash-looping and intermittently unresolvable, not an actual DNS problem.)

We hit exactly this in production after a version upgrade that recreated the qdrant container: the container restart-looped ~100 times and knowledge-base retrieval was down.

[dosubot]

Your analysis is spot-on. The early-return guard in clean_when_dataset_deleted.py is indeed stricter than the task it protects — clean_dataset_task already has a fallback that defaults doc_form to PARAGRAPH_INDEX when it's empty [1], so the handler's guard needlessly blocks cleanup for migrated datasets.

This guard was introduced in PR #27083 (merged Oct 2025) to replace hard assert statements that were crashing the handler [2]. The intent was to gracefully skip datasets that were "never indexed," but it over-corrects by also blocking datasets that were indexed but simply have a null doc_form or indexing_technique due to schema changes across upgrades.

Related work in progress:

• PR #38519 (opened yesterday) tackles a complementary orphaning path — when clean_dataset_task does run but the single large transaction rolls back on timeout, leaving millions of orphaned rows and tens of GBs of orphaned vector tables [3]. It refactors the task to use bounded, individually-committed batch deletions.
• PR #37773 fixes a race condition where concurrent clean_document_task deletes document rows before clean_dataset_task runs, causing segment cleanup to be skipped [4].

However, neither of those PRs addresses the specific early-return path you've identified — the case where the cleanup task is never dispatched at all because of the overly aggressive guard. The fix you're implying (relaxing or removing the if not dataset.doc_form or not dataset.indexing_technique: return guard) appears correct and low-risk given that the downstream task already handles empty values gracefully.

Workaround (for existing orphans): You'll need to manually identify and delete orphaned Qdrant collections. You can cross-reference Vector_index_<uuid>_Node collection names against existing datasets.id values in your DB — any collection whose UUID doesn't match a live dataset is an orphan and can be safely deleted via the Qdrant API.

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[6mvp6]

qdrant_sync_fix.zip Man, i ran into this exact same headache with the qdrant orphan collections. Turns out the worker was skipping the delete call when those metadata fields were null. I put together a quick python script that syncs the db state with the vector store to clear the junk.

I cleaned up the existing orphans using the official Qdrant REST API

[Zhengcy05]

@dosu

The key part is that clean_when_dataset_deleted.py returns early when either dataset.doc_form or dataset.indexing_technique is empty, while clean_dataset_task.py already has a fallback for an empty doc_form and defaults it to PARAGRAPH_INDEX before running cleanup. So for migrated datasets where doc_form is missing, the handler blocks a case that the downstream task is already prepared to handle, which can prevent vector cleanup from being dispatched at all.

One detail I would narrow a bit is the indexing_technique part. From the current code, that value still affects the cleanup path inside the index processors, so it may be safer not to relax that branch without verifying the full behavior first. A lower-risk fix seems to be removing the short-circuit for missing doc_form and adding a regression test for the case where doc_form is None but cleanup should still be scheduled.

If this direction looks good, I’d like to work on this and submit a PR.