Performance: Parallelize ingestion pipelines

[mikewaters]

In-Pipeline

I can use num_workers to parallelize an individual ingestion.
BLOCKER: sqlite allows only one writer, and our transforms run in a database session which carries through to each (now parallelized via multiproeccing) worker. No good, database locking issues.

Across Pipelines

I can use LLamaindex Workflows tro parallelize multiple jobs, whihc I think I will do atthe Substrate layer.

[mikewaters]

Feb 13, 2026
Nancy — GPT-5.2 Thinking

If your goal is to run multiple LlamaIndex IngestionPipelines at the same time, the best “real-world” approach depends on what’s actually slow (CPU parsing/chunking vs network-bound embedding/LLM calls) and what shared resources you’re writing to (docstore/vector DB).

Recommendation 1: Prefer “one pipeline per dataset” + parallelize inside each pipeline (CPU-bound)

Use when: you’re doing heavy CPU work (PDF parsing, OCR, chunking, regex, layout parsing) and you’re ingesting large backfills.

LlamaIndex supports process-based parallelism inside IngestionPipeline.run() via num_workers, distributing batches across a multiprocessing.Pool. ([LlamaIndex]1)

Why this is often best in production

• Keeps your operational model simpler: one pipeline owns one dataset/source definition (transformations, schema mapping, routing).
• Lets you tune batching + worker count per dataset without multiplying orchestration complexity. ([LlamaIndex]2)

Caution (important):

• There have been versions where num_workers could hang, and there are known multi-worker + cache gotchas; you should validate on your pinned version and treat caches as potentially not process-safe unless documented otherwise. ([GitHub]3)

Sketch

from llama_index.core.ingestion import IngestionPipeline

pipeline_a = IngestionPipeline(transformations=[...])

# CPU-bound speedup (processes)
nodes = pipeline_a.run(documents=docs_a, num_workers=8)

(Parallel execution example in LlamaIndex docs.) (LlamaIndex)

---

Recommendation 2: Run multiple pipelines concurrently with asyncio (I/O-bound)

Use when: the bottleneck is network latency (OpenAI/Azure embeddings, LLM-based extractors, remote object stores) or you’re ingesting lots of small docs.

LlamaIndex has been moving more ingestion/extraction capabilities toward async, and the docs explicitly position async as a throughput win for awaitable operations. ([LlamaIndex]4)

Real-world use case patterns

• Multi-tenant ingestion: one pipeline per tenant (different metadata mapping / security rules), running concurrently but rate-limited.
• Multi-source ingestion: separate pipelines for GitHub, Google Drive, Jira, etc., kicked off together on a schedule.

Sketch (run many pipelines at once)

import asyncio
from llama_index.core.ingestion import IngestionPipeline

pipelines = {
  "notion": IngestionPipeline(transformations=[...]),
  "gdrive": IngestionPipeline(transformations=[...]),
  "github": IngestionPipeline(transformations=[...]),
}

async def run_all(docs_by_source):
    tasks = [
        pipelines[src].arun(documents=docs)   # async run
        for src, docs in docs_by_source.items()
    ]
    return await asyncio.gather(*tasks)

results = asyncio.run(run_all(docs_by_source))

Operational tip: add explicit concurrency control (semaphores) around the actual external bottleneck (embedding calls / extractor calls), so you don’t DDoS your own rate limits.

---

Recommendation 3: If you truly need “N pipelines at once” on one host, use process-per-pipeline (hard isolation)

Use when: pipelines have conflicting dependencies/configs, you need fault isolation, or you’ve got mixed workloads (some CPU-heavy, some I/O-heavy), and you want OS-level separation.

This is the most robust option when:

• you’re writing to shared stores and want clean boundaries (separate connections, separate caches),
• you’ve had issues with multi-worker caching/thread safety,
• you want independent retries/backoff per pipeline.

Sketch

from concurrent.futures import ProcessPoolExecutor
from llama_index.core.ingestion import IngestionPipeline

def run_pipeline(serialized_cfg, docs):
    pipe = IngestionPipeline(**serialized_cfg)
    return pipe.run(documents=docs)

with ProcessPoolExecutor(max_workers=4) as ex:
    futures = [
        ex.submit(run_pipeline, cfg_a, docs_a),
        ex.submit(run_pipeline, cfg_b, docs_b),
        ex.submit(run_pipeline, cfg_c, docs_c),
    ]
    results = [f.result() for f in futures]

Caution: if you use LlamaIndex ingestion caching, verify how your cache backend behaves across processes; there are reported cases where multi-worker ingestion + caches don’t behave as expected without per-process initialization. ([GitHub]5)

---

When to switch to a “real orchestrator” (Ray / Celery / Prefect / Airflow)

If ingestion is becoming a platform feature (retries, backpressure, distributed workers, job visibility), move orchestration out of your app process.

Ray is a common choice for scaling Python workloads with concurrency controls and distributed execution (actors + async). ([Ray]6)
And if you’re already using LlamaIndex Workflows, they have a documented concurrency model for steps (useful if you refactor ingestion into workflow steps). ([LlamaIndex]7)

---

Practical “default” I’d pick

• If you’re ingesting big corpora with heavy parsing/chunking: Recommendation 1 (parallelize inside pipeline with num_workers) first. ([LlamaIndex]1)
• If you’re embedding/extracting over the network: Recommendation 2 (asyncio across pipelines) with a global rate limiter. ([LlamaIndex]8)
• If reliability/isolation matters most: Recommendation 3 (process-per-pipeline), then later graduate to Ray/Celery.

If you tell me whether your bottleneck is CPU parsing, embedding calls, or metadata extraction, I’ll pin this to a concrete architecture (including safe cache/docstore/vector-store wiring and rate-limits) for your specific setup.