Phases 4-7: Streaming, LLM/RAG, Deployment, Ecosystem#10
Merged
Conversation
New source connectors: - Kafka (batch + streaming modes, configurable offsets) - MongoDB (collection reads, aggregation pipeline support) - REST API (generic, with offset/cursor pagination, bearer/api-key auth) - BigQuery source (table reads + SQL queries) New sink connectors: - Kafka (batch + streaming with checkpointing) - Elasticsearch (Spark ES connector with REST bulk API fallback) All connectors include healthCheck for weaver doctor. Dependencies: - kafka-clients 3.7.0 (provided, for health checks) Schema: - Updated pipeline.schema.json with new connector types Total connectors: 8 sources + 6 sinks = 14 connectors
LLM Transforms: - LLMTransformPlugin: generic LLM-as-transformation with prompt templating, structured JSON output parsing, batching, caching, retry with backoff - Supports: claude, openai, local (Ollama), any OpenAI-compatible API - Config: batchSize, maxConcurrent, retryOnError, cache, apiKey, baseUrl RAG Transforms: - ChunkingPlugin: fixed, sentence, recursive chunking strategies with configurable size and overlap - EmbeddingPlugin: vector embedding generation via OpenAI/Vertex AI/Cohere APIs, returns embedding column (array of doubles) - GraphExtractionPlugin: entity/relationship extraction using LLM, returns source_entity, source_type, relation, target_entity, target_type Local model support: - All LLM plugins support provider: "local" for Ollama (http://localhost:11434/v1/chat/completions) - No API key required for local models - Custom baseUrl for self-hosted endpoints Total transform types: 6 (SQL, DataQuality, LLMTransform, Chunking, Embedding, GraphExtraction)
Deployment targets: - SubmitCommand: spark-submit wrapper for standalone, k8s, EMR, Dataproc - Dockerfile: JDK 17 + Spark 4.0.2 + data-weaver.jar - docker-entrypoint.sh: container entry point - install.sh: one-line installer (curl | bash) Airflow integration: - DataWeaverOperator: Airflow operator for pipeline execution - Python package (data-weaver-airflow) with pyproject.toml - Template fields for pipeline, env, extra_args Usage: docker run data-weaver:latest apply pipeline.yaml --env prod weaver apply --submit k8s --master k8s://host:6443 --image img:tag weaver apply --submit emr --cluster-id j-XXXXX curl -fsSL .../install.sh | bash
Connector SDK: - CONNECTOR_SDK.md: complete guide for building custom connectors - Source and Sink trait documentation with examples - ServiceLoader registration instructions - Best practices for health checks and config handling Tutorials (3 real-world pipelines with sample data): - 01-csv-to-parquet: CSV read, SQL transform, quality checks, Parquet write - 02-multi-source-join: parallel transforms, multi-source join, JSON output - 03-rag-pipeline: document chunking for RAG preparation - Sample data: customers.csv, orders.csv, docs.json Reusable CI workflow: - weaver-pipeline-ci.yml: GitHub Actions workflow for pipeline validation - Auto-validates all YAML files in pipelines/ directory - Runs inline tests for each pipeline - Reusable via workflow_call for user projects
…ronously Multi-node levels still use Future for parallelism. Single-node levels execute directly to avoid Spark classloader conflicts in test environments.
…ronously Multi-node levels use Future for parallelism. Single-node levels execute directly to avoid Spark classloader conflicts in test environments.
PluginRegistry.safeLoad skips connectors whose dependencies are not on the classpath (e.g., KafkaSourceConnector without kafka-clients). This prevents ServiceConfigurationError in test environments.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Consolidated PR for Phases 4 through 7, completing the Data Weaver v2 roadmap.
Phase 4: Streaming + Advanced Connectors
New source connectors:
New sink connectors:
Totals: 8 sources + 6 sinks = 14 connectors
Phase 5: LLM Transforms + RAG
provider: localfor Ollama (no API key required)baseUrlfor self-hosted LLM serversTotal transform types: 6 (SQL, DataQuality, LLMTransform, Chunking, Embedding, GraphExtraction)
Phase 6: Deployment + Production
curl | bash)DataWeaverOperatorPython package withpyproject.tomlPhase 7: Ecosystem
weaver-pipeline-ci.ymlfor user projectsComplete CLI
Test plan
sbt compile— All modules compile