Ray integration for Dagster.
dagster-ray enables working with distributed Ray compute from Dagster pipelines, combining Dagster's excellent orchestration capabilities and Ray's distributed computing power together.
Note
This project is ready for production use, but some APIs may change between minor releases.
Learn more in the docs
- 🎯 Run Launchers & Executors: Submit Dagster runs or individual steps by submitting Ray jobs
- 🔧 Ray Resources: Automatically create and destroy ephemeral Ray clusters and connect to them in client mode
- 📡 Dagster Pipes Integration: Submit external scripts as Ray jobs, stream back logs and rich Dagster metadata
- ☸️ KubeRay Support: Utilize
RayJobandRayClustercustom resources in client or job submission mode (tutorial) - 🏭 Production Ready: Tested against a matrix of core dependencies, integrated with Dagster+
- ⚡ Instant Startup: Leverage
RayClusterwith Cluster Sharing for lightning-fast development cycles with zero cold start times (intended for development environments)
pip install dagster-ray- Tutorial: Step-by-step guide with examples
- API Reference: Complete API reference
- Changelog: History of changes and links to docs for older
dagster-rayreleases.
These happy users are running dagster-ray in production:
Contributions are very welcome! See CONTRIBUTING.md for details.