|
| 1 | +# Multistore |
| 2 | + |
| 3 | +A set of Rust crates for building multi-runtime S3 gateway proxies. Multistore provides the components — S3 request parsing, SigV4 authentication, backend resolution, configuration, and middleware — to assemble a proxy that sits between S3-compatible clients and backend object stores. Combine them into a native Tokio/Hyper server, a Cloudflare Worker at the edge, or your own custom runtime. |
| 4 | + |
| 5 | +> [!WARNING] |
| 6 | +> This project is in early stages and should be considered experimental. APIs and interfaces may change significantly between versions. |
| 7 | +
|
| 8 | +## Features |
| 9 | + |
| 10 | +- **Unified interface** — Single stable URL per dataset regardless of backend provider. Backend migrations are invisible to data consumers. |
| 11 | +- **Native S3 compatibility** — Works with aws-cli, boto3, DuckDB, obstore, GDAL, or any S3-compatible client. Just set the endpoint URL. |
| 12 | +- **Flexible authentication** — Anonymous access, long-lived access keys, or OIDC/STS token exchange for temporary credentials. |
| 13 | +- **Multi-runtime** — Same core logic deploys as a native server in containers or as a Cloudflare Worker at the edge. |
| 14 | +- **Zero-copy streaming** — Presigned URLs enable direct streaming between clients and backends without buffering. |
| 15 | +- **Extensible** — Pluggable traits for request resolution, configuration, and backend I/O. |
| 16 | + |
| 17 | +## Crate Layout |
| 18 | + |
| 19 | +``` |
| 20 | +crates/ |
| 21 | +├── core/ (multistore) # Runtime-agnostic: traits, S3 parsing, SigV4, registries |
| 22 | +├── metering/ (multistore-metering) # Usage metering and quota enforcement middleware |
| 23 | +├── sts/ (multistore-sts) # OIDC/STS token exchange |
| 24 | +├── oidc-provider/ (multistore-oidc-provider) # Outbound OIDC provider |
| 25 | +├── static-config/ (multistore-static-config) # Static config provider |
| 26 | +├── path-mapping/ (multistore-path-mapping) # Path mapping utilities |
| 27 | +└── cf-workers/ (multistore-cf-workers) # Cloudflare Workers runtime (WASM) |
| 28 | +
|
| 29 | +examples/ |
| 30 | +├── server/ (multistore-server) # Tokio/Hyper native server |
| 31 | +├── lambda/ (multistore-lambda) # AWS Lambda runtime |
| 32 | +└── cf-workers/ (multistore-cf-workers) # Cloudflare Workers |
| 33 | +``` |
| 34 | + |
| 35 | +## Getting Started |
| 36 | + |
| 37 | +See the [full documentation](https://developmentseed.org/multistore/getting-started/) for configuration, deployment, and usage guides. |
| 38 | + |
| 39 | +## Development |
| 40 | + |
| 41 | +```bash |
| 42 | +# Format |
| 43 | +cargo fmt |
| 44 | + |
| 45 | +# Lint |
| 46 | +cargo clippy --fix --allow-dirty --allow-staged |
| 47 | + |
| 48 | +# Check (native) |
| 49 | +cargo check |
| 50 | + |
| 51 | +# Check (WASM) |
| 52 | +cargo check -p multistore-cf-workers --target wasm32-unknown-unknown |
| 53 | +``` |
| 54 | + |
| 55 | +## License |
| 56 | + |
| 57 | +[MIT](LICENSE) |
0 commit comments