diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml new file mode 100644 index 0000000..b4283b4 --- /dev/null +++ b/.github/workflows/docs.yml @@ -0,0 +1,71 @@ +name: Deploy Docs + +on: + push: + branches: + - refactor + paths: + - 'docs/**' + pull_request: + branches: + - refactor + paths: + - 'docs/**' + workflow_dispatch: + +permissions: + contents: read + pages: write + id-token: write + +concurrency: + group: pages + cancel-in-progress: false + +jobs: + build: + name: Build Docs + runs-on: ubuntu-latest + steps: + - name: Checkout Repository + uses: actions/checkout@v4 + + - name: Setup Node.js + uses: actions/setup-node@v4 + with: + node-version: '20' + + - name: Install Mintlify CLI + run: npm install -g mintlify + + - name: Export Docs + working-directory: docs + run: mintlify export + + - name: Unzip Export + working-directory: docs + run: unzip export.zip -d site + + - name: Add SPA fallback + run: | + cp docs/site/index.html docs/site/404.html + + - name: Upload artifact + uses: actions/upload-pages-artifact@v3 + with: + path: docs/site + + deploy: + name: Deploy to GitHub Pages + if: github.event_name == 'push' + needs: build + runs-on: ubuntu-latest + + environment: + name: github-pages + url: ${{ steps.deployment.outputs.page_url }} + + steps: + - name: Deploy to GitHub Pages + id: deployment + uses: actions/deploy-pages@v4 diff --git a/Cargo.toml b/Cargo.toml index 86d073e..5bf92e4 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -14,7 +14,7 @@ members = [ [workspace.package] version = "0.1.0" edition = "2024" -license = "MIT" +license = "Apache-2.0" [workspace.dependencies] axum = "0.8" diff --git a/docs/api-reference/grpc.mdx b/docs/api-reference/grpc.mdx new file mode 100644 index 0000000..9e8297e --- /dev/null +++ b/docs/api-reference/grpc.mdx @@ -0,0 +1,474 @@ +--- +title: "gRPC API" +description: "High-performance Protocol Buffer API reference" +--- + +# gRPC API Reference + +VortexDB's gRPC API provides high-performance vector operations using Protocol Buffers over HTTP/2. + +## Connection + +| Parameter | Default | +|-----------|---------| +| Host | `localhost` | +| Port | `50051` | +| Protocol | HTTP/2 (plaintext) | + +```bash +# Test connection with grpcurl +grpcurl -plaintext localhost:50051 list +``` + +## Authentication + +All gRPC calls require the `authorization` header with your API key: + +```bash +-H "authorization: your-api-key" +``` + +The API key is set via the `GRPC_ROOT_PASSWORD` environment variable. + +--- + +## Service Definition + +```protobuf +syntax = "proto3"; +package vectordb; + +service VectorDB { + rpc InsertVector(InsertVectorRequest) returns (PointID); + rpc InsertVectorsBatch(InsertVectorsBatchRequest) returns (InsertVectorsBatchResponse); + rpc DeletePoint(PointID) returns (google.protobuf.Empty); + rpc GetPoint(PointID) returns (Point); + rpc SearchPoints(SearchRequest) returns (SearchResponse); + rpc SearchPointsBatch(SearchPointsBatchRequest) returns (SearchPointsBatchResponse); +} +``` + +--- + +## Methods + +### InsertVector + +Insert a vector with its associated payload. + + + The vector to insert. Must match the configured `DIMENSION`. + + + + Metadata associated with the vector. + + +**Request:** + +```protobuf +message InsertVectorRequest { + DenseVector vector = 1; + Payload payload = 2; +} +``` + +**Response:** + +```protobuf +message PointID { + UUID id = 1; +} +``` + +**Example:** + +```bash +grpcurl -plaintext \ + -H "authorization: secret" \ + -d '{ + "vector": {"values": [0.1, 0.2, 0.3, 0.4]}, + "payload": {"content_type": 1, "content": "Hello world"} + }' \ + localhost:50051 vectordb.VectorDB/InsertVector +``` + +**Response:** + +```json +{ + "id": { + "value": "550e8400-e29b-41d4-a716-446655440000" + } +} +``` + +--- + +### GetPoint + +Retrieve a point by its ID. + + + The unique identifier of the point. + + +**Request:** + +```protobuf +message PointID { + UUID id = 1; +} +``` + +**Response:** + +```protobuf +message Point { + PointID id = 1; + Payload payload = 2; + DenseVector vector = 3; +} +``` + +**Example:** + +```bash +grpcurl -plaintext \ + -H "authorization: secret" \ + -d '{"id": {"value": "550e8400-e29b-41d4-a716-446655440000"}}' \ + localhost:50051 vectordb.VectorDB/GetPoint +``` + +**Response:** + +```json +{ + "id": { + "id": { + "value": "550e8400-e29b-41d4-a716-446655440000" + } + }, + "payload": { + "contentType": "Text", + "content": "Hello world" + }, + "vector": { + "values": [0.1, 0.2, 0.3, 0.4] + } +} +``` + +--- + +### DeletePoint + +Delete a point by its ID. + + + The unique identifier of the point to delete. + + +**Request:** + +```protobuf +message PointID { + UUID id = 1; +} +``` + +**Response:** + +```protobuf +google.protobuf.Empty +``` + +**Example:** + +```bash +grpcurl -plaintext \ + -H "authorization: secret" \ + -d '{"id": {"value": "550e8400-e29b-41d4-a716-446655440000"}}' \ + localhost:50051 vectordb.VectorDB/DeletePoint +``` + +**Response:** + +```json +{} +``` + +--- + +### SearchPoints + +Search for the k nearest neighbors to a query vector. + + + The vector to search with. Must match the configured `DIMENSION`. + + + + The distance metric to use. + + + + Maximum number of results to return. + + + + Search breadth for HNSW. Larger values trade speed for accuracy. Defaults to the server's `HNSW_EF` setting. + + +**Request:** + +```protobuf +message SearchRequest { + DenseVector query_vector = 1; + Similarity similarity = 2; + uint64 limit = 3; + uint64 ef = 4; +} +``` + +**Response:** + +```protobuf +message SearchResponse { + repeated PointID result_point_ids = 1; +} +``` + +**Example:** + +```bash +grpcurl -plaintext \ + -H "authorization: secret" \ + -d '{ + "query_vector": {"values": [0.1, 0.2, 0.3, 0.4]}, + "similarity": 3, + "limit": 5, + "ef": 200 + }' \ + localhost:50051 vectordb.VectorDB/SearchPoints +``` + +**Response:** + +```json +{ + "resultPointIds": [ + {"id": {"value": "550e8400-e29b-41d4-a716-446655440000"}}, + {"id": {"value": "6ba7b810-9dad-11d1-80b4-00c04fd430c8"}} + ] +} +``` + +--- + +### InsertVectorsBatch + +Insert multiple vectors in a single request. + +**Request:** + +```protobuf +message InsertVectorsBatchRequest { + repeated InsertVectorRequest vectors = 1; +} +``` + +**Response:** + +```protobuf +message InsertVectorsBatchResponse { + repeated PointID ids = 1; +} +``` + +**Example:** + +```bash +grpcurl -plaintext \ + -H "authorization: secret" \ + -d '{ + "vectors": [ + {"vector": {"values": [0.1, 0.2, 0.3]}, "payload": {"content_type": 1, "content": "doc one"}}, + {"vector": {"values": [0.4, 0.5, 0.6]}, "payload": {"content_type": 1, "content": "doc two"}} + ] + }' \ + localhost:50051 vectordb.VectorDB/InsertVectorsBatch +``` + +--- + +### SearchPointsBatch + +Search against multiple query vectors in a single request. + +**Request:** + +```protobuf +message SearchPointsBatchRequest { + repeated SearchRequest queries = 1; +} +``` + +**Response:** + +```protobuf +message SearchPointsBatchResponse { + repeated SearchResponse results = 1; +} +``` + +**Example:** + +```bash +grpcurl -plaintext \ + -H "authorization: secret" \ + -d '{ + "queries": [ + {"query_vector": {"values": [0.1, 0.2, 0.3]}, "similarity": 3, "limit": 2}, + {"query_vector": {"values": [0.4, 0.5, 0.6]}, "similarity": 0, "limit": 2} + ] + }' \ + localhost:50051 vectordb.VectorDB/SearchPointsBatch +``` + +--- + +## Message Types + +### UUID + +```protobuf +message UUID { + string value = 1; // UUID v4 string +} +``` + +### DenseVector + +```protobuf +message DenseVector { + repeated float values = 1; // Vector components +} +``` + +### Point + +```protobuf +message Point { + PointID id = 1; // Unique identifier + Payload payload = 2; // Associated metadata + DenseVector vector = 3; // Vector values +} +``` + +### PointID + +```protobuf +message PointID { + UUID id = 1; +} +``` + +### Payload + +```protobuf +message Payload { + ContentType content_type = 1; // Type of content + string content = 2; // Content string +} +``` + +--- + +## Enums + +### Similarity + +Distance/similarity metric for search operations. + +| Value | Name | Description | +|-------|------|-------------| +| `0` | `Euclidean` | L2 distance (straight line) | +| `1` | `Manhattan` | L1 distance (city block) | +| `2` | `Hamming` | Count of differing elements | +| `3` | `Cosine` | Angular distance | + +### ContentType + +Type of payload content. + +| Value | Name | Description | +|-------|------|-------------| +| `0` | `Image` | Image reference or data | +| `1` | `Text` | Text content | + +--- + +## Error Codes + +| gRPC Code | Name | Description | +|-----------|------|-------------| +| `0` | `OK` | Success | +| `3` | `INVALID_ARGUMENT` | Invalid request (e.g., wrong dimensions) | +| `5` | `NOT_FOUND` | Point does not exist | +| `13` | `INTERNAL` | Server error | +| `16` | `UNAUTHENTICATED` | Invalid or missing API key | + +--- + +## Client Libraries + +### Python + +```python +from vortexdb import VortexDB, DenseVector, Payload, Similarity + +with VortexDB(grpc_url="localhost:50051", api_key="secret") as db: + # Insert + point_id = db.insert( + vector=DenseVector([0.1, 0.2, 0.3, 0.4]), + payload=Payload.text("Hello") + ) + + # Batch insert + ids = db.batch_insert(items=[ + (DenseVector([0.1, 0.2, 0.3]), Payload.text("doc one")), + (DenseVector([0.4, 0.5, 0.6]), Payload.text("doc two")), + ]) + + # Search with ef parameter + results = db.search( + vector=DenseVector([0.1, 0.2, 0.3, 0.4]), + similarity=Similarity.COSINE, + limit=5, + ef=200, + ) +``` + +### Generate Clients + +Use `protoc` to generate clients in any language: + +```bash +python -m grpc_tools.protoc \ + -I./crates/grpc/proto \ + --python_out=./client \ + --grpc_python_out=./client \ + ./crates/grpc/proto/vector-db.proto +``` + +## Next Steps + + + + REST API reference + + + Python client documentation + + diff --git a/docs/api-reference/http.mdx b/docs/api-reference/http.mdx new file mode 100644 index 0000000..26ae86f --- /dev/null +++ b/docs/api-reference/http.mdx @@ -0,0 +1,404 @@ +--- +title: "HTTP API" +description: "RESTful JSON API reference" +--- + +# HTTP API Reference + +VortexDB's HTTP API provides a RESTful interface for vector operations using JSON over HTTP/1.1. + +## Base URL + +``` +http://localhost:3000 +``` + +The port can be configured via the `HTTP_PORT` environment variable. The server binds to `127.0.0.1` by default. + + +The HTTP API has no authentication. Always deploy behind a reverse proxy or disable it with `DISABLE_HTTP=true` in production. + + +--- + +## Endpoints + +### Health Check + +Check if the server is running. + + + Returns "OK" if the server is healthy. + + + +```bash Request +curl http://localhost:3000/health +``` + +```text Response +OK +``` + + +--- + +### Root + +Verify the server is running. + + +```bash Request +curl http://localhost:3000/ +``` + +```text Response +Vector Database server is running! +``` + + +--- + +### Insert Point + + + Array of floating-point numbers representing the vector. Must match the configured `DIMENSION`. + + + + Metadata object associated with the vector. + + + + Type of content: `"Text"` or `"Image"` + + + The content string + + + + + + UUID of the created point. + + + +```bash Request +curl -X POST http://localhost:3000/points \ + -H "Content-Type: application/json" \ + -d '{ + "vector": [0.1, 0.2, 0.3, 0.4], + "payload": { + "content_type": "Text", + "content": "Hello, VortexDB!" + } + }' +``` + +```json Response (201 Created) +{ + "point_id": "550e8400-e29b-41d4-a716-446655440000" +} +``` + + +**Error Responses:** + +| Status | Description | +|--------|-------------| +| `400 Bad Request` | Invalid JSON or missing fields | +| `500 Internal Server Error` | Server error during insertion | + +--- + +### Batch Insert + +Insert multiple vectors in a single request. + + + Array of insert objects, each with `vector` and `payload` fields (same shape as single insert). + + + + Array of UUIDs for each created point, in the same order as the input. + + + +```bash Request +curl -X POST http://localhost:3000/points/batch \ + -H "Content-Type: application/json" \ + -d '{ + "vectors": [ + { + "vector": [0.1, 0.2, 0.3, 0.4], + "payload": {"content_type": "Text", "content": "Document one"} + }, + { + "vector": [0.5, 0.6, 0.7, 0.8], + "payload": {"content_type": "Text", "content": "Document two"} + } + ] + }' +``` + +```json Response (201 Created) +{ + "point_ids": [ + "550e8400-e29b-41d4-a716-446655440000", + "6ba7b810-9dad-11d1-80b4-00c04fd430c8" + ] +} +``` + + +**Error Responses:** + +| Status | Description | +|--------|-------------| +| `400 Bad Request` | Invalid JSON or missing fields | +| `500 Internal Server Error` | Server error during insertion | + +--- + +### Get Point + +Retrieve a point by its ID. + + + UUID of the point to retrieve. + + + + The point's UUID. + + + + The stored vector values. + + + + The associated payload metadata. + + + +```bash Request +curl http://localhost:3000/points/550e8400-e29b-41d4-a716-446655440000 +``` + +```json Response (200 OK) +{ + "id": "550e8400-e29b-41d4-a716-446655440000", + "vector": [0.1, 0.2, 0.3, 0.4], + "payload": { + "content_type": "Text", + "content": "Hello, VortexDB!" + } +} +``` + + +**Error Responses:** + +| Status | Description | +|--------|-------------| +| `404 Not Found` | Point does not exist | +| `500 Internal Server Error` | Server error during retrieval | + +--- + +### Delete Point + +Delete a point by its ID. + + + UUID of the point to delete. + + + +```bash Request +curl -X DELETE http://localhost:3000/points/550e8400-e29b-41d4-a716-446655440000 +``` + +```text Response (204 No Content) +(empty body) +``` + + +**Error Responses:** + +| Status | Description | +|--------|-------------| +| `500 Internal Server Error` | Server error during deletion | + +--- + +### Search Points + +Search for the k nearest neighbors to a query vector. + + + Query vector. Must match the configured `DIMENSION`. + + + + Distance metric: `"Euclidean"`, `"Manhattan"`, `"Hamming"`, or `"Cosine"` + + + + Maximum number of results to return. + + + + Array of point IDs ordered by similarity (closest first). + + + +```bash Request +curl -X POST http://localhost:3000/points/search \ + -H "Content-Type: application/json" \ + -d '{ + "vector": [0.1, 0.2, 0.3, 0.4], + "similarity": "Cosine", + "limit": 5 + }' +``` + +```json Response (200 OK) +{ + "results": [ + "550e8400-e29b-41d4-a716-446655440000", + "6ba7b810-9dad-11d1-80b4-00c04fd430c8", + "f47ac10b-58cc-4372-a567-0e02b2c3d479" + ] +} +``` + + +**Error Responses:** + +| Status | Description | +|--------|-------------| +| `400 Bad Request` | Invalid JSON or missing fields | +| `500 Internal Server Error` | Server error during search | + +--- + +### Batch Search + +Search against multiple query vectors in a single request. + + + Array of search query objects, each with `vector`, `similarity`, and `limit`. + + + + Array of result arrays, one per input query, each ordered by similarity. + + + +```bash Request +curl -X POST http://localhost:3000/points/search/batch \ + -H "Content-Type: application/json" \ + -d '{ + "queries": [ + {"vector": [0.1, 0.2, 0.3, 0.4], "similarity": "Cosine", "limit": 2}, + {"vector": [0.5, 0.6, 0.7, 0.8], "similarity": "Euclidean", "limit": 2} + ] + }' +``` + +```json Response (200 OK) +{ + "results": [ + ["550e8400-...", "6ba7b810-..."], + ["f47ac10b-...", "9a1b2c3d-..."] + ] +} +``` + + +**Error Responses:** + +| Status | Description | +|--------|-------------| +| `400 Bad Request` | Invalid JSON or missing fields | +| `500 Internal Server Error` | Server error during search | + +--- + +## Error Format + +Errors are returned as plain text with an appropriate HTTP status code: + +```bash +curl -v http://localhost:3000/points/nonexistent-id +``` + +``` +< HTTP/1.1 404 Not Found +< content-type: text/plain; charset=utf-8 +< +Point not found +``` + +--- + +## Examples + +### Complete Workflow + +```bash +# 1. Check health +curl http://localhost:3000/health +# OK + +# 2. Insert a vector +POINT_ID=$(curl -s -X POST http://localhost:3000/points \ + -H "Content-Type: application/json" \ + -d '{ + "vector": [0.1, 0.2, 0.3, 0.4], + "payload": {"content_type": "Text", "content": "First document"} + }' | jq -r '.point_id') + +echo "Created point: $POINT_ID" + +# 3. Get the point +curl http://localhost:3000/points/$POINT_ID + +# 4. Search for similar vectors +curl -X POST http://localhost:3000/points/search \ + -H "Content-Type: application/json" \ + -d '{ + "vector": [0.15, 0.25, 0.35, 0.45], + "similarity": "Cosine", + "limit": 10 + }' + +# 5. Delete the point +curl -X DELETE http://localhost:3000/points/$POINT_ID +``` + +--- + +## OpenAPI Specification + +The complete OpenAPI specification is available at: + +``` +/docs/openapi.yaml +``` + +You can import this into tools like Postman or Swagger UI for interactive API exploration. + +--- + +## Next Steps + + + + High-performance gRPC API reference + + + Python client documentation + + diff --git a/docs/api-reference/overview.mdx b/docs/api-reference/overview.mdx new file mode 100644 index 0000000..b06f3b9 --- /dev/null +++ b/docs/api-reference/overview.mdx @@ -0,0 +1,175 @@ +--- +title: "API Overview" +description: "VortexDB exposes two APIs: gRPC and HTTP" +--- + +# API Overview + +VortexDB provides two complementary APIs for different use cases: + +## Available APIs + + + + High-performance Protocol Buffer interface for production workloads + + + RESTful JSON interface for quick testing and prototyping + + + +## Comparison + +| Feature | gRPC | HTTP | +|---------|------|------| +| **Protocol** | HTTP/2 + Protobuf | HTTP/1.1 + JSON | +| **Default Port** | 50051 | 3000 | +| **Authentication** | API key required | None | +| **Performance** | Higher throughput | Lower latency for simple requests | +| **Client Libraries** | Auto-generated | Any HTTP client | +| **Best For** | Production, SDKs | Debugging, curl | + +## Authentication + +### gRPC + +The gRPC API requires authentication via the `authorization` header: + +```bash +# Using grpcurl +grpcurl -plaintext \ + -H "authorization: your-api-key" \ + localhost:50051 vectordb.VectorDB/GetPoint +``` + +```python +# Python SDK +db = VortexDB( + grpc_url="localhost:50051", + api_key="your-api-key" +) +``` + +### HTTP + +The HTTP API does not require authentication by default. It's designed for trusted internal networks and development. + + +In production, always deploy the HTTP API behind a reverse proxy with authentication, or disable it entirely using `DISABLE_HTTP=true`. + + +## Common Operations + +Both APIs support the same core operations: + +| Operation | gRPC Method | HTTP Endpoint | +|-----------|-------------|---------------| +| Insert vector | `InsertVector` | `POST /points` | +| Batch insert | `InsertVectorsBatch` | `POST /points/batch` | +| Get point | `GetPoint` | `GET /points/:id` | +| Delete point | `DeletePoint` | `DELETE /points/:id` | +| Search vectors | `SearchPoints` | `POST /points/search` | +| Batch search | `SearchPointsBatch` | `POST /points/search/batch` | +| Health check | - | `GET /health` | + +## Error Handling + +### gRPC Error Codes + +| Code | Name | Description | +|------|------|-------------| +| `0` | OK | Success | +| `3` | INVALID_ARGUMENT | Invalid request parameters | +| `5` | NOT_FOUND | Point not found | +| `13` | INTERNAL | Server error | +| `16` | UNAUTHENTICATED | Invalid or missing API key | + +### HTTP Status Codes + +| Code | Description | +|------|-------------| +| `200` | Success | +| `201` | Created (for insert) | +| `204` | No Content (for delete) | +| `400` | Bad Request | +| `404` | Not Found | +| `500` | Internal Server Error | + +## Data Types + +### Vector + +A dense vector of floating-point values: + + + + ```protobuf + message DenseVector { + repeated float values = 1; + } + ``` + + + ```json + { + "vector": [0.1, 0.2, 0.3, 0.4] + } + ``` + + + +### Payload + +Metadata attached to vectors: + + + + ```protobuf + enum ContentType { + Image = 0; + Text = 1; + } + + message Payload { + ContentType content_type = 1; + string content = 2; + } + ``` + + + ```json + { + "payload": { + "content_type": "Text", + "content": "Hello, world!" + } + } + ``` + + + +### Similarity Metric + +Distance function for search: + +| Value | gRPC Enum | HTTP String | +|-------|-----------|-------------| +| Euclidean (L2) | `0` | `"Euclidean"` | +| Manhattan (L1) | `1` | `"Manhattan"` | +| Hamming | `2` | `"Hamming"` | +| Cosine | `3` | `"Cosine"` | + +## Rate Limiting + +VortexDB does not implement built-in rate limiting. For production deployments, use a reverse proxy or API gateway to enforce limits. + +## Next Steps + + + + Complete gRPC API documentation + + + Complete HTTP API documentation + + diff --git a/docs/concepts/architecture.mdx b/docs/concepts/architecture.mdx new file mode 100644 index 0000000..13d5307 --- /dev/null +++ b/docs/concepts/architecture.mdx @@ -0,0 +1,184 @@ +--- +title: "Architecture" +description: "Understanding VortexDB's modular architecture" +--- + +# Architecture + +VortexDB is a high-performance vector database built in Rust, designed with modularity and flexibility at its core. + +## Crate Structure + +VortexDB is organized as a Rust workspace with the following crates: + +| Crate | Purpose | +|-------|---------| +| `server` | Main entry point, configuration, server startup | +| `api` | Core database logic and error handling | +| `grpc` | Protocol Buffers definitions and gRPC service | +| `http` | REST API handlers using Axum | +| `index` | Vector indexing algorithms (Flat, KD-Tree, HNSW) | +| `storage` | Persistence backends (InMemory, RocksDB) | +| `snapshot` | Point-in-time backup and restore | +| `defs` | Shared type definitions | +| `tui` | Terminal user interface | + +## Transport Layers + +VortexDB exposes two APIs that share the same underlying database: + +### gRPC API + +The **gRPC layer** is the primary high-performance interface: + +- **Protocol**: HTTP/2 with Protocol Buffers +- **Port**: 50051 (default) +- **Authentication**: API key via `authorization` header +- **Use cases**: Production workloads, SDKs, high-throughput scenarios + +```protobuf +service VectorDB { + rpc InsertVector(InsertVectorRequest) returns (PointID); + rpc DeletePoint(PointID) returns (google.protobuf.Empty); + rpc GetPoint(PointID) returns (Point); + rpc SearchPoints(SearchRequest) returns (SearchResponse); +} +``` + +### HTTP API + +The **HTTP layer** provides a RESTful interface: + +- **Protocol**: HTTP/1.1 with JSON +- **Port**: 3000 (default) +- **Authentication**: None (designed for internal/trusted networks) +- **Use cases**: Quick testing, curl commands, prototyping + +| Method | Endpoint | Description | +|--------|----------|-------------| +| `GET` | `/` | Root endpoint | +| `GET` | `/health` | Health check | +| `POST` | `/points` | Insert a point | +| `GET` | `/points/:id` | Get a point by ID | +| `DELETE` | `/points/:id` | Delete a point | +| `POST` | `/points/search` | Search for similar vectors | + +### When to Use Which? + + + + - Building production applications + - Using the Python SDK (it uses gRPC) + - Need authentication + - Processing high volumes of requests + - Want strongly-typed client libraries + + + - Quick prototyping with curl + - Integrating with systems that don't support gRPC + - Debugging and testing + - Building browser-based tools + + + +## Storage Layer + +The storage layer persists vectors and their payloads using a trait-based design: + +```rust +pub trait StorageEngine: Send + Sync { + fn insert(&self, vector: DenseVector, payload: Payload) -> Result; + fn get(&self, point_id: PointId) -> Result>; + fn delete(&self, point_id: PointId) -> Result; + fn checkpoint_at(&self, path: &Path) -> Result; + fn restore_checkpoint(&mut self, checkpoint: &StorageCheckpoint) -> Result<()>; +} +``` + +### Available Backends + +| Backend | Description | Use Case | +|---------|-------------|----------| +| **InMemory** | Stores data in RAM | Development, testing, ephemeral workloads | +| **RocksDB** | LSM-tree persistent storage | Production deployments | + +Set the backend via the `STORAGE_TYPE` environment variable: +```bash +STORAGE_TYPE=rocksdb # or 'inmemory' +``` + +## Index Layer + +The index layer provides fast similarity search. See [Indexers](/concepts/indexers) for details on choosing the right index. + +```rust +pub trait VectorIndex: Send + Sync { + fn insert(&mut self, vector: IndexedVector) -> Result<()>; + fn delete(&mut self, point_id: PointId) -> Result; + fn search(&self, query: DenseVector, similarity: Similarity, k: usize) -> Result>; +} +``` + +## Data Flow + +Here's what happens when you insert a vector: + + + + Client sends insert request via gRPC or HTTP + + + Server validates vector dimensions match configuration + + + Vector and payload are persisted to the storage backend + + + Vector is added to the index for fast searching + + + Server returns the generated point ID to client + + + +## Configuration + +VortexDB is configured via environment variables: + +| Variable | Required | Default | Description | +|----------|----------|---------|-------------| +| `GRPC_ROOT_PASSWORD` | Yes | - | Authentication password for gRPC | +| `DIMENSION` | Yes | - | Vector dimensionality | +| `DATA_PATH` | No | system temp dir | Directory for persistent storage | +| `HTTP_HOST` | No | `127.0.0.1` | HTTP server bind address | +| `HTTP_PORT` | No | `3000` | HTTP server port | +| `GRPC_HOST` | No | `127.0.0.1` | gRPC server bind address | +| `GRPC_PORT` | No | `50051` | gRPC server port | +| `STORAGE_TYPE` | No | `inmemory` | Storage backend: `inmemory` or `rocksdb` | +| `INDEX_TYPE` | No | `flat` | Index algorithm: `flat`, `kdtree`, or `hnsw` | +| `SIMILARITY` | No | `cosine` | Default metric: `cosine`, `euclidean`, `manhattan`, or `hamming` | +| `LOGGING` | No | `true` | Enable logging | +| `DISABLE_HTTP` | No | `false` | Run gRPC only | +| `HNSW_M` | No | `16` | HNSW max connections per layer | +| `HNSW_M0` | No | `2 * HNSW_M` | HNSW max connections for layer 0 | +| `HNSW_EF_CONSTRUCTION` | No | `200` | HNSW search breadth during construction | +| `HNSW_EF` | No | `100` | HNSW default search breadth at query time | + +## Thread Safety + +VortexDB is designed for concurrent access: + +- The **storage layer** uses `Arc` for thread-safe reference counting +- The **index layer** uses `RwLock` for concurrent reads with exclusive writes +- Both gRPC and HTTP handlers are fully async using Tokio + +## Next Steps + + + + Learn about index algorithms and when to use each + + + Understand backup and restore mechanisms + + diff --git a/docs/concepts/indexers.mdx b/docs/concepts/indexers.mdx new file mode 100644 index 0000000..bbf8056 --- /dev/null +++ b/docs/concepts/indexers.mdx @@ -0,0 +1,147 @@ +--- +title: "Indexers" +description: "Choosing the right vector index for your use case" +--- + +# Indexers + +VortexDB supports multiple indexing algorithms, each optimized for different use cases. The index determines how vectors are organized for similarity search. + +## Flat Index + +The **Flat** index performs brute-force exhaustive search by computing distances to every vector. + +### When to Use + +- **Dataset size**: < 10,000 vectors +- **Requirements**: Need exact/guaranteed results +- **Use cases**: Testing, prototyping, small production workloads + +```bash +INDEX_TYPE=flat +``` + +## KD-Tree Index + +The **KD-Tree** (k-dimensional tree) is a space-partitioning data structure that recursively divides the vector space. + +At each level, the tree splits data along a different dimension (cycling through x, y, z, ...). + +### When to Use + +- **Vector dimensions**: < 20 dimensions +- **Dataset size**: Thousands to hundreds of thousands of vectors +- **Use cases**: Geographic data, low-dimensional embeddings, spatial queries + +```bash +INDEX_TYPE=kdtree +``` + + +For high-dimensional embeddings (e.g., 384, 768, 1536 dimensions common in ML), KD-Tree is not recommended. Use HNSW instead. + + +## HNSW Index + +**HNSW** (Hierarchical Navigable Small World) is a state-of-the-art approximate nearest neighbor algorithm based on proximity graphs. It constructs a multi-layered graph where each layer represents a different level of granularity, enabling efficient navigation from coarse to fine-grained similarity search. + +Check out this [blog post](https://blog.sdslabs.co/2026/03/hnsw-index) for more theoretical details and [this blog](https://blog.sdslabs.co/2026/03/hnsw-indexp2) covering implementation in VortexDB. + +### When to Use + +- **Vector dimensions**: Any, but especially > 20 dimensions +- **Dataset size**: 100,000+ vectors +- **Use cases**: Semantic search, recommendation systems, RAG applications + +```bash +INDEX_TYPE=hnsw +``` + +## Distance Metrics + +All indexes support four distance/similarity metrics: + + + + Measures the angle between two vectors, ignoring magnitude. + + $$d = 1 - \frac{A \cdot B}{\|A\| \|B\|}$$ + + - **Range**: 0 (identical) to 2 (opposite) + - **Best for**: Text embeddings, normalized vectors + + ```python + similarity=Similarity.COSINE + ``` + + + L2 distance—the straight-line distance between points. + + $$d = \sqrt{\sum_{i=1}^{n} (a_i - b_i)^2}$$ + + - **Range**: 0 (identical) to ∞ + - **Best for**: General purpose + + ```python + similarity=Similarity.EUCLIDEAN + ``` + + + L1 distance—the sum of absolute differences. + + $$d = \sum_{i=1}^{n} |a_i - b_i|$$ + + - **Range**: 0 (identical) to ∞ + - **Best for**: Sparse vectors, grid-based data + + ```python + similarity=Similarity.MANHATTAN + ``` + + + Counts positions where elements differ. + + - **Range**: 0 (identical) to n (completely different) + - **Best for**: Binary vectors, categorical data + + ```python + similarity=Similarity.HAMMING + ``` + + + +## Choosing an Index + +Use this decision tree: + +| Vectors | Dimensions | Recommended Index | +|---------|------------|-------------------| +| < 10,000 | Any | **Flat** | +| 10K - 100K | < 20 | **KD-Tree** | +| 10K - 100K | ≥ 20 | **HNSW** | +| > 100K | Any | **HNSW** | + +## Configuration Example + +```bash +# For a semantic search application with OpenAI embeddings +DIMENSION=1536 +INDEX_TYPE=hnsw +STORAGE_TYPE=rocksdb + +# For a small prototype with sentence-transformers +DIMENSION=384 +INDEX_TYPE=flat +STORAGE_TYPE=inmemory +``` + +## Next Steps + + + + Learn how to backup and restore your index + + + Explore the complete API + + diff --git a/docs/concepts/snapshots.mdx b/docs/concepts/snapshots.mdx new file mode 100644 index 0000000..5443828 --- /dev/null +++ b/docs/concepts/snapshots.mdx @@ -0,0 +1,196 @@ +--- +title: "Snapshots" +description: "Backup and restore your vector database" +--- + +# Snapshots + +VortexDB's snapshot system provides point-in-time backups of your entire database, including both the index topology and stored data. + +## Overview + +A snapshot captures: +- **Index topology**: The structure of your index (graph edges, tree nodes, etc.) +- **Index metadata**: Configuration specific to the index type +- **Storage data**: All vectors and their payloads +- **Database metadata**: Dimensions, timestamps, version info + +## How It Works + +### Creating a Snapshot + +The snapshot process involves two parallel operations: + + + + The index serializes its topology and metadata into binary format: + ```rust + pub trait SerializableIndex { + fn serialize_topology(&self) -> Result>; + fn serialize_metadata(&self) -> Result>; + fn snapshot(&self) -> Result; + } + ``` + + + The storage backend creates a consistent checkpoint: + ```rust + fn checkpoint_at(&self, path: &Path) -> Result; + ``` + For RocksDB, this uses the native checkpoint API for efficiency. + + + A manifest file is created with: + - Snapshot UUID + - Timestamp + - Parser version (for compatibility) + - SHA-256 checksums of all files + + + All files are bundled into a compressed tarball (`.tar.gz`). + + + +### Restoring a Snapshot + + + + The tarball is extracted to a temporary directory. + + + All file checksums are verified against the manifest. + + + The parser version is checked for compatibility. + + + The storage checkpoint is restored first. + + + The index is deserialized from topology and metadata, then populated with vectors from storage. + + + +## Index-Specific Serialization + +Each index type has its own serialization format: + + + + - **Topology**: List of point IDs in insertion order + - **Metadata**: Minimal (just magic bytes) + - **Restore**: O(n) - simply rebuild the vector list + + + - **Topology**: Tree structure with node relationships + - **Metadata**: Dimension count, tree depth + - **Restore**: O(n) - rebuild tree structure, populate vectors + + + - **Topology**: Multi-layer graph with all edges + - **Metadata**: Level multiplier, max connections, entry point + - **Restore**: O(n) - rebuild graph layers, populate vectors + + + +## Snapshot Data Structures + +### Snapshot Object + +```rust +pub struct Snapshot { + pub id: Uuid, // Unique identifier + pub date: SystemTime, // Creation timestamp + pub sem_ver: Version, // Parser version + pub index_snapshot: IndexSnapshot, // Index data + pub storage_snapshot: StorageCheckpoint, // Storage data + pub dimensions: usize, // Vector dimensions +} +``` + +### Index Snapshot + +```rust +pub struct IndexSnapshot { + pub index_type: IndexType, // Flat, KDTree, or HNSW + pub magic: Magic, // 4-byte identifier + pub topology_b: Vec, // Serialized graph/tree + pub metadata_b: Vec, // Index-specific config +} +``` + +### Storage Checkpoint + +```rust +pub struct StorageCheckpoint { + pub path: PathBuf, // Path to checkpoint files + pub storage_type: StorageType, // InMemory or RocksDB +} +``` + +## Snapshot Engine + +The `SnapshotEngine` manages snapshot creation and restoration: + +### Registry Interface + +```rust +pub trait SnapshotRegistry: Send + Sync { + // Add a new snapshot to the registry + fn add_snapshot(&mut self, path: &Path) -> Result; + + // List snapshots with pagination + fn list_snapshots(&mut self, limit: usize, offset: usize) -> Result; + + // Get the most recent snapshot + fn get_latest_snapshot(&mut self) -> Result; + + // Get metadata for a specific snapshot + fn get_metadata(&mut self, id: String) -> Result; + + // Remove a snapshot + fn remove_snapshot(&mut self, id: String) -> Result; + + // Load and restore a snapshot + fn load(&mut self, id: String, path: &Path) -> Result; +} +``` + +## Manifest Format + +The manifest (`manifest.json`) contains all metadata needed for restore: + +```json +{ + "snapshot_id": "550e8400-e29b-41d4-a716-446655440000", + "created_at": "2026-03-31T12:00:00Z", + "parser_version": "1.0.0", + "dimensions": 384, + "index_type": "HNSW", + "files": { + "index_metadata": { + "filename": "hnsw-index-meta.bin", + "checksum": "sha256:abc123..." + }, + "index_topology": { + "filename": "hnsw-index-topo.bin", + "checksum": "sha256:def456..." + }, + "storage": { + "filename": "rocksdb-checkpoint/", + "checksum": "sha256:789ghi..." + } + } +} +``` + +## Next Steps + + + + Understand how components fit together + + + Choose the right index for your use case + + diff --git a/docs/docs.json b/docs/docs.json new file mode 100644 index 0000000..81adb4e --- /dev/null +++ b/docs/docs.json @@ -0,0 +1,121 @@ +{ + "$schema": "https://mintlify.com/docs.json", + "theme": "mint", + "name": "VortexDB", + "colors": { + "primary": "#00d4aa", + "light": "#00e6b8", + "dark": "#00b894" + }, + "favicon": "/assets/favicon.svg", + "navigation": { + "tabs": [ + { + "tab": "Documentation", + "groups": [ + { + "group": "Getting Started", + "pages": [ + "getting-started/installation", + "getting-started/quickstart" + ] + }, + { + "group": "Concepts", + "pages": [ + "concepts/architecture", + "concepts/indexers", + "concepts/snapshots" + ] + } + ] + }, + { + "tab": "API Reference", + "groups": [ + { + "group": "API Reference", + "pages": [ + "api-reference/overview", + "api-reference/grpc", + "api-reference/http" + ] + } + ] + }, + { + "tab": "SDK", + "groups": [ + { + "group": "SDK", + "pages": [ + "sdk/reference", + "sdk/examples" + ] + } + ] + } + ], + "global": { + "anchors": [ + { + "anchor": "GitHub", + "href": "https://github.com/sdslabs/VortexDB", + "icon": "github" + }, + { + "anchor": "Community", + "href": "https://discord.gg/dXkVEgTPu9", + "icon": "discord" + }, + { + "anchor": "SDSLabs", + "href": "https://sdslabs.co", + "icon": "globe" + } + ] + } + }, + "logo": { + "light": "/logo_horizontal.svg", + "dark": "/logo_horizontal.svg" + }, + "api": { + "openapi": "openapi.yaml", + "mdx": { + "server": "http://localhost:3000" + } + }, + "appearance": { + "default": "dark" + }, + "background": { + "color": { + "dark": "#0a0a0f" + } + }, + "navbar": { + "links": [ + { + "label": "SDSLabs", + "href": "https://sdslabs.co" + }, + { + "label": "Discord", + "href": "https://discord.gg/dXkVEgTPu9" + } + ], + "primary": { + "type": "button", + "label": "Github", + "href": "https://github.com/sdslabs/VortexDB" + } + }, + "footer": { + "socials": { + "github": "https://github.com/sdslabs/VortexDB", + "discord": "https://discord.gg/dXkVEgTPu9", + "website": "https://sdslabs.co" + } + } +} \ No newline at end of file diff --git a/docs/getting-started/installation.mdx b/docs/getting-started/installation.mdx new file mode 100644 index 0000000..5cb05e7 --- /dev/null +++ b/docs/getting-started/installation.mdx @@ -0,0 +1,156 @@ +--- +title: "Installation" +description: "Get VortexDB up and running in minutes" +--- + +# Installation + +VortexDB can be installed using Docker (recommended) or built from source. This guide covers both methods. + +## Prerequisites + + + + Docker 20.10+ and Docker Compose v2 + + + Rust 1.88+, protobuf-compiler, clang + + + +## Docker Installation (Recommended) + +The fastest way to get started is using Docker: + +```bash +# Clone the repository +git clone https://github.com/sdslabs/VortexDB.git +cd VortexDB + +# Copy the environment template +cp .env.example .env +``` + +### Configure Environment + +Edit the `.env` file with your settings: + +```bash +# Required settings +GRPC_ROOT_PASSWORD=your-secure-password +DIMENSION=384 # Vector dimension (e.g., 384 for MiniLM embeddings) +DATA_PATH=/data + +# Optional settings (with defaults) +HTTP_HOST=127.0.0.1 +HTTP_PORT=3000 +GRPC_HOST=127.0.0.1 +GRPC_PORT=50051 +STORAGE_TYPE=inmemory # inmemory | rocksdb +INDEX_TYPE=flat # flat | kdtree | hnsw +SIMILARITY=cosine # cosine | euclidean | manhattan | hamming +LOGGING=true +DISABLE_HTTP=false +``` + +### Start VortexDB + +```bash +docker compose up +``` + + +Use `docker compose up --build` after making code changes to rebuild the image. + + +VortexDB is now running: +- **HTTP API**: http://localhost:3000 +- **gRPC API**: localhost:50051 + +## Building from Source + +### Install Dependencies + + + + ```bash + sudo apt-get update && sudo apt-get install -y \ + protobuf-compiler \ + clang \ + libclang-dev \ + llvm-dev \ + build-essential + ``` + + + ```bash + brew install protobuf llvm + ``` + + + ```bash + sudo pacman -S protobuf clang llvm + ``` + + + +### Build and Run + +```bash +# Clone the repository +git clone https://github.com/sdslabs/VortexDB.git +cd VortexDB + +# Build in release mode +cargo build --release + +# Run the server +./target/release/server +``` + +## Python SDK Installation + +Install the Python client to interact with VortexDB: + +```bash +pip install vortexdb +``` + +Or install from source: + +```bash +cd client/python +pip install -e . +``` + + +The Python SDK requires Python 3.9+ and communicates with VortexDB via gRPC. + + +## Verify Installation + +Check that VortexDB is running correctly: + + +```bash Health Check (HTTP) +curl http://localhost:3000/health +# Response: OK +``` + +```python Health Check (Python) +from vortexdb import VortexDB + +db = VortexDB( + grpc_url="localhost:50051", + api_key="your-secure-password" +) +print("Connected successfully!") +db.close() +``` + + +## Next Steps + + + Insert your first vector in 60 seconds + diff --git a/docs/getting-started/quickstart.mdx b/docs/getting-started/quickstart.mdx new file mode 100644 index 0000000..8c9622c --- /dev/null +++ b/docs/getting-started/quickstart.mdx @@ -0,0 +1,225 @@ +--- +title: "Quickstart" +description: "Insert your first vector in 60 seconds" +--- + +# Quickstart + +This guide will have you inserting and searching vectors in under 60 seconds. + +## Prerequisites + +Make sure VortexDB is running (see [Installation](/getting-started/installation)). + +## Insert Your First Vector + + + + ```python + from vortexdb import VortexDB, DenseVector, Payload, Similarity + + # Connect to VortexDB + db = VortexDB( + grpc_url="localhost:50051", + api_key="your-secure-password" + ) + + # Insert a vector with a text payload + point_id = db.insert( + vector=DenseVector([0.1, 0.2, 0.3, 0.4]), + payload=Payload.text("Hello, VortexDB!") + ) + print(f"Inserted point: {point_id}") + + # Clean up + db.close() + ``` + + + ```bash + curl -X POST http://localhost:3000/points \ + -H "Content-Type: application/json" \ + -d '{ + "vector": [0.1, 0.2, 0.3, 0.4], + "payload": { + "content_type": "Text", + "content": "Hello, VortexDB!" + } + }' + ``` + + Response: + ```json + { + "point_id": "550e8400-e29b-41d4-a716-446655440000" + } + ``` + + + ```protobuf + // Using grpcurl + grpcurl -plaintext \ + -H "authorization: your-secure-password" \ + -d '{ + "vector": {"values": [0.1, 0.2, 0.3, 0.4]}, + "payload": {"content_type": 1, "content": "Hello, VortexDB!"} + }' \ + localhost:50051 vectordb.VectorDB/InsertVector + ``` + + + +## Search for Similar Vectors + + + + ```python + from vortexdb import VortexDB, DenseVector, Similarity + + db = VortexDB( + grpc_url="localhost:50051", + api_key="your-secure-password" + ) + + # Search for 5 most similar vectors using cosine similarity + results = db.search( + vector=DenseVector([0.1, 0.2, 0.3, 0.4]), + similarity=Similarity.COSINE, + limit=5 + ) + + print(f"Found {len(results)} similar vectors:") + for point_id in results: + print(f" - {point_id}") + + db.close() + ``` + + + ```bash + curl -X POST http://localhost:3000/points/search \ + -H "Content-Type: application/json" \ + -d '{ + "vector": [0.1, 0.2, 0.3, 0.4], + "similarity": "Cosine", + "limit": 5 + }' + ``` + + Response: + ```json + { + "results": [ + "550e8400-e29b-41d4-a716-446655440000", + "6ba7b810-9dad-11d1-80b4-00c04fd430c8" + ] + } + ``` + + + +## Retrieve a Point + + + + ```python + # Get the point you just inserted + point = db.get(point_id=point_id) + + if point: + print(point.pretty()) + # Output: + # Point ID: 550e8400-e29b-41d4-a716-446655440000 + # Vector: [0.1, 0.2, 0.3, 0.4] + # Payload: Hello, VortexDB! + ``` + + + ```bash + curl http://localhost:3000/points/550e8400-e29b-41d4-a716-446655440000 + ``` + + Response: + ```json + { + "id": "550e8400-e29b-41d4-a716-446655440000", + "vector": [0.1, 0.2, 0.3, 0.4], + "payload": { + "content_type": "Text", + "content": "Hello, VortexDB!" + } + } + ``` + + + +## Delete a Point + + + + ```python + db.delete(point_id=point_id) + print("Point deleted successfully") + ``` + + + ```bash + curl -X DELETE http://localhost:3000/points/550e8400-e29b-41d4-a716-446655440000 + ``` + + + +## Complete Example + +Here's a complete example using the Python SDK with context manager: + +```python +from vortexdb import VortexDB, DenseVector, Payload, Similarity + +# Using context manager for automatic cleanup +with VortexDB(grpc_url="localhost:50051", api_key="secret") as db: + # Insert some vectors + vectors = [ + ([0.1, 0.2, 0.3, 0.4], "First document"), + ([0.2, 0.3, 0.4, 0.5], "Second document"), + ([0.9, 0.8, 0.7, 0.6], "Third document"), + ] + + point_ids = [] + for vec, text in vectors: + pid = db.insert( + vector=DenseVector(vec), + payload=Payload.text(text) + ) + point_ids.append(pid) + print(f"Inserted: {text} -> {pid}") + + # Search for vectors similar to the first one + results = db.search( + vector=DenseVector([0.15, 0.25, 0.35, 0.45]), + similarity=Similarity.COSINE, + limit=2 + ) + + print(f"\nTop 2 similar vectors:") + for pid in results: + point = db.get(point_id=pid) + print(f" - {point.payload.content}") +``` + +## Next Steps + + + + Understand how VortexDB works under the hood + + + Choose the right index for your use case + + + Python client documentation + + + Explore the complete API + + diff --git a/docs/logo_horizontal.svg b/docs/logo_horizontal.svg new file mode 100644 index 0000000..89c7ecb --- /dev/null +++ b/docs/logo_horizontal.svg @@ -0,0 +1,8 @@ + + + + + + + + diff --git a/docs/openapi.yaml b/docs/openapi.yaml new file mode 100644 index 0000000..f1b3a69 --- /dev/null +++ b/docs/openapi.yaml @@ -0,0 +1,470 @@ +openapi: 3.0.3 +info: + title: VortexDB HTTP API + description: | + RESTful API for VortexDB - a high-performance vector database built in Rust. + + This API provides CRUD operations for vector points and similarity search. + version: "1.0.0" + license: + name: Apache 2.0 + url: https://www.apache.org/licenses/LICENSE-2.0 + contact: + name: SDSLabs + url: https://github.com/sdslabs/VortexDB + +servers: + - url: http://localhost:3000 + description: Local development server + +tags: + - name: Health + description: Server health endpoints + - name: Points + description: Vector point operations + +paths: + /: + get: + tags: + - Health + summary: Root endpoint + description: Verify the server is running + operationId: getRoot + responses: + '200': + description: Server is running + content: + text/plain: + schema: + type: string + example: Vector Database server is running! + + /health: + get: + tags: + - Health + summary: Health check + description: Check if the server is healthy + operationId: getHealth + responses: + '200': + description: Server is healthy + content: + text/plain: + schema: + type: string + example: OK + + /points: + post: + tags: + - Points + summary: Insert a point + description: Insert a new vector with its associated payload + operationId: insertPoint + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/InsertRequest' + example: + vector: [0.1, 0.2, 0.3, 0.4] + payload: + content_type: Text + content: Hello, VortexDB! + responses: + '201': + description: Point created successfully + content: + application/json: + schema: + $ref: '#/components/schemas/InsertResponse' + example: + point_id: 550e8400-e29b-41d4-a716-446655440000 + '400': + description: Invalid request + content: + text/plain: + schema: + type: string + example: Invalid vector dimensions + '500': + description: Server error + content: + text/plain: + schema: + type: string + example: Failed to insert point + + /points/batch: + post: + tags: + - Points + summary: Batch insert points + description: Insert multiple vectors in a single request + operationId: batchInsertPoints + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/BatchInsertRequest' + example: + vectors: + - vector: [0.1, 0.2, 0.3, 0.4] + payload: + content_type: Text + content: Document one + - vector: [0.5, 0.6, 0.7, 0.8] + payload: + content_type: Text + content: Document two + responses: + '201': + description: Points created successfully + content: + application/json: + schema: + $ref: '#/components/schemas/BatchInsertResponse' + example: + point_ids: + - 550e8400-e29b-41d4-a716-446655440000 + - 6ba7b810-9dad-11d1-80b4-00c04fd430c8 + '400': + description: Invalid request + content: + text/plain: + schema: + type: string + '500': + description: Server error + content: + text/plain: + schema: + type: string + + /points/{point_id}: + get: + tags: + - Points + summary: Get a point + description: Retrieve a point by its ID + operationId: getPoint + parameters: + - name: point_id + in: path + required: true + description: UUID of the point + schema: + type: string + format: uuid + example: 550e8400-e29b-41d4-a716-446655440000 + responses: + '200': + description: Point found + content: + application/json: + schema: + $ref: '#/components/schemas/Point' + example: + id: 550e8400-e29b-41d4-a716-446655440000 + vector: [0.1, 0.2, 0.3, 0.4] + payload: + content_type: Text + content: Hello, VortexDB! + '404': + description: Point not found + content: + text/plain: + schema: + type: string + example: Point not found + '500': + description: Server error + content: + text/plain: + schema: + type: string + example: Failed to get point + + delete: + tags: + - Points + summary: Delete a point + description: Delete a point by its ID + operationId: deletePoint + parameters: + - name: point_id + in: path + required: true + description: UUID of the point + schema: + type: string + format: uuid + example: 550e8400-e29b-41d4-a716-446655440000 + responses: + '204': + description: Point deleted successfully + '500': + description: Server error + content: + text/plain: + schema: + type: string + example: Failed to delete point + + /points/search: + post: + tags: + - Points + summary: Search for similar vectors + description: Find the k nearest neighbors to a query vector + operationId: searchPoints + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/SearchRequest' + example: + vector: [0.1, 0.2, 0.3, 0.4] + similarity: Cosine + limit: 5 + responses: + '200': + description: Search results + content: + application/json: + schema: + $ref: '#/components/schemas/SearchResponse' + example: + results: + - 550e8400-e29b-41d4-a716-446655440000 + - 6ba7b810-9dad-11d1-80b4-00c04fd430c8 + '400': + description: Invalid request + content: + text/plain: + schema: + type: string + example: Invalid vector dimensions + '500': + description: Server error + content: + text/plain: + schema: + type: string + example: Search failed + + /points/search/batch: + post: + tags: + - Points + summary: Batch search for similar vectors + description: Search against multiple query vectors in a single request + operationId: batchSearchPoints + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/BatchSearchRequest' + example: + queries: + - vector: [0.1, 0.2, 0.3, 0.4] + similarity: Cosine + limit: 2 + - vector: [0.5, 0.6, 0.7, 0.8] + similarity: Euclidean + limit: 2 + responses: + '200': + description: Batch search results + content: + application/json: + schema: + $ref: '#/components/schemas/BatchSearchResponse' + example: + results: + - - 550e8400-e29b-41d4-a716-446655440000 + - 6ba7b810-9dad-11d1-80b4-00c04fd430c8 + - - f47ac10b-58cc-4372-a567-0e02b2c3d479 + '400': + description: Invalid request + content: + text/plain: + schema: + type: string + '500': + description: Server error + content: + text/plain: + schema: + type: string + +components: + schemas: + DenseVector: + type: array + items: + type: number + format: float + description: Array of floating-point values representing the vector + example: [0.1, 0.2, 0.3, 0.4] + + ContentType: + type: string + enum: + - Text + - Image + description: Type of payload content + example: Text + + Payload: + type: object + required: + - content_type + - content + properties: + content_type: + $ref: '#/components/schemas/ContentType' + content: + type: string + description: Content string + example: Hello, VortexDB! + + Similarity: + type: string + enum: + - Euclidean + - Manhattan + - Hamming + - Cosine + description: | + Distance/similarity metric: + - **Euclidean**: L2 distance (straight line) + - **Manhattan**: L1 distance (city block) + - **Hamming**: Count of differing elements + - **Cosine**: Angular distance (1 - cosine similarity) + example: Cosine + + Point: + type: object + required: + - id + - vector + - payload + properties: + id: + type: string + format: uuid + description: Unique identifier for the point + example: 550e8400-e29b-41d4-a716-446655440000 + vector: + $ref: '#/components/schemas/DenseVector' + payload: + $ref: '#/components/schemas/Payload' + + InsertRequest: + type: object + required: + - vector + - payload + properties: + vector: + $ref: '#/components/schemas/DenseVector' + payload: + $ref: '#/components/schemas/Payload' + + InsertResponse: + type: object + required: + - point_id + properties: + point_id: + type: string + format: uuid + description: UUID of the created point + example: 550e8400-e29b-41d4-a716-446655440000 + + BatchInsertRequest: + type: object + required: + - vectors + properties: + vectors: + type: array + items: + $ref: '#/components/schemas/InsertRequest' + description: Array of insert objects + + BatchInsertResponse: + type: object + required: + - point_ids + properties: + point_ids: + type: array + items: + type: string + format: uuid + description: UUIDs of the created points, in input order + + SearchRequest: + type: object + required: + - vector + - similarity + - limit + properties: + vector: + $ref: '#/components/schemas/DenseVector' + similarity: + $ref: '#/components/schemas/Similarity' + limit: + type: integer + minimum: 1 + description: Maximum number of results to return + example: 5 + + SearchResponse: + type: object + required: + - results + properties: + results: + type: array + items: + type: string + format: uuid + description: Array of point IDs ordered by similarity (closest first) + example: + - 550e8400-e29b-41d4-a716-446655440000 + - 6ba7b810-9dad-11d1-80b4-00c04fd430c8 + + BatchSearchRequest: + type: object + required: + - queries + properties: + queries: + type: array + items: + $ref: '#/components/schemas/SearchRequest' + description: Array of search query objects + + BatchSearchResponse: + type: object + required: + - results + properties: + results: + type: array + items: + type: array + items: + type: string + format: uuid + description: One result array per input query diff --git a/docs/sdk/examples.mdx b/docs/sdk/examples.mdx new file mode 100644 index 0000000..0ba249c --- /dev/null +++ b/docs/sdk/examples.mdx @@ -0,0 +1,144 @@ +--- +title: "SDK Examples" +description: "Working code examples for the VortexDB Python SDK" +--- + +# SDK Examples + +Ready-to-run code examples demonstrating VortexDB Python SDK usage. + +## Basic Usage + +```python +from vortexdb import VortexDB, DenseVector, Payload, Similarity + +with VortexDB(grpc_url="localhost:50051", api_key="secret") as db: + # Insert + point_id = db.insert( + vector=DenseVector([0.1, 0.2, 0.3]), + payload=Payload.text("hello world"), + ) + print(f"Inserted: {point_id}") + + # Batch insert + ids = db.batch_insert(items=[ + (DenseVector([0.1, 0.2, 0.3]), Payload.text("doc one")), + (DenseVector([0.4, 0.5, 0.6]), Payload.text("doc two")), + ]) + + # Search + results = db.search( + vector=DenseVector([0.1, 0.2, 0.3]), + similarity=Similarity.COSINE, + limit=3, + ) + print(f"Found {len(results)} results") +``` + +--- + +## Semantic Search + +Using sentence-transformers for text embedding: + +```python +from vortexdb import VortexDB, DenseVector, Payload, Similarity +from sentence_transformers import SentenceTransformer + +model = SentenceTransformer('all-MiniLM-L6-v2') + +documents = [ + "The quick brown fox jumps over the lazy dog", + "Machine learning is a subset of artificial intelligence", + "Python is a popular programming language", +] + +def embed(text: str) -> DenseVector: + return DenseVector(model.encode(text).tolist()) + +with VortexDB(grpc_url="localhost:50051", api_key="secret") as db: + for doc in documents: + db.insert(vector=embed(doc), payload=Payload.text(doc)) + + results = db.search( + vector=embed("AI and programming"), + similarity=Similarity.COSINE, + limit=2, + ) + for pid in results: + point = db.get(point_id=pid) + print(f" {point.payload.content}") +``` + +--- + +## Batch Processing + +```python +from vortexdb import VortexDB, DenseVector, Payload, Similarity + +with VortexDB(grpc_url="localhost:50051", api_key="secret") as db: + items = [(DenseVector([i * 0.1 for _ in range(3)]), Payload.text(f"doc {i}")) for i in range(100)] + ids = db.batch_insert(items=items) + + # Batch search + queries = [ + (DenseVector([0.1, 0.2, 0.3]), Similarity.COSINE, 3), + (DenseVector([0.4, 0.5, 0.6]), Similarity.EUCLIDEAN, 3), + ] + batch_results = db.batch_search(queries=queries) + for i, res in enumerate(batch_results): + print(f"Query {i}: {len(res)} results") +``` + +--- + +## Testing with pytest + +```python +import pytest +from vortexdb import VortexDB, DenseVector, Payload, Similarity + +@pytest.fixture +def db(): + client = VortexDB(grpc_url="localhost:50051", api_key="secret") + yield client + client.close() + +class TestVortexDB: + def test_insert_and_get(self, db): + point_id = db.insert( + vector=DenseVector([0.1, 0.2, 0.3, 0.4]), + payload=Payload.text("Test document"), + ) + point = db.get(point_id=point_id) + assert point is not None + assert point.payload.content == "Test document" + db.delete(point_id=point_id) + + def test_search(self, db): + point_id = db.insert( + vector=DenseVector([1.0, 2.0, 3.0]), + payload=Payload.text("target"), + ) + results = db.search( + vector=DenseVector([1.0, 2.0, 3.0]), + similarity=Similarity.COSINE, + limit=10, + ) + assert point_id in results + db.delete(point_id=point_id) +``` + +--- + +## Next Steps + + + + Complete API documentation + + + gRPC and HTTP API docs + + diff --git a/docs/sdk/reference.mdx b/docs/sdk/reference.mdx new file mode 100644 index 0000000..ad1e6e2 --- /dev/null +++ b/docs/sdk/reference.mdx @@ -0,0 +1,626 @@ +--- +title: "SDK Reference" +description: "Complete Python SDK API documentation" +--- + +# Python SDK Reference + +Complete API documentation for the VortexDB Python client. + +## Installation + +```bash +pip install vortexdb +``` + +--- + +## VortexDB + +The main client class for interacting with VortexDB. + +```python +from vortexdb import VortexDB +``` + +### Constructor + +```python +VortexDB( + *, + grpc_url: str | None = None, + api_key: str | None = None, + timeout: float | None = None, +) +``` + + + The gRPC server address. Can also be set via `VORTEXDB_GRPC_URL` environment variable. + + + + Authentication key for the gRPC API. Can also be set via `VORTEXDB_API_KEY` environment variable. + + + + Request timeout in seconds. Can also be set via `VORTEXDB_TIMEOUT` environment variable. + + +**Example:** + +```python +# Explicit configuration +db = VortexDB( + grpc_url="localhost:50051", + api_key="secret", + timeout=60.0, +) + +# Using environment variables +import os +os.environ["VORTEXDB_GRPC_URL"] = "localhost:50051" +os.environ["VORTEXDB_API_KEY"] = "secret" +db = VortexDB() +``` + +--- + +### Methods + +#### insert + +Insert a vector with its payload into the database. + +```python +def insert( + self, + *, + vector: DenseVector, + payload: Payload, +) -> str +``` + + + UUID of the created point. + + +**Example:** + +```python +point_id = db.insert( + vector=DenseVector([0.1, 0.2, 0.3, 0.4]), + payload=Payload.text("My document"), +) +``` + +--- + +#### batch_insert + +Insert multiple vectors in a single request. + +```python +def batch_insert( + self, + *, + items: list[tuple[DenseVector, Payload]], +) -> list[str] +``` + + + List of UUIDs for the created points, in input order. + + +**Example:** + +```python +ids = db.batch_insert(items=[ + (DenseVector([0.1, 0.2, 0.3]), Payload.text("doc one")), + (DenseVector([0.4, 0.5, 0.6]), Payload.text("doc two")), +]) +``` + +--- + +#### get + +Retrieve a point by its ID. + +```python +def get( + self, + *, + point_id: str, +) -> Point | None +``` + + + The point if found, `None` otherwise. + + +**Example:** + +```python +point = db.get(point_id="550e8400-e29b-41d4-a716-446655440000") +if point: + print(f"Vector: {point.vector.to_list()}") + print(f"Payload: {point.payload.content}") +``` + +--- + +#### search + +Search for the k nearest neighbors to a query vector. + +```python +def search( + self, + *, + vector: DenseVector | None = None, + similarity: Similarity | None = None, + limit: int | None = None, + query: SearchQuery | None = None, + ef: int | None = None, +) -> List[str] +``` + + + A `SearchQuery` object bundling vector, similarity, and limit. Use this or pass individual args. + + + + Search breadth for HNSW. Uses server default if not set. + + + + List of point IDs ordered by similarity (closest first). + + +**Example:** + +```python +# Using a SearchQuery +query = SearchQuery(DenseVector([0.1, 0.2, 0.3, 0.4]), Similarity.COSINE, 10) +results = db.search(query=query) + +# Using individual args with ef +results = db.search( + vector=DenseVector([0.1, 0.2, 0.3, 0.4]), + similarity=Similarity.COSINE, + limit=10, + ef=200, +) +``` + +--- + +#### batch_search + +Search against multiple query vectors in a single request. + +```python +def batch_search( + self, + *, + queries, + similarity: Similarity | None = None, + limit: int | None = None, + ef: int | None = None, +) -> List[List[str]] +``` + +Accepts `List[SearchQuery]`, `List[(DenseVector, Similarity, int)]`, or bare `List[DenseVector]` with global `similarity` and `limit`. + + + One result list per input query. + + +**Example:** + +```python +results = db.batch_search(queries=[ + SearchQuery(DenseVector([0.1, 0.2, 0.3]), Similarity.COSINE, 5), + (DenseVector([0.4, 0.5, 0.6]), Similarity.EUCLIDEAN, 3), +]) +``` + +--- + +#### delete + +Delete a point by its ID. + +```python +def delete( + self, + *, + point_id: str, +) -> None +``` + +**Example:** + +```python +db.delete(point_id="550e8400-e29b-41d4-a716-446655440000") +``` + +--- + +#### close + +Close the gRPC connection. + +```python +def close(self) -> None +``` + +**Example:** + +```python +db = VortexDB(grpc_url="localhost:50051", api_key="secret") +# ... use the client ... +db.close() +``` + +--- + +### Context Manager + +The client supports the context manager protocol for automatic cleanup: + +```python +with VortexDB(grpc_url="localhost:50051", api_key="secret") as db: + point_id = db.insert( + vector=DenseVector([0.1, 0.2, 0.3]), + payload=Payload.text("Hello"), + ) +# Connection automatically closed +``` + +--- + +## DenseVector + +An immutable dense vector of floating-point values. + +```python +from vortexdb import DenseVector +``` + +### Constructor + +```python +DenseVector(values: List[float] | Tuple[float, ...]) +``` + + + The vector components. Must be non-empty and contain numeric values. + + +**Raises:** +- `TypeError`: If values is not a list or tuple +- `ValueError`: If values is empty +- `TypeError`: If any value is not numeric + +**Example:** + +```python +# From list +vec = DenseVector([0.1, 0.2, 0.3, 0.4]) + +# From tuple +vec = DenseVector((0.1, 0.2, 0.3, 0.4)) + +# Integers are converted to floats +vec = DenseVector([1, 2, 3, 4]) # -> [1.0, 2.0, 3.0, 4.0] +``` + +### Methods + +#### to_list + +Convert the vector to a Python list. + +```python +def to_list(self) -> list[float] +``` + +**Example:** + +```python +vec = DenseVector([0.1, 0.2, 0.3]) +values = vec.to_list() # [0.1, 0.2, 0.3] +``` + +### Properties + +#### values + +Access the vector values (read-only). + +```python +vec = DenseVector([0.1, 0.2, 0.3]) +print(vec.values) # (0.1, 0.2, 0.3) +``` + +--- + +## Payload + +Metadata associated with a vector. + +```python +from vortexdb import Payload +``` + +### Factory Methods + +#### text + +Create a text payload. + +```python +@staticmethod +def text(content: str) -> Payload +``` + +**Example:** + +```python +payload = Payload.text("This is my document content") +``` + +#### image + +Create an image payload. + +```python +@staticmethod +def image(content: str) -> Payload +``` + +**Example:** + +```python +payload = Payload.image("path/to/image.jpg") +``` + +### Constructor + +```python +Payload(content_type: ContentType, content: str) +``` + + + The type of content (`ContentType.TEXT` or `ContentType.IMAGE`). + + + + The content string. + + +### Properties + +| Property | Type | Description | +|----------|------|-------------| +| `content_type` | `ContentType` | Type of payload | +| `content` | `str` | Content string | + +--- + +## Point + +A point returned from the database (vector + payload + ID). + +```python +from vortexdb.models import Point +``` + +### Properties + +| Property | Type | Description | +|----------|------|-------------| +| `id` | `str` | Point UUID | +| `vector` | `DenseVector` | The vector values | +| `payload` | `Payload` | Associated metadata | + +### Methods + +#### pretty + +Return a formatted string representation. + +```python +def pretty(self) -> str +``` + +**Example:** + +```python +point = db.get(point_id="...") +print(point.pretty()) +# Output: +# Point ID: 550e8400-e29b-41d4-a716-446655440000 +# Vector: [0.1, 0.2, 0.3, 0.4] +# Payload Type: Text +# Payload Content: My document +``` + +--- + +## Similarity + +Enum for distance/similarity metrics. + +```python +from vortexdb import Similarity +``` + +### Values + +| Value | Description | +|-------|-------------| +| `Similarity.EUCLIDEAN` | L2 distance (straight line) | +| `Similarity.MANHATTAN` | L1 distance (city block) | +| `Similarity.HAMMING` | Count of differing elements | +| `Similarity.COSINE` | Angular distance | + +**Example:** + +```python +from vortexdb import Similarity + +# Use in search +results = db.search( + vector=DenseVector([0.1, 0.2, 0.3]), + similarity=Similarity.COSINE, + limit=5, +) +``` + +--- + +## ContentType + +Enum for payload content types. + +```python +from vortexdb.models import ContentType +``` + +### Values + +| Value | Description | +|-------|-------------| +| `ContentType.TEXT` | Text content | +| `ContentType.IMAGE` | Image reference | + +--- + +## SearchQuery + +Bundles a search's parameters into a single object. + +```python +from vortexdb import SearchQuery + +query = SearchQuery(DenseVector([0.1, 0.2, 0.3]), Similarity.COSINE, 10) +results = db.search(query=query) +``` + +| Param | Type | Description | +|-------|------|-------------| +| `vector` | `DenseVector` | Query vector | +| `similarity` | `Similarity` | Distance metric | +| `limit` | `int` | Max results | + +--- + +## Exceptions + +All exceptions inherit from `VortexDBError`. + +```python +from vortexdb import ( + VortexDBError, + AuthenticationError, + NotFoundError, + InvalidArgumentError, + TimeoutError, + ServiceUnavailableError, + InternalServerError, +) +``` + +### Exception Hierarchy + +| Exception | Description | +|-----------|-------------| +| `VortexDBError` | Base exception for all errors | +| `AuthenticationError` | Invalid or missing API key | +| `NotFoundError` | Requested resource not found | +| `InvalidArgumentError` | Invalid input parameters | +| `TimeoutError` | Request timed out | +| `ServiceUnavailableError` | Server is unavailable | +| `InternalServerError` | Server-side error | + +**Example:** + +```python +from vortexdb.exceptions import ( + AuthenticationError, + NotFoundError, + VortexDBError, +) + +try: + point = db.get(point_id="nonexistent") +except NotFoundError: + print("Point does not exist") +except AuthenticationError: + print("Check your API key") +except VortexDBError as e: + print(f"Unexpected error: {e}") +``` + +--- + +## Configuration + +### VortexDBConfig + +Internal configuration class (usually not used directly). + +```python +from vortexdb.config import VortexDBConfig + +config = VortexDBConfig.from_env( + grpc_url="localhost:50051", + api_key="secret", + timeout=30.0, +) +``` + +### Environment Variables + +| Variable | Description | Default | +|----------|-------------|---------| +| `VORTEXDB_GRPC_URL` | Server address | `localhost:50051` | +| `VORTEXDB_API_KEY` | Authentication key | None | +| `VORTEXDB_TIMEOUT` | Request timeout (seconds) | `30.0` | + +--- + +## Type Hints + +The SDK is fully typed. Import types for type hints: + +```python +from typing import List, Optional +from vortexdb import VortexDB, DenseVector, Payload, Similarity +from vortexdb.models import Point, ContentType + +def search_documents( + db: VortexDB, + query_vector: List[float], + limit: int = 10, +) -> List[Optional[Point]]: + results = db.search( + vector=DenseVector(query_vector), + similarity=Similarity.COSINE, + limit=limit, + ) + return [db.get(point_id=pid) for pid in results] +``` + +## Next Steps + + + + Working code examples + + + gRPC and HTTP API docs + +