Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,29 @@
- `generate-compose.sh` merges templates into final compose files
- `setup.sh` generates secrets using `generate-secrets.sh`
- `.env.example` documents all configurable variables
- Documentation lives in `on-prem/docs/`

## Documentation Requirements

**Always update documentation when making changes.** The docs folder (`on-prem/docs/`) contains user-facing documentation that must stay in sync with the codebase.

### When to Update Each File

| File | Update When... |
|------|----------------|
| `docs/configuration.md` | Adding/removing/changing environment variables, changing defaults, adding new `DC_*` variables |
| `docs/quickstart.md` | Changing setup flow, adding new features users need to configure, changing ports/volumes/services |
| `docs/support.md` | Changing support boundaries, adding new component categories |
| `docs/README.md` | Adding new documentation pages, changing known limitations |
| `.env.example` | Adding any new environment variable (always document generation commands for secrets) |
| `on-prem/README.md` | Changing scripts, profiles, or file structure |

### Documentation Style

- Use tables for configuration references (Variable | Type | Default | Description)
- Group settings into **Required**, **Frequently Used**, and **Other** sections
- Include example values and generation commands for secrets
- Document both localhost development and production deployment patterns

## Environment Variables Reference

Expand Down
5 changes: 4 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,14 @@ Docker configurations for running Currents services.

## On-Premises

For self-hosted Currents deployment, see the [On-Prem documentation](on-prem/README.md).
For self-hosted Currents deployment, see the [On-Prem documentation](on-prem/docs/).

```bash
cd on-prem
./scripts/setup.sh
docker compose up -d
```

## Development

- [On-Prem Development Guide](on-prem/README.md) — Architecture, scripts, and release steps
61 changes: 39 additions & 22 deletions on-prem/.env.example
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
# =============================================================================
# Required Configuration
# =============================================================================
# These values MUST be set before starting. Run `./scripts/setup.sh` to generate secrets.
# These values MUST be set before starting.
# Run `./scripts/setup.sh` to generate all secrets automatically,
# or use the generate-secrets.sh commands below to generate them manually.

# Initial Setup
ON_PREM_EMAIL=admin@localhost
Expand All @@ -11,20 +13,35 @@ SMTP_HOST=localhost
SMTP_USER=test
SMTP_PASS=test

# Authentication (generated by setup.sh)
# Authentication
# Generate with: ./scripts/generate-secrets.sh token 64
JWT_SECRET=
# Generate with: ./scripts/generate-secrets.sh token 64
API_SECRET=

# MongoDB (password generated by setup.sh)
# Object Storage (additional settings)
# RUSTFS (optional, set only if using the included rustfs)
# Generate with: ./scripts/generate-secrets.sh token 32
RUSTFS_SECRET_KEY=

# MongoDB
# Generate with: ./scripts/generate-secrets.sh token 32
# Or set to your existing password
MONGODB_PASSWORD=

# ClickHouse (passwords generated by setup.sh)
# ClickHouse
# Default password only needed if using the included clickhouse
# Generate with: ./scripts/generate-secrets.sh token 32
CLICKHOUSE_DEFAULT_PASSWORD=
# Password for the currents-user for clickhouse
# Generate with: ./scripts/generate-secrets.sh token 32
# Or provide your existing password
CLICKHOUSE_CURRENTS_PASSWORD=

# GitLab Integration
# Run `./scripts/generate-secrets key gitlab-key.pem`
# Then `base64 < "gitlab-key.pem" | tr -d '\n'` to get the base64 secret
# GitLab Integration (base64-encoded RSA private key)
# Generate with:
# ./scripts/generate-secrets.sh key .gitlab-key.pem
# base64 < .gitlab-key.pem | tr -d '\n'
GITLAB_STATE_SECRET=

# =============================================================================
Expand All @@ -47,6 +64,16 @@ FILE_STORAGE_ACCESS_KEY_ID=${RUSTFS_ACCESS_KEY}
FILE_STORAGE_SECRET_ACCESS_KEY=${RUSTFS_SECRET_KEY}
# FILE_STORAGE_REGION=us-east-1

# ClickHouse
CLICKHOUSE_URL=http://clickhouse:8123
CLICKHOUSE_USERNAME=currents
# CLICKHOUSE_ACCESS_TOKEN=

# MongoDB
MONGODB_USERNAME=currents-user
MONGODB_DATABASE=currents
MONGODB_URI=mongodb://${MONGODB_USERNAME}:${MONGODB_PASSWORD}@mongodb:27017/${MONGODB_DATABASE}?authSource=admin&replicaSet=rs0

# Domain configuration for Traefik routing
# TRAEFIK_DOMAIN=example.com
# TRAEFIK_API_SUBDOMAIN=currents-app
Expand All @@ -68,25 +95,16 @@ GITLAB_REDIRECT_URL=${APP_BASE_URL}/integrations/gitlab/callback
# Authentication settings
JWT_SECRET_EXPIRY=10m

# MongoDB
MONGODB_USERNAME=currents-user
MONGODB_DATABASE=currents
MONGODB_URI=mongodb://${MONGODB_USERNAME}:${MONGODB_PASSWORD}@mongodb:27017/${MONGODB_DATABASE}?authSource=admin&replicaSet=rs0

# Redis (default points to redis service in compose network)
REDIS_URI=redis://redis:6379
REDIS_URI_SLAVE=redis://redis:6379

# ClickHouse
CLICKHOUSE_URL=http://clickhouse:8123
CLICKHOUSE_USERNAME=currents
# CLICKHOUSE_ACCESS_TOKEN=

# Object Storage (additional settings)
# Object Storage (addition configuration)
## RustFS (optional)
RUSTFS_ACCESS_KEY=rustfs-access-key
RUSTFS_SECRET_KEY=rustfs-secret-key

FILE_STORAGE_INTERNAL_ENDPOINT="http://host.docker.internal:9000"
FILE_STORAGE_FORCE_PATH_STYLE=true
# FILE_STORAGE_FORCE_PATH_STYLE=true # Set automatically when using RustFS profile

# Email (additional settings)
AUTOMATED_REPORTS_EMAIL_BCC=
Expand All @@ -96,7 +114,6 @@ AUTOMATED_REPORTS_EMAIL_BCC=
# CORALOGIX_API_KEY=
# CORALOGIX_APP_NAME="customer-name"


# =============================================================================
# Docker Compose Configuration
# =============================================================================
Expand Down Expand Up @@ -146,4 +163,4 @@ AUTOMATED_REPORTS_EMAIL_BCC=
# DC_TRAEFIK_CERTS_DIR=./data/traefik/certs
# DC_TRAEFIK_CONFIG_DIR=./data/traefik/config
# DC_TRAEFIK_DYNAMIC_CONFIG_PATH=/etc/traefik/dynamic.yml
# TRAEFIK_GENERATE_TEMP_CERTS=false
# TRAEFIK_GENERATE_TEMP_CERTS=false
94 changes: 44 additions & 50 deletions on-prem/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,46 +2,52 @@

Docker Compose configuration for running Currents on-premises.

## Quick Start
## Documentation

```bash
# Run the interactive setup
./scripts/setup.sh
```
📚 **For setup and configuration instructions, see the [docs](./docs/) folder:**

This will:
1. Guide you through selecting a configuration profile
2. Generate the appropriate `docker-compose.yml`
3. Create a `.env` file with auto-generated secrets
- [Quickstart Guide](./docs/quickstart.md) — Get up and running
- [Configuration Reference](./docs/configuration.md) — All environment variables
- [Support Policy](./docs/support.md) — What's supported

Then start the services:
---

```bash
docker compose up -d
## Development Guide

This section is for developers working on the docker-compose configuration itself.

### Architecture

The configuration is built from modular templates that are merged together based on the selected profile. This allows users to include only the services they need.

```
templates/
├── compose.currents.yml # Core Currents application services (always included)
├── compose.redis.yml # Redis cache
├── compose.mongodb.yml # MongoDB database
├── compose.clickhouse.yml # ClickHouse analytics
├── compose.rustfs.yml # S3-compatible object storage
└── compose.traefik.yml # TLS termination proxy (optional profile)
```

## Configuration Profiles
### Configuration Profiles

| Profile | Services Included | Use Case |
|---------|-------------------|----------|
| `full` | Redis, MongoDB, ClickHouse, RustFS | Running everything locally |
| `database` | Redis, MongoDB, ClickHouse | External S3-compatible storage |
| `cache` | Redis | External MongoDB, ClickHouse, and S3 |

## Scripts
### Scripts

### `scripts/setup.sh`
#### `scripts/setup.sh`

Interactive setup wizard that:
- Prompts for profile selection (or custom service selection)
- Generates the docker-compose configuration
- Creates `.env` from `.env.example` with auto-generated secrets

```bash
./scripts/setup.sh
```

### `scripts/generate-compose.sh`
#### `scripts/generate-compose.sh`

Generates a docker-compose file for a specific profile. Used by `setup.sh` but can also be run directly.

Expand Down Expand Up @@ -75,7 +81,7 @@ Generates a docker-compose file for a specific profile. Used by `setup.sh` but c
| `clickhouse` | ClickHouse only |
| `rustfs` | RustFS only |

### `scripts/generate-secrets.sh`
#### `scripts/generate-secrets.sh`

Utility for generating secrets and keys.

Expand All @@ -90,7 +96,7 @@ Utility for generating secrets and keys.
./scripts/generate-secrets.sh key mykey.pem
```

## Files
### Files

| File | Description |
|------|-------------|
Expand All @@ -101,48 +107,30 @@ Utility for generating secrets and keys.
| `.env` | Your environment configuration (git-ignored) |
| `.env.example` | Template for environment configuration |
| `templates/` | Source templates for compose generation |
| `docs/` | User-facing documentation |

## Environment Configuration

Copy `.env.example` to `.env` and configure as needed. The `setup.sh` script does this automatically and generates required secrets.

### Application Variables

- `JWT_SECRET` - Authentication token secret
- `RUSTFS_ACCESS_KEY` / `RUSTFS_SECRET_KEY` - RustFS credentials
- `GITLAB_STATE_SECRET` - GitLab integration key (base64-encoded PEM)

### Compose-Only Variables
### Services

Variables prefixed with `DC_` are used only by docker-compose (not passed to containers):

- `DC_IMAGE_REPOSITORY` - Docker image repository prefix
- `DC_IMAGE_TAG` - Docker image tag
- `DC_REDIS_VOLUME` - Redis data volume path
- `DC_MONGODB_VOLUME` - MongoDB data volume path
- `DC_CLICKHOUSE_VOLUME` - ClickHouse data volume path
- `DC_RUSTFS_VOLUME` - RustFS data volume path
- `DC_SCHEDULER_STARTUP_VOLUME` - Scheduler startup data volume path

## Services

### Currents Application
#### Currents Application
- **director** (port 1234) - Test orchestration service
- **api** (port 4000) - API and dashboard
- **changestreams-worker** - MongoDB change stream processor
- **write-worker** - Async write operations
- **scheduler** - Scheduled tasks and startup migrations
- **webhooks** - Webhook delivery service

### Data Services (optional, based on profile)
- **redis** (port 6379, 8001) - Cache and pub/sub
#### Data Services (optional, based on profile)
- **redis** (port 6379) - Cache and pub/sub
- **mongodb** (port 27017) - Primary database
- **clickhouse** (port 8123, 9123) - Analytics database
- **rustfs** (port 9000, 9001) - S3-compatible object storage

## Development
#### Optional Services
- **traefik** (ports 80, 443) - TLS termination proxy (enabled with `--profile tls`)

To regenerate the pre-committed docker-compose files after modifying templates:
### Regenerating Compose Files

After modifying templates, regenerate the pre-committed docker-compose files:

```bash
./scripts/generate-compose.sh full
Expand All @@ -152,3 +140,9 @@ To regenerate the pre-committed docker-compose files after modifying templates:

A GitHub Action validates that committed compose files stay in sync with templates.

### Variable Naming Convention

- **`DC_` prefix**: Docker-compose-only variables (not passed to containers)
- `DC_MONGODB_PORT`, `DC_CURRENTS_IMAGE_TAG`, `DC_REDIS_VOLUME`
- **No prefix**: App config variables that containers need
- `MONGODB_PASSWORD`, `APP_BASE_URL`, `CLICKHOUSE_CURRENTS_PASSWORD`
8 changes: 7 additions & 1 deletion on-prem/docker-compose.full.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ name: currents_full
# RustFS service configuration
# S3-compatible object storage - usually excluded when using external S3/cloud storage
services:
# Currents services depend on redis
# Configure Currents services to use path-style S3 URLs (required for RustFS)
director:
hostname: director
image: ${DC_CURRENTS_IMAGE_REPOSITORY:-currents-}director:${DC_CURRENTS_IMAGE_TAG:-dev}
Expand All @@ -25,6 +25,7 @@ services:
environment:
CURRENTS_ENV: onprem
EMAIL_TRANSPORTER: smtp
FILE_STORAGE_FORCE_PATH_STYLE: "true"
depends_on:
clickhouse:
condition: service_healthy
Expand All @@ -50,6 +51,7 @@ services:
INCLUDE_DASHBOARD: onprem
CLICKHOUSE_PASSWORD: ${CLICKHOUSE_CURRENTS_PASSWORD}
EMAIL_TRANSPORTER: smtp
FILE_STORAGE_FORCE_PATH_STYLE: "true"
depends_on:
clickhouse:
condition: service_healthy
Expand All @@ -71,6 +73,7 @@ services:
CURRENTS_ENV: onprem
CLICKHOUSE_PASSWORD: ${CLICKHOUSE_CURRENTS_PASSWORD}
EMAIL_TRANSPORTER: smtp
FILE_STORAGE_FORCE_PATH_STYLE: "true"
depends_on:
clickhouse:
condition: service_healthy
Expand All @@ -92,6 +95,7 @@ services:
CURRENTS_ENV: onprem
CLICKHOUSE_PASSWORD: ${CLICKHOUSE_CURRENTS_PASSWORD}
EMAIL_TRANSPORTER: smtp
FILE_STORAGE_FORCE_PATH_STYLE: "true"
depends_on:
clickhouse:
condition: service_healthy
Expand All @@ -114,6 +118,7 @@ services:
CLICKHOUSE_PASSWORD: ${CLICKHOUSE_CURRENTS_PASSWORD}
RUN_STARTUP_TASKS: "true"
EMAIL_TRANSPORTER: smtp
FILE_STORAGE_FORCE_PATH_STYLE: "true"
volumes:
- ${DC_SCHEDULER_STARTUP_VOLUME:-./data/startup}:/app/packages/scheduler/dist/.startup
depends_on:
Expand All @@ -136,6 +141,7 @@ services:
environment:
CURRENTS_ENV: onprem
EMAIL_TRANSPORTER: smtp
FILE_STORAGE_FORCE_PATH_STYLE: "true"
depends_on:
clickhouse:
condition: service_healthy
Expand Down
Loading