Skip to content
This repository was archived by the owner on Nov 8, 2026. It is now read-only.
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -147,8 +147,8 @@ jobs:

qualification:
name: External test suite
uses: batchstream/sink-production-suite/.github/workflows/server-qualification.yml@e084c611c2ce6efb2e2175cbcfc931dd497c9811
uses: batchstream/sink-production-suite/.github/workflows/server-qualification.yml@924aa80684708a06685625b8161fc3529c218529
with:
suite_ref: e084c611c2ce6efb2e2175cbcfc931dd497c9811
suite_ref: 924aa80684708a06685625b8161fc3529c218529
sink_ref: ${{ github.event.pull_request.head.sha || github.sha }}
sdk_ref: ${{ github.head_ref || 'main' }}
11 changes: 7 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -154,21 +154,24 @@ Choose the annotated configuration for the component you are deploying:
| Engine | [configs/engine.yaml](configs/engine.yaml) | One Store's synchronous execution and optional asynchronous publication. |
| Worker | [configs/worker.yaml](configs/worker.yaml) | One Store's Kafka consumption and mutation execution. |

For an Engine, copy its example, edit the backend connection, and mount the file:
For an Engine, copy the role and shared Store examples, edit the Store connection,
and mount both files. Clients connect through Gateway.

```shell
cp configs/engine.yaml config.yaml
# Edit config.yaml for the target backend.
cp configs/stores/primary.yaml store.yaml
# Edit store.yaml for the target backend.
docker run --rm -p 8080:8080 \
--mount type=bind,source="$(pwd)/config.yaml",target=/etc/sink/config.yaml,readonly \
ghcr.io/batchstream/sink:latest --config /etc/sink/config.yaml
--mount type=bind,source="$(pwd)/store.yaml",target=/etc/sink/store.yaml,readonly \
ghcr.io/batchstream/sink:latest --config /etc/sink/config.yaml --store-config /etc/sink/store.yaml
```

HTTP health endpoints are always available at `health.address` (default `:8081`).
Prometheus uses a separate `prometheus.address` (default `:9090`) and requires
`prometheus.enabled: true`. Publish the corresponding ports when needed.

Gateway includes its Store routes under `gateway.routes`. Restart after changing
Gateway includes its Store routes under `forwarding.routes`. Restart after changing
configuration; routes are loaded only at startup. Worker needs its own configuration and no
public gRPC port. The [quickstart](examples/quickstart/README.md) runs all three roles.

Expand Down
18 changes: 3 additions & 15 deletions cmd/sink/config_command_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,21 +7,9 @@ import (
)

func TestConfigCheckDoesNotOpenDependenciesOrPrintValues(t *testing.T) {
path := writeConfig(t, `mode: engine
storage:
name: primary
driver: mongodb
mongodb:
uri: mongodb://127.0.0.1:1
kafka:
enabled: true
brokers: [127.0.0.1:1]
topic:
name: mutations
consumer:
group_id: workers
`)
args := []string{"config", "check", "--config", path}
path := writeConfig(t, "mode: engine\n")
shared := writeConfig(t, "name: primary\nstorage:\n driver: mongodb\n mongodb: {uri: 'mongodb://127.0.0.1:1'}\nkafka:\n enabled: true\n brokers: ['127.0.0.1:1']\n topic: {name: mutations}\n")
args := []string{"config", "check", "--config", path, "--store-config", shared}
var stdout, stderr bytes.Buffer
if err := executeCommand(args, &stdout, &stderr); err != nil {
t.Fatal(err)
Expand Down
12 changes: 6 additions & 6 deletions cmd/sink/config_path_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,19 +8,19 @@ import (

func TestParseConfigPath(t *testing.T) {
args := []string{"--config", "/etc/sink/config.yaml"}
path, err := parseConfigPath(args)
path, err := parseConfigPaths(args)
if err != nil {
t.Fatalf("parseConfigPath() error = %v", err)
t.Fatalf("parseConfigPaths() error = %v", err)
}
if path != "/etc/sink/config.yaml" {
t.Fatalf("parseConfigPath() = %q", path)
if path.Component != "/etc/sink/config.yaml" {
t.Fatalf("parseConfigPaths() = %q", path)
}
}

func TestParseConfigPathRequiresFlag(t *testing.T) {
_, err := parseConfigPath(nil)
_, err := parseConfigPaths(nil)
if err == nil || err.Error() != "--config is required" {
t.Fatalf("parseConfigPath() error = %v", err)
t.Fatalf("parseConfigPaths() error = %v", err)
}
}

Expand Down
9 changes: 5 additions & 4 deletions cmd/sink/dead_letter_command.go
Original file line number Diff line number Diff line change
Expand Up @@ -28,11 +28,12 @@ type deadLetterReport struct {

func runDeadLetterCommand(args []string, stdout io.Writer, stderr io.Writer) error {
if len(args) == 0 || (args[0] != "inspect" && args[0] != "replay") {
return errors.New("usage: sink dlq inspect|replay --config FILE --store NAME --partition N --offset N --count N")
return errors.New("usage: sink dlq inspect|replay --config FILE --store-config FILE --store NAME --partition N --offset N --count N")
}
flags := flag.NewFlagSet("dlq "+args[0], flag.ContinueOnError)
flags.SetOutput(stderr)
configPath := flags.String("config", "", "server configuration")
storePath := flags.String("store-config", "", "shared Store configuration")
store := flags.String("store", "", "configured store")
partition := flags.Int("partition", -1, "exact dead-letter partition")
offset := flags.Int64("offset", -1, "first dead-letter offset")
Expand All @@ -43,7 +44,7 @@ func runDeadLetterCommand(args []string, stdout io.Writer, stderr io.Writer) err
if flags.NArg() != 0 || *configPath == "" || *store == "" || *partition < 0 || *partition > 1<<31-1 || *offset < 0 || *count < 1 || *count > 100 {
return errors.New("config, store, partition, offset, and a count between 1 and 100 are required")
}
loaded, err := config.Load(*configPath)
loaded, err := config.Load(*configPath, *storePath)
if err != nil {
return err
}
Expand All @@ -53,7 +54,7 @@ func runDeadLetterCommand(args []string, stdout io.Writer, stderr io.Writer) err
}
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
selection := queuekafka.DeadLetterRange{Brokers: selected.Kafka.Brokers, Topic: selected.Kafka.DeadLetter.Topic,
selection := queuekafka.DeadLetterRange{Brokers: selected.Kafka.Brokers, Topic: selected.Kafka.DeadLetter.Name,
Partition: int32(*partition), Offset: *offset, Count: *count}
records, err := queuekafka.ReadDeadLetters(ctx, selection)
if err != nil {
Expand All @@ -66,7 +67,7 @@ func runDeadLetterCommand(args []string, stdout io.Writer, stderr io.Writer) err
return err
}
opts := queuekafka.PublisherOptions{Brokers: selected.Kafka.Brokers, Topic: selected.Kafka.Topic.Name,
MaxRecordBytes: selected.Kafka.Topic.MaxRecordBytes, MaxBufferedBytes: selected.Kafka.Producer.MaxBufferedBytes}
MaxRecordBytes: selected.Kafka.MaxRecordBytes, MaxBufferedBytes: selected.Kafka.Producer.MaxBufferedBytes}

publisher, err := queuekafka.NewPublisher(opts)
if err != nil {
Expand Down
26 changes: 13 additions & 13 deletions cmd/sink/dead_letter_command_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -70,22 +70,22 @@ func TestDeadLetterReplayPreservesPublisherRouting(t *testing.T) {
if driver == config.DriverMongoDB {
backend = " mongodb:\n uri: mongodb://127.0.0.1:1\n"
}
contents := fmt.Sprintf(`mode: engine
contents := fmt.Sprintf(`name: primary
storage:
name: primary
driver: %s
%s kafka:
enabled: true
brokers: [%q]
topic:
name: source
replication_factor: 1
min_insync_replicas: 1
dead_letter:
topic: source.dlq
%skafka:
enabled: true
brokers: [%q]
replication_factor: 1
min_insync_replicas: 1
topic:
name: source
dead_letter:
name: source.dlq
`, driver, backend, cluster.ListenAddrs()[0])
configPath := writeConfig(t, contents)
args := []string{"replay", "--config", configPath, "--store", "primary", "--partition", strconv.Itoa(int(letter.Partition)), "--offset", strconv.FormatInt(letter.Offset, 10)}
sharedPath := writeConfig(t, contents)
configPath := writeConfig(t, "mode: engine\n")
args := []string{"replay", "--config", configPath, "--store-config", sharedPath, "--store", "primary", "--partition", strconv.Itoa(int(letter.Partition)), "--offset", strconv.FormatInt(letter.Offset, 10)}
var stdout, stderr bytes.Buffer
if err := runDeadLetterCommand(args, &stdout, &stderr); err != nil {
t.Fatalf("replay: %v stdout=%s stderr=%s", err, &stdout, &stderr)
Expand Down
26 changes: 16 additions & 10 deletions cmd/sink/lua_command.go
Original file line number Diff line number Diff line change
Expand Up @@ -23,14 +23,15 @@ import (
)

type luaTestFlags struct {
script string
config string
cases string
encoding string
current string
incoming string
expected string
observedAt string
script string
config string
storeConfig string
cases string
encoding string
current string
incoming string
expected string
observedAt string
}

type luaTestCaseFile struct {
Expand Down Expand Up @@ -84,7 +85,7 @@ func runLuaTestCommand(args []string, stdout io.Writer, stderr io.Writer) error
}
luaOptions := merge.LuaOptions{}
if parsed.config != "" {
loaded, loadErr := config.Load(parsed.config)
loaded, loadErr := config.Load(parsed.config, parsed.storeConfig)
if loadErr != nil {
return fmt.Errorf("load Lua test limits from config: %w", loadErr)
}
Expand Down Expand Up @@ -117,7 +118,8 @@ func parseLuaTestFlags(args []string, stderr io.Writer) (luaTestFlags, error) {
flags := flag.NewFlagSet("sink lua test", flag.ContinueOnError)
flags.SetOutput(stderr)
flags.StringVar(&parsed.script, "script", "", "path to the Lua merge script")
flags.StringVar(&parsed.config, "config", "", "optional Sink config whose service.merge.lua limits are applied")
flags.StringVar(&parsed.storeConfig, "store-config", "", "Store configuration used with --config")
flags.StringVar(&parsed.config, "config", "", "optional Sink config whose execution.merge.lua limits are applied")
flags.StringVar(&parsed.cases, "cases", "", "path to one YAML case or a directory of YAML cases")
flags.StringVar(&parsed.encoding, "encoding", "", "single-case document encoding: json or bson")
flags.StringVar(&parsed.current, "current", "", "optional single-case current JSON or Extended JSON document")
Expand All @@ -132,6 +134,10 @@ func parseLuaTestFlags(args []string, stderr io.Writer) (luaTestFlags, error) {
}
parsed.script = strings.TrimSpace(parsed.script)
parsed.config = strings.TrimSpace(parsed.config)
parsed.storeConfig = strings.TrimSpace(parsed.storeConfig)
if parsed.storeConfig != "" && parsed.config == "" {
return parsed, errors.New("--store-config requires --config")
}
parsed.cases = strings.TrimSpace(parsed.cases)
parsed.encoding = strings.TrimSpace(parsed.encoding)
parsed.current = strings.TrimSpace(parsed.current)
Expand Down
10 changes: 3 additions & 7 deletions cmd/sink/lua_command_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -234,21 +234,17 @@ end`)

func TestLuaTestCommandUsesConfiguredLimitsWithoutOpeningBackends(t *testing.T) {
fixture := newLuaCommandFixture(t, `return function(current, incoming) return incoming end`)
shared := fixture.write(t, "store.yaml", "name: primary\nstorage:\n driver: mongodb\n mongodb: {uri: mongodb://unreachable.invalid:27017}\n")
config := fixture.write(t, "sink.yaml", `mode: engine
storage:
name: primary
driver: mongodb
mongodb:
uri: mongodb://unreachable.invalid:27017
service:
execution:
merge:
lua:
max_source_bytes: 1
`)
incoming := fixture.write(t, "incoming.json", `{}`)
args := []string{
"--script", fixture.script,
"--config", config,
"--config", config, "--store-config", shared,
"--encoding", "json",
"--incoming", incoming,
"--observed-at", "2026-08-31T10:20:30Z",
Expand Down
38 changes: 23 additions & 15 deletions cmd/sink/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -48,22 +48,22 @@ func executeCommand(args []string, stdout io.Writer, stderr io.Writer) error {
return runConfigCommand(args[1:], stdout)
}
}
configPath, err := parseConfigPath(args)
paths, err := parseConfigPaths(args)
if err != nil {
return err
}
return run(configPath)
return run(paths)
}

func runConfigCommand(args []string, stdout io.Writer) error {
if len(args) == 0 || args[0] != "check" {
return errors.New("usage: sink config check --config FILE")
return errors.New("usage: sink config check --config FILE [--store-config FILE]")
}
configPath, err := parseConfigPath(args[1:])
paths, err := parseConfigPaths(args[1:])
if err != nil {
return err
}
loaded, err := config.Load(configPath)
loaded, err := config.Load(paths.Component, paths.Store)
if err != nil {
return err
}
Expand All @@ -76,27 +76,35 @@ func runConfigCommand(args []string, stdout io.Writer) error {
return err
}

func parseConfigPath(args []string) (string, error) {
type configPaths struct {
Component string
Store string
}

func parseConfigPaths(args []string) (configPaths, error) {
var paths configPaths
flags := flag.NewFlagSet("sink", flag.ContinueOnError)
flags.SetOutput(io.Discard)
configPath := flags.String("config", "", "path to the YAML configuration file")
flags.StringVar(&paths.Component, "config", "", "path to the component YAML configuration")
flags.StringVar(&paths.Store, "store-config", "", "path to the shared Store YAML (required for Engine/Worker)")
if err := flags.Parse(args); err != nil {
return "", fmt.Errorf("parse command arguments: %w", err)
return paths, fmt.Errorf("parse command arguments: %w", err)
}
if flags.NArg() != 0 {
return "", fmt.Errorf("unexpected command argument %q", flags.Arg(0))
return paths, fmt.Errorf("unexpected command argument %q", flags.Arg(0))
}
trimmed := strings.TrimSpace(*configPath)
if trimmed == "" {
return "", errors.New("--config is required")
paths.Component = strings.TrimSpace(paths.Component)
paths.Store = strings.TrimSpace(paths.Store)
if paths.Component == "" {
return paths, errors.New("--config is required")
}
return trimmed, nil
return paths, nil
}

type reportedError struct{ error }

func run(configPath string) (runErr error) {
loaded, err := config.Load(configPath)
func run(paths configPaths) (runErr error) {
loaded, err := config.Load(paths.Component, paths.Store)
if err != nil {
return err
}
Expand Down
14 changes: 7 additions & 7 deletions configs/README.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,7 @@
# Component configuration examples

Copy the file for the component you are deploying. Every field has a comment;
MongoDB is the active backend example, with a search-driver alternative in the
Engine and Worker files.
MongoDB is the active backend example in the shared Store file.

Both backend examples include file-based credential alternatives (`uri_file`,
`username_file`, `password_file`, and `api_key_file`). Remove the matching inline
Expand All @@ -13,8 +12,9 @@ API key. Credential files are read at startup; restart after rotating them.
| File | Component |
| --- | --- |
| [gateway.yaml](gateway.yaml) | Public request limits, inline Store routes, and forwarding capacity. |
| [engine.yaml](engine.yaml) | One Store's database, execution, batching, Lua, and optional Kafka publishing. |
| [worker.yaml](worker.yaml) | One Store's database, Kafka consumption, execution, and Lua. |
| [engine.yaml](engine.yaml) | Execution, batching, Lua, and Kafka publishing buffer. |
| [worker.yaml](worker.yaml) | Kafka consumption, execution, and Lua. |
| [stores/primary.yaml](stores/primary.yaml) | Shared Store identity, database connection and Kafka Topic policy. |

Every role serves `/livez` and `/readyz` on `health.address` (default `:8081`).
Prometheus uses its own `prometheus.address` (default `:9090`) and is disabled
Expand All @@ -23,15 +23,15 @@ unless `prometheus.enabled: true`. Disabling metrics never disables health check
Logs default to warn-level text on stderr. See the [logging reference](../docs/logging.md)
for JSON output, direct OTLP export, log levels and optional severe-failure bodies.

Gateway routes live under `gateway.routes` and all configured routes are active.
Gateway routes live under `forwarding.routes` and all configured routes are active.
Configuration is loaded at startup; restart the component after changes.

Validate without connecting to databases or Kafka, from the repository root:

```sh
sink config check --config configs/gateway.yaml
sink config check --config configs/engine.yaml
sink config check --config configs/worker.yaml
sink config check --config configs/engine.yaml --store-config configs/stores/primary.yaml
sink config check --config configs/worker.yaml --store-config configs/stores/primary.yaml
```

See the [configuration reference](../docs/configuration.md) for constraints and
Expand Down
Loading
Loading