Operations

Operations

On-disk layout

db_root/
  manifest/
    CURRENT                          (single line: latest valid generation u64)
    manifest-000001.json
    manifest-000002.json
    ...                              (last 10 generations retained)
  wal/
    wal-000001.log                   (numbered; <= last_sealed_wal_id are deletable)
    wal-000002.log
    ...
  raw_<uuid>.seg                     (raw segment files)
  compacted_<uuid>.seg               (post-compaction outputs)
  rollup_<uuid>.rseg                 (rollup segment files)
  tmp/                               (transient — cleaned on recovery)

Manifest generations (Phase A)

Every Manifest::save writes a new generation file (manifest-NNNNNN.json) and atomically advances CURRENT. The last 10 generations are retained; older ones are pruned after each save.

On startup, Manifest::load:

1. Reads CURRENT. Parses the generation number.
2. Tries loading manifest-NNNNNN.json for that generation. On corruption (parse error), walks backwards — manifest-NNNNNN-1, then NNNNNN-2, etc. — and uses the first one that parses cleanly, logging a warning.
3. Fails closed only if no generation parses. The error message points the operator at the directory for manual inspection.

If manifest/ doesn't exist but a legacy single-file manifest.json does, it's auto-migrated to generation 1 and the old file is deleted.

If neither exists, the DB is treated as fresh.

Durability mode

Set via Config.durability_mode:

• Strict (default): WAL flush + fsync before acking each batch. Billing-safe.
• Fast: WAL flush only — bytes hit the kernel page cache but no disk round-trip. Loses the tail batches on a host crash. Use for at-least-once upstream retry pipelines that tolerate this.

Balanced (spec §9.3 group commit) is documented as future work.

Rebuilding rollups

RollupWorker::rebuild_rollups(from_ms, to_ms) drops rollup segments overlapping [from_ms, to_ms), rewinds manifest.watermarks.hourly_rollup_ms to from_ms, and saves. The next worker tick refills the gap from raw segments.

Use cases:

• A rollup builder bug was fixed and cached rollups need to be regenerated.
• Late events arrived for a period that was already sealed and the rollups undercounted.
• Operator wants to verify rollup-vs-raw drift.

Recovery flow

1. Load manifest (generation directory; falls back through corrupt generations).
2. Remove tmp/ contents.
3. Delete WAL files with id ≤ last_sealed_wal_id (durably in segments).
4. Replay WAL files > last_sealed_wal_id into dedupe + memtable.
5. Scan raw segments within the dedupe TTL window (7 days) and re-register their event_id hashes in dedupe — so retries across restart of previously-committed events are detected.

Correction events

Correction (EventKind::Correction) and Retraction (EventKind::Retraction) events:

• Validation: must include a non-empty correction_ref { original_event_id, reason }.
• Rollups: the builder sums quantities uniformly, so a Correction event with quantity = -40 correctly subtracts from the original's +100 in the rollup row covering its hour. kind is intentionally not in the rollup key so net totals roll up correctly.
• Queries: filter or group by kind ("Usage", "Correction", "Retraction") to isolate adjustments for forensics. Raw audit queries should use the RawEvents source.

Validation

The ingest path rejects events that fail any of:

• empty event_id, account_id, product_id, meter_id
• timestamp_ms ≤ 0
• dimensions size > 16 (spec §21)
• Correction or Retraction without correction_ref

Rejected events do not reach the WAL or dedupe. The ingest response counts them under rejected; the server logs each one with a reason.

Compaction grace window

After a compaction commits (manifest swap atomic), old segment files are not deleted immediately. The ReplacementRecord carries a committed_at_ms timestamp, and the deletion happens on a future tick where now - committed_at_ms ≥ compaction_grace_ms (default 30 s). This protects queries that snapshotted the prior manifest from racing with file unlinks.