v0.3.0 - Datadog and Google Cloud Logging

v0.3.0 adds Datadog and Google Cloud Logging, completing the first full set of major log source adapters. SiftLog now correlates across file/stdin, Loki, CloudWatch, Elasticsearch, Datadog, and Google Cloud Logging simultaneously.

New adapters

Datadog Logs

Logs Search API v2 with cursor pagination
Authenticates via DD_API_KEY / DD_APP_KEY environment variables, or config-supplied keys
Trace ID extracted from structured attributes (trace_id, dd.trace_id, traceId)
Optional query filter field maps to Datadog log search syntax
Google Cloud Logging

REST API v2 via Application Default Credentials - works with service account key files or GCP workload identity
Service name resolved from labels, resource labels (GKE container, Cloud Run service, App Engine module), or log name
Trace ID from jsonPayload fields (trace_id, traceId, request_id) or top-level trace field
Optional query field maps to Cloud Logging filter syntax (e.g. severity>=WARNING)

Configuration

sources:
  - name: datadog-prod
    type: datadog
    auth:
      token_env: DD_API_KEY
      app_key_env: DD_APP_KEY
    query: "service:payment-service"

  - name: gcp-backend
    type: googlecloud
    project: my-gcp-project
    query: "severity>=WARNING"

What's next

v1.0.0 targets a stable config schema, --replay flag, and Homebrew distribution.

Platform track (siftlogd - always-on daemon with persistent baselines, alerting, and web UI) is in active development.

Full changelog: v0.2.0...v0.3.0

v0.2.0 - Live Streaming

Live streaming mode for real-time log correlation.

• --live flag: tail file sources continuously after EOF (truncation detection included)
• Flush ticker: oldest buffered event emitted across all sources within --flush-ms (default 500ms)
• --window flag wired: override correlation window from CLI (default raised from 500ms to 5000ms)
• --flush-ms flag: tune live mode latency vs. ordering tradeoff
• Remote adapters (Loki, CloudWatch, Elasticsearch) now fully wired into the dispatch path
• Cascade and silence detection confirmed working in live mode

The open source CLI is available at github.com/mmediasoftwarelab/siftlog.

v0.1.0 - First working release

SiftLog is a CLI tool for distributed log correlation. Point it at your logs and it surfaces the signal: what broke, in what order, which service caused it.

What's in this release

Signal detection

• Cascade detection - identifies when one service's failures propagate to dependents, using trace ID correlation and temporal co-failure patterns. Builds and displays the full chain: payment-service → order-service → notification-svc
• Anomaly rate detection - flags services whose error rate exceeds a configurable multiple of their recent baseline
• Silence detection - flags services that drop below their expected log volume, with a bootstrap period and deduplication

Log sources

• File / stdin - JSON structured logs and plain text, 8 timestamp formats
• Grafana Loki - query_range API, bearer auth, label-based LogQL queries
• AWS CloudWatch Logs - FilterLogEvents, multi-group, IAM credential chain
• Elasticsearch / OpenSearch - scroll API, API key + basic auth

Output

• Human (default) - colored terminal output, quiet mode, verbose mode
• JSON (--output json) - newline-delimited, pipeable to jq

Per-source timestamp offset - correct systematic clock skew per source before correlation, not by widening the window

Install

go install github.com/mmediasoftwarelab/siftlog@latest

Quickstart

siftlog app.log worker.log
siftlog --since 15m --quiet app.log
siftlog --output json --quiet app.log | jq 'select(.is_signal)'