Log Analyzer

A multi-tenant log ingest service. 5.35M logs/sec hot-path throughput; p99 latency 9.3 ms; 100% match rate. Single MacBook, batch 1000, concurrency 32. Full numbers in Performance.

Each record is classified into a template (a regex pattern with named variables) and persisted to ClickHouse alongside the template's content-hashed id. The resulting template histograms support distribution-shift analytics — Jensen-Shannon divergence between any two time windows — which the grafana-hover-plugin uses to answer "which log shapes correlate with the moment I'm hovering on?"

Logs arrive over OTLP/gRPC on port 4317. Templates are generated by an in-memory parse tree at the hot path; templates that fail to match are forwarded to a single-consumer LLM worker that synthesizes a regex pattern, and subsequent records of that shape match in-memory.

Quick start

# 1. Bring up ClickHouse (parent docker-compose contains the schema)
cd ../ && docker compose up -d clickhouse && cd log_analysis

# 2. Configure the LLM in .env
#    LLM_PROVIDER=openai
#    LLM_MODEL=gpt-4o
#    LLM_API_KEY=sk-...
#    (or LLM_PROVIDER=ollama with a local model)

# 3. Run the ingest service (OTLP/gRPC + HTTP diagnostics)
cargo run --release --bin log-ingest-service

# 4. Verify
curl http://localhost:3002/health
# {"status":"healthy","templates_loaded":3,"clickhouse_connected":true}

Ingest: OTLP/gRPC (port 4317)

The service implements opentelemetry.proto.collector.logs.v1.LogsService. Point any OTel-compatible collector at localhost:4317 and export logs. Resource attributes drive tenant identification:

Resource attribute	Maps to	Default
org_id	tenant id	"default"
log_stream_id	per-stream id	"default-stream"
log_stream_name	human-readable name	== log_stream_id
service.name	service	empty
cloud.region	region	empty

The log message comes from LogRecord.body (string-typed records only; non-string bodies are skipped).

Diagnostics (HTTP, port 3002)

• GET /health — liveness, templates loaded, ClickHouse connectivity.
• GET /stats — current matcher state.

Performance

Measured on a single MacBook against local Docker ClickHouse with batch size 1000 and concurrency 32:

Metric	Value
Hot-path throughput	~5.35M logs/sec
p50 latency	4.4 ms
p99 latency	9.3 ms
Match rate	100%
Delivery (rows landed)	100%

LLM cold-path latency, per novel shape, by provider:

• gpt-4o: ~250 ms.
• o1: ~3–5 s; produces structurally tighter patterns.
• llama3:8b (Ollama): ~1 s; patterns are coarser.

Configuration

.env (loaded via dotenvy):

CLICKHOUSE_URL=http://localhost:8123
INGEST_PORT=3002              # HTTP diagnostics port
OTLP_GRPC_PORT=4317           # OTLP/gRPC ingest port

LLM_PROVIDER=openai           # openai | anthropic | ollama
LLM_MODEL=gpt-4o
LLM_API_KEY=sk-...

# Or for multi-LLM consensus:
# LLM_CONFIG_FILE=./llm-config.json

See LLM_CONFIG.md for the full provider/consensus shape.

Library usage

The matcher is usable standalone:

use log_analyzer::log_matcher::{LogMatcher, LogTemplate};
use log_analyzer::template_id::template_id_from_pattern;

let matcher = LogMatcher::new();

// IDs are content-derived hashes — the same pattern always produces
// the same ID across processes and restarts.
let pattern = r"User (\w+) logged in from (\d+\.\d+\.\d+\.\d+)";
matcher.add_template(LogTemplate {
    template_id: template_id_from_pattern(pattern),
    pattern: pattern.to_string(),
    variables: vec!["user".into(), "ip".into()],
    example: "User alice logged in from 10.0.0.1".into(),
});

let result = matcher.match_log("User bob logged in from 10.0.0.2");

// Batched matching for higher throughput
let logs = vec!["...", "..."];
let results = matcher.match_batch(&logs);

// Parallel matching for very large batches (>1000)
let results = matcher.match_batch_parallel(&logs);

Architecture

   OTel collector ──▶ OTLP/gRPC :4317 ──▶ LogMatcher (parse tree)
                                              ├─ HIT  ──▶ writer channel ──▶ buffered writer ──▶ ClickHouse
                                              └─ MISS ──▶ unmatched mpsc ──▶ LLM consumer  ──▶ ClickHouse
                                                                                              (logs + templates)

For deeper notes on the parse-tree, the rematch-before-LLM dedup, and the content-hash template-id scheme, see DESIGN.md.

Testing

# Unit + integration tests
cargo test --lib --release

# OpenStack grouping accuracy against a configured LLM
cargo test --test openstack_accuracy_test --release -- --nocapture

# LLM connectivity smoke check
cargo run --release --example llm_smoke_test

# End-to-end OTLP/gRPC perf test
cargo run --release --example perf_test_otlp
# Tunables: TOTAL_LOGS=200000 BATCH_SIZE=1000 CONCURRENCY=32 ORG_ID=mytest

Layout

log_analysis/
├── src/
│   ├── log_matcher.rs       # Aho-Corasick parse-tree matcher
│   ├── template_id.rs       # Content-hash IDs + canonicalization
│   ├── otlp_server.rs       # OTLP/gRPC LogsService (the ingest path)
│   ├── buffered_writer.rs   # Decoupled, channel-batched ClickHouse writer
│   ├── llm_service.rs       # Multi-provider LLM client (OpenAI/Anthropic/Ollama)
│   ├── llm_config.rs        # Provider / consensus configuration
│   ├── clickhouse_client.rs # ClickHouse persistence
│   ├── matcher_config.rs    # Tunables (fragment threshold, batch size, …)
│   └── bin/
│       ├── log-ingest-service.rs   # Main service binary
│       ├── regenerate-cache.rs     # Rebuild benchmark caches
│       └── sync-templates.rs       # Push cached templates into ClickHouse
├── examples/
│   ├── perf_test_otlp.rs          # End-to-end OTLP/gRPC perf test
│   ├── llm_smoke_test.rs          # LLM connectivity check
│   ├── llm_distinctness_test.rs   # Model-vs-matcher diagnostic
│   └── multi_llm_test.rs          # Multi-provider consensus example
├── tests/
│   ├── benchmarks.rs              # Throughput / parallel matcher benchmarks
│   ├── matcher_serialization_test.rs
│   └── openstack_accuracy_test.rs # End-to-end LLM template-generation accuracy
├── hover-schema/                  # ClickHouse schema (submodule)
└── docs:
    ├── DESIGN.md                  # Internal design notes (template-id stability, dedup, etc.)
    ├── INGEST_SERVICE_API.md      # OTLP ingest details
    ├── BENCHMARKS.md              # Benchmark guide
    ├── BENCHMARK_ALL_DATASETS.md  # LogHub dataset benchmarks
    └── LLM_CONFIG.md              # LLM provider configuration