Skip to content

Latest commit

 

History

History
308 lines (253 loc) · 14.3 KB

operations.md

File metadata and controls

308 lines (253 loc) · 14.3 KB

Operator runbook

Pitot is one process and one persistent data directory. Operate those two things as a unit: preserve the directory, pin the binary or image version, and give ingest credentials only to trusted services and collectors.

Runtime secret delivery

Runtime configuration uses built-in defaults followed by nonempty process environment values. Every setting requires a restart. Pitot reads no configuration document or dotenv file. Generated .env.example lists the settings as commented-out empty entries, with types, defaults, and secret file forms in comments. Copying the template sets nothing and preserves image ENV; uncomment only settings you want to supply. For development, just run loads optional ignored ./.env with Node's native parseEnv; inherited values win, including empty values. Personal development credentials may live there. Personal environment files and operator profiles are excluded from Git and Docker.

Custom names use PITOT_ plus a clear name, adding the area only when it clarifies. Vendor credentials and endpoints keep their documented names. Unknown PITOT_ names fail startup, except platform service-discovery names; unrelated names in shared vendor namespaces are ignored.

Environment and mounted secret files

Deliver credentials through deployment environment injection or managed mounts. Every secret accepts either its value or its file path:

Value variable File path variable
PITOT_AUTH_CODE_SECRET PITOT_AUTH_CODE_SECRET_FILE
PITOT_OIDC_CLIENT_SECRET PITOT_OIDC_CLIENT_SECRET_FILE
PITOT_SMTP_PASSWORD PITOT_SMTP_PASSWORD_FILE
PITOT_INGEST_TOKEN PITOT_INGEST_TOKEN_FILE
PITOT_METRICS_TOKEN PITOT_METRICS_TOKEN_FILE
PITOT_SCIM_TOKEN PITOT_SCIM_TOKEN_FILE
ANTHROPIC_API_KEY ANTHROPIC_API_KEY_FILE
OPENAI_API_KEY OPENAI_API_KEY_FILE

Both forms nonempty fail startup. Pitot strips exactly one trailing LF or CRLF from file contents, then rejects an empty result. Errors identify the variable and path, never the contents. Pitot performs no file-mode check; the deployment controls mount access. Do not bake credentials into images or frontend bundles. Unselected AI providers' value and file forms are ignored.

For Docker, mount a provisioned secret and inject its path:

docker run --name pitot --restart unless-stopped -p 7520:7520 \
  -v pitot-data:/var/lib/pitot/data \
  --mount type=bind,src="$AUTH_SECRET_PATH",dst=/run/secrets/auth_code,readonly \
  -e PITOT_AUTH_CODE_SECRET_FILE=/run/secrets/auth_code \
  labstack/pitot:latest

The source secret must be readable by the image's UID 65532. In Compose, grant only the required secret to the service:

services:
  pitot:
    image: labstack/pitot:latest
    ports: ["7520:7520"]
    volumes: ["pitot-data:/var/lib/pitot/data"]
    environment:
      PITOT_AUTH_CODE_SECRET_FILE: /run/secrets/auth_code
    secrets: [auth_code]
secrets:
  auth_code:
    file: ${AUTH_SECRET_PATH}
volumes:
  pitot-data:

For Kubernetes, put ordinary settings in a ConfigMap and credentials in a Secret provisioned through your platform's secret manager. This Pod fragment injects both without putting credential values in the manifest:

containers:
  - name: pitot
    image: labstack/pitot:latest
    envFrom:
      - configMapRef:
          name: pitot-settings
    env:
      - name: PITOT_AUTH_CODE_SECRET
        valueFrom:
          secretKeyRef:
            name: pitot-credentials
            key: auth_code_secret

Alternatively mount the Secret as files and set the corresponding _FILE paths. Use the existing persistent-volume and access policy for telemetry data.

SOPS runtime injection

Use one authoritative SOPS-encrypted profile per actual workload/credential set, such as ignored secrets/pitot.env. Select a complete profile explicitly; never layer different credential sets or fall back on a missing profile. Platform-owned credentials remain platform-owned. This public repository ships no operator secrets, recipient policy, or decryption keys.

With a provisioned SOPS key and existing encrypted profile, launch through SOPS runtime injection without writing decrypted credentials to disk:

sops exec-env secrets/pitot.env './bin/pitot'
# For local development with optional personal settings:
sops exec-env secrets/pitot.env 'just run'

The profile supplies environment values to the child; ordinary settings may also come from deployment inputs. A dotenv parser neither decrypts SOPS nor resolves 1Password references. Verify synthetic values containing dollars, quotes, spaces, and newlines through the actual deployment parser before using a profile. Keep private age identities out of Git, restrict their local files to mode 0600, and document key recovery in the existing 1Password item. 1Password holds recovery rather than a second maintained copy of SOPS-owned application secrets.

AI chat is disabled by default. Select exactly anthropic or openai with PITOT_AI_PROVIDER, then deliver its required ANTHROPIC_API_KEY or OPENAI_API_KEY. A selected provider without its key fails startup. Inherited keys never enable chat. Only the selected endpoint is validated and used. Overrides are ANTHROPIC_BASE_URL and OPENAI_BASE_URL; the OpenAI base includes /v1, while Anthropic's excludes it.

Network surface

Default Purpose Exposure
127.0.0.1:7520 (native), :7520 (image) Browser, API, MCP, health, metrics, OTLP/gRPC and OTLP/HTTP Users, trusted services or a reverse proxy

Both native and container deployments use one listener. Native installs default to loopback; set PITOT_ADDR=:7520 for remote access. The image supplies PITOT_ADDR=:7520 and PITOT_DATA_DIR=/var/lib/pitot/data. Every OTLP request requires an active ingest token, created during first-admin setup, by an administrator, or from PITOT_INGEST_TOKEN on an empty install; browser sessions and MCP tokens do not authorize ingest. Do not embed the ingest token in a browser or shipped mobile application.

For OTLP/HTTP, set http/protobuf, use the base endpoint http://pitot:7520, and supply Authorization: Bearer <token>. Exporters derive /v1/traces, /v1/metrics, and /v1/logs from the general endpoint. Signal-specific endpoint variables require the full path. OTLP/gRPC uses the same address and authorization value, over HTTP/2.

Mobile boundary

Pitot remains a backend receiver, not a public mobile collector. The supported path is mobile SDK -> customer-controlled public Collector or gateway -> private Pitot; the gateway owns public-client authentication, rate limits, privacy policy, and abuse controls. Never compile Pitot's shared ingest token into a shipped application.

Propagate the standard traceparent and tracestate headers from mobile HTTP requests to backend services. When the customer's gateway exports client spans to Pitot, that context correlates them with the backend trace without a Pitot mobile SDK or a public Pitot listener.

TLS and reverse proxies

Setting both PITOT_TLS_CERT_FILE and PITOT_TLS_KEY_FILE enables TLS 1.3 on the shared listener. A proxy must forward HTTP/2 to Pitot for gRPC: use h2c for a plaintext backend, or HTTP/2 over TLS when Pitot has a certificate. HTTP/1.1 forwarding supports OTLP/HTTP, but cannot carry gRPC. PITOT_TLS_CLIENT_CA_FILE additionally requires every client, including a TLS-terminating proxy such as Cloudflare with Authenticated Origin Pulls, to present a certificate from that CA bundle. The bundle is read at startup, so rotation needs a restart; keep the old and new CA in the bundle while clients move to certificates from the new one.

Set PITOT_PUBLIC_URL to the externally reachable HTTPS browser origin. When using forwarded client addresses, set PITOT_TRUSTED_PROXY_CIDRS to only the proxy networks; never use 0.0.0.0/0. The MCP resource URL is the same origin with /mcp appended. Pitot accepts public MCP transport requests only when their HTTP Host matches that origin, so the reverse proxy must preserve the original host.

MCP also validates Origin before authentication; see Connect an agent over MCP.

Health and monitoring

pitot healthcheck derives /healthz from PITOT_ADDR or its built-in default. Empty, 0.0.0.0, and :: bind hosts become 127.0.0.1. PITOT_TLS_CERT_FILE selects HTTPS. Derived probes bypass environment proxies. HTTPS IP-literal probes trust only the configured certificate and verify its first DNS or IP SAN, which may differ from the bind IP; hostname probes verify normally. Derived HTTPS probes present PITOT_HEALTHCHECK_CERT_FILE and PITOT_HEALTHCHECK_KEY_FILE when set, so they pass a listener with PITOT_TLS_CLIENT_CA_FILE. Healthcheck reads only these settings and does not require valid secrets. An explicit pitot healthcheck <url> uses default proxy and certificate verification behaviour.

  • /healthz is the liveness check. It fails only when Pitot's own work on the query engine stalls: a Parquet publication holds reads for 10 minutes, or an idle engine does not answer a ping while no read holds it. An overloaded engine stays alive, and so does one running a query past its deadline.
  • /readyz includes storage readiness and resolved runtime sizing. A busy engine reports degraded with 200; a stalled one reports 503. A query still running 25 seconds past its deadline reports queries as degraded, logs a warning with a hash of the query (never its text), and increments pitot_duckdb_statement_overruns_total. In OIDC mode, the oidc check reports degraded with 200 while provider discovery is failing; only new OIDC sign-ins are affected, and discovery retries without a restart.
  • /metrics exposes Prometheus metrics. Keep it private or configure PITOT_METRICS_TOKEN; do not make it public merely to simplify scraping.

Monitor admission and compaction together:

  • pitot_ingest_throttled_total{reason} counts refused whole exports by debt, reads, or budget. Exporters need retries enabled.
  • pitot_ingest_admission_rows_per_second is the current row-rate cap; zero means uncapped.
  • pitot_ingest_backpressure_seconds_total measures time with an admission cap active. Earlier releases measured blocked-writer time; do not compare them directly.
  • pitot_compaction_backlog_batches is total generation-0 inventory, including batches excluded from reclaimable debt.
  • pitot_compaction_pass_seconds measures pass duration, and pitot_compaction_retired_batches_total counts input batches retired by successful publications. Compare its rate with backlog growth.

Run Pitot under a supervisor that restarts it on exit: Docker or Compose with restart: unless-stopped (--restart unless-stopped), systemd with Restart=on-failure, or Kubernetes. Docker's HEALTHCHECK only marks a container unhealthy; it does not restart it. Pitot exits with status 1 when the query engine fails beyond recovery, for example when it runs out of memory while writing its cache, or when a stall that fails /healthz lasts another 25 seconds. It first finishes writing received telemetry; the restart rebuilds the query cache from stored telemetry.

Alert on repeated restarts, readiness failures, ingest authentication failures, telemetry drops, sustained query latency, and available disk space. Retention is controlled by PITOT_RETENTION_DAYS; maintenance removes expired data on its configured cycle rather than immediately when the setting changes.

Control database migrations

Pitot uses SQLite with the modernc driver, sqlc query bindings, and embedded Goose SQL migrations. Pending migrations run forward at startup. PostgreSQL and pgx are not required.

Backup

The supported portable baseline is a cold backup of the complete PITOT_DATA_DIR. It contains atomic telemetry Parquet batches, query state, and the control SQLite database; copying only one subdirectory does not produce a recoverable installation.

  1. Record the running Pitot version and configuration, excluding secrets from ordinary logs or tickets.
  2. Stop Pitot cleanly and wait for the process to exit. Shutdown stops the shared listener and allows five seconds for active requests before closing remaining connections, then drains the telemetry commit workers.
  3. Snapshot or copy the complete data directory with ownership and permissions preserved.
  4. Restart Pitot and confirm /readyz.
  5. Periodically restore a backup into an isolated directory; an untested backup is not a recovery plan.

Do not use a normal recursive copy while Pitot is writing. A storage-level snapshot is acceptable only when it provides a crash-consistent point-in-time view of the entire data directory.

Restore

  1. Stop the destination Pitot process.
  2. Move the existing data directory aside rather than mixing it with a backup.
  3. Restore the complete directory and its original ownership and permissions.
  4. Start the same Pitot version that created the backup and verify /readyz, sign-in, and a known telemetry query.
  5. Upgrade separately after the restored installation is known healthy.

Never recover an account by deleting its row from the control database; that cascades to sessions, OAuth grants, and owned dashboards. Use pitot login-link <email> as documented in authentication.md.

Upgrade and rollback

  1. Read the release notes and pin the target image tag or binary version.
  2. Take and verify a cold backup.
  3. Stop Pitot, replace only the binary/image, and retain the data directory.
  4. Start Pitot and check /readyz, authentication, both configured OTLP transports, and representative queries.

Database migrations run forward during startup. A binary rollback after migration is not assumed safe. Restore the pre-upgrade backup with the previous binary when rollback is required.

Local users without SMTP

SMTP is optional. Creating a local user without it succeeds and returns invite_delivery: "not_configured" with login_link_required: true. An operator with access to the same configuration and data directory then runs:

pitot login-link user@example.com

The resulting link expires after 15 minutes, works once, and is recorded in the authentication audit history. A configured SMTP relay that fails delivery is reported as an error rather than silently claiming that an invitation arrived.