Skip to content

fix(observability): drop the hard-coded MCP OTLP collector endpoint and enable Cloud Trace for MCP spans - #238

Merged
amenegola merged 1 commit into
mainfrom
fix/mcp-otlp-endpoint
Nov 17, 2026
Merged

amenegola merged 1 commit into
mainfrom
fix/mcp-otlp-endpoint

Conversation

@amenegola

Copy link
Copy Markdown
Collaborator

Closes #185

Problem

  • The mcp-service module set OTEL_EXPORTER_OTLP_ENDPOINT=http://collector.telemetry.internal:4317 on every MCP Cloud Run service. No environment deploys a collector there.
  • The variable had no effect at runtime: the services' otel_setup.py passes an explicit endpoint (telemetry.googleapis.com:443, Application Default Credentials) to the OTLP exporters, and an explicit endpoint wins over the variable. It was still misleading config.
  • The MCP services' span export was actually failing for a different reason. The MCP project didn't have cloudtrace.googleapis.com enabled, so every export to the Telemetry API returned PERMISSION_DENIED ("Cloud Trace API has not been used in project ... or it is disabled") and MCP spans were dropped.

Changes

  • mcp-service module: new otlp_endpoint variable (default "", validated as empty or an http(s):// URL). OTEL_EXPORTER_OTLP_ENDPOINT is set only when it isn't empty. The three service wrapper modules pass it through.
  • Units (dev and prd): otlp_endpoint = try(local.env_vars.locals.telemetry.otlp_endpoint, ""). A new telemetry.otlp_endpoint in env.yaml is empty in both environments.
  • Services (apps/services/*/otel_setup.py): _exporter_kwargs() uses OTEL_EXPORTER_OTLP_ENDPOINT when it is set (plaintext gRPC for http://, TLS for https://, no Google credentials). When it is unset or empty, the services export to the Google Cloud Telemetry API with ADC, the same as before. Unit tests cover unset, empty, http:// and https://, and check that real exporters accept the arguments. legacy-dms keeps its span exporter off, as before.
  • Layer 1: cloudtrace.googleapis.com added to the MCP project's APIs.
  • Guard: tests/guards/test_otlp_endpoint_guards.py (in make test-guards) fails when an OTLP endpoint is written as a literal in a module, a unit, env.yaml or a service's telemetry setup. It also checks that mcp-service sets the variable only from var.otlp_endpoint, and that committed env.yaml values stay empty until a collector unit exists. Self-tests prove the patterns catch the old hard-coded block.
  • Docs: new section "MCP Server Traces and Metrics (OTLP Export)" on the monitoring page, and a corrected sentence on the AgentOps overview.

Live test (dev)

  • Plans: layer-1-projects +1 (google_project_service.mcps["cloudtrace.googleapis.com"]). Each of the 3 service units: ~1, Cloud Run template only. The same units planned from main were 0/0/0, so the only diff is the removed env var.
  • Applied (destroy guard on). The services now have no OTEL_* env vars and their images are unchanged.
  • Re-plan of all 4 units: 0/0/0.
  • esmeralda verify --env dev: gateway 8/8, registry egress 18/18, specialist and orchestrator queries pass. esmeralda verify --partner --env dev: 7/7.
  • Telemetry, checked after the apply:
    • 0 OTLP export errors in the MCP service logs (there were 34 trace-export PERMISSION_DENIED errors in the 24 h before).
    • Traces from the MCP services now appear in Cloud Trace in the MCP project (/mcp spans).
    • All three services served requests during verify, including the expected 403s for the unregistered probe.

Follow-ups / notes

  • prd: the prd layer-1-projects plan will show the same +1 (cloudtrace.googleapis.com), and the 3 prd service units will show ~1 (env var removed).
  • Trace correlation: this PR doesn't show MCP spans in the same trace as the calling agent; they are in the MCP project's traces. Joining them is left to Reference OpenTelemetry Collector for non-Python workloads #157 (reference OpenTelemetry Collector) or a follow-up on trace-context propagation through the gateway.
  • CI: the repo has no CI job that runs make test-guards yet (only the gateway, docs and lock-check workflows run). The new guard runs in make test-guards / make test-all.

Tests

make test (corporate-email 10, income-verification 13, legacy-dms 19, agents + package 764, release 11), make test-guards (246), make test-terraform, make docs-build, esmeralda guard --diff, leak scan: all pass.

…nd enable Cloud Trace for MCP spans (#185)

The mcp-service module set OTEL_EXPORTER_OTLP_ENDPOINT to a collector that no environment deploys. otlp_endpoint is now a module variable (env.yaml telemetry.otlp_endpoint, empty in dev and prd): the env var is set only when it is non-empty. The services' otel_setup.py honours it when set (plaintext for http://, TLS for https://, no Google credentials) and otherwise exports to the Google Cloud Telemetry API with ADC, as before.

The MCP project lacked cloudtrace.googleapis.com, so every span export to the Telemetry API failed with PERMISSION_DENIED; layer 1 now enables it. A guard (tests/guards/test_otlp_endpoint_guards.py) fails on a literal OTLP endpoint in modules, units, env.yaml or the services' telemetry setup.

Closes #185
@amenegola
amenegola merged commit 0dcd306 into main Nov 17, 2026
9 checks passed
@amenegola
amenegola deleted the fix/mcp-otlp-endpoint branch November 17, 2026 02:15
@amenegola amenegola mentioned this pull request Nov 17, 2026
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: MCP services export OpenTelemetry to a collector that doesn't exist

1 participant