Repository navigation
fix(observability): drop the hard-coded MCP OTLP collector endpoint and enable Cloud Trace for MCP spans - #238
Merged
Merged
Conversation
…nd enable Cloud Trace for MCP spans (#185) The mcp-service module set OTEL_EXPORTER_OTLP_ENDPOINT to a collector that no environment deploys. otlp_endpoint is now a module variable (env.yaml telemetry.otlp_endpoint, empty in dev and prd): the env var is set only when it is non-empty. The services' otel_setup.py honours it when set (plaintext for http://, TLS for https://, no Google credentials) and otherwise exports to the Google Cloud Telemetry API with ADC, as before. The MCP project lacked cloudtrace.googleapis.com, so every span export to the Telemetry API failed with PERMISSION_DENIED; layer 1 now enables it. A guard (tests/guards/test_otlp_endpoint_guards.py) fails on a literal OTLP endpoint in modules, units, env.yaml or the services' telemetry setup. Closes #185
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #185
Problem
mcp-servicemodule setOTEL_EXPORTER_OTLP_ENDPOINT=http://collector.telemetry.internal:4317on every MCP Cloud Run service. No environment deploys a collector there.otel_setup.pypasses an explicit endpoint (telemetry.googleapis.com:443, Application Default Credentials) to the OTLP exporters, and an explicit endpoint wins over the variable. It was still misleading config.cloudtrace.googleapis.comenabled, so every export to the Telemetry API returnedPERMISSION_DENIED("Cloud Trace API has not been used in project ... or it is disabled") and MCP spans were dropped.Changes
mcp-servicemodule: newotlp_endpointvariable (default"", validated as empty or anhttp(s)://URL).OTEL_EXPORTER_OTLP_ENDPOINTis set only when it isn't empty. The three service wrapper modules pass it through.otlp_endpoint = try(local.env_vars.locals.telemetry.otlp_endpoint, ""). A newtelemetry.otlp_endpointinenv.yamlis empty in both environments.apps/services/*/otel_setup.py):_exporter_kwargs()usesOTEL_EXPORTER_OTLP_ENDPOINTwhen it is set (plaintext gRPC forhttp://, TLS forhttps://, no Google credentials). When it is unset or empty, the services export to the Google Cloud Telemetry API with ADC, the same as before. Unit tests cover unset, empty,http://andhttps://, and check that real exporters accept the arguments.legacy-dmskeeps its span exporter off, as before.cloudtrace.googleapis.comadded to the MCP project's APIs.tests/guards/test_otlp_endpoint_guards.py(inmake test-guards) fails when an OTLP endpoint is written as a literal in a module, a unit,env.yamlor a service's telemetry setup. It also checks thatmcp-servicesets the variable only fromvar.otlp_endpoint, and that committedenv.yamlvalues stay empty until a collector unit exists. Self-tests prove the patterns catch the old hard-coded block.Live test (dev)
layer-1-projects+1 (google_project_service.mcps["cloudtrace.googleapis.com"]). Each of the 3 service units: ~1, Cloud Run template only. The same units planned frommainwere 0/0/0, so the only diff is the removed env var.OTEL_*env vars and their images are unchanged.esmeralda verify --env dev: gateway 8/8, registry egress 18/18, specialist and orchestrator queries pass.esmeralda verify --partner --env dev: 7/7.PERMISSION_DENIEDerrors in the 24 h before)./mcpspans).Follow-ups / notes
layer-1-projectsplan will show the same +1 (cloudtrace.googleapis.com), and the 3 prd service units will show ~1 (env var removed).make test-guardsyet (only the gateway, docs and lock-check workflows run). The new guard runs inmake test-guards/make test-all.Tests
make test(corporate-email 10, income-verification 13, legacy-dms 19, agents + package 764, release 11),make test-guards(246),make test-terraform,make docs-build,esmeralda guard --diff, leak scan: all pass.