OTel - Log signal for successfully routed anonymous requests

[balhar-jakub]

When APIML successfully (the service is onboarded) routes an anonymous request (without any authentication) to a donwnstream service, produce a log signal with attributes:

• time of event (timestamp)
• apiml url path (url.path)
• apiml url scheme (url.scheme)
• apiml http method (http.request.method)
• apiml error type (error.type)
• apiml error message (error.message)
• user id (user.id)
• service id (service.id)
• service instance id (service.instance.id)
• service response code (service.response.status_code)

Acceptance criteria:

• If the request does not have a User ID (e.g., an unauthenticated public endpoint):
  • The user.id attribute should be recorded as "anonymous", rather than preventing the log generation.
• Acceptance tests
• Documentation updated

[balhar-jakub]

Problem Analysis — OTel Log signal for successfully routed anonymous requests (#4704)

Refined Problem Statement

When APIML's Gateway successfully routes a request to a downstream service where no authentication is required (i.e., the service is configured with AuthenticationScheme.BYPASS or no auth scheme), the existing OTel log signal infrastructure does not emit a complete log entry. The OtelServiceFilterFactory already sets authMethod to BYPASS along with serviceId and instanceId, but it does not set user.id to any value — the userId() method on OtelRequestContext is never called for anonymous/bypass requests. The issue requests that when a request is anonymous (no user identity), the user.id attribute should be recorded as "anonymous" rather than being omitted entirely.

In a real-world scenario: an operator monitoring OTel signals wants to see every routed request in their observability stack. Today, anonymous requests produce an OTel log signal but without a user.id attribute, making it impossible to distinguish authenticated vs. anonymous traffic in dashboards and alerts. Recording user.id = "anonymous" provides a consistent signal shape for all successfully routed requests.

Current Architecture Touchpoints

1. OtelRequestContext (gateway-service/.../product/opentelemetry/OtelRequestContext.java)

   • Central context object that accumulates OTel attributes per request via a builder pattern.
   • Lives as an exchange attribute under key "otel-context".
   • userId(String) sets user.id (uppercased); authMethod(AuthenticationScheme) sets auth.service.auth.method.
   • issue() serializes all attributes to JSON and logs via SLF4J logger org.zowe.apiml.opentelemetry.
   • Currently has no concept of an "anonymous" user — if userId() is never called, the attribute is simply absent.

2. OtelServiceFilterFactory (gateway-service/.../filters/OtelServiceFilterFactory.java)

   • Applied as a per-route filter when OTel is enabled (otel.sdk.disabled=false).
   • Sets authMethod(BYPASS), serviceId, and instanceId on the OtelRequestContext.
   • This is the key extension point for the anonymous request case — it already handles bypass routes but does not set user.id.

3. OtelRequestFilter (apiml/.../filter/OtelRequestFilter.java)

   • Global filter that initializes the OtelRequestContext for every request (sets method, scheme, path, serviceId, instanceId defaults).
   • Calls otelContext.issue() in doFinally() to emit the log signal after the response.
   • Reads the HTTP response status code and sets service.response_code.

4. Authentication filters (AbstractAuthSchemeFactory, AbstractTokenFilterFactory, X509FilterFactory, PassticketFilterFactory, etc.)

   • Authenticated requests have their userId set downstream in ZaasSchemeTransformApi after successful authentication.
   • AbstractAuthSchemeFactory.cleanHeadersOnAuthSuccess() calls authenticationSuccess().
   • AbstractAuthSchemeFactory.cleanHeadersOnAuthFail() calls authenticationFailed() + authErrorMessage().
   • The bypass path has NO corresponding userId or auth-status call.

5. RouteLocator (gateway-service/.../service/RouteLocator.java)

   • Adds OtelServiceFilterFactory as a post-routing filter when OTel is enabled.
   • Determines the auth scheme per route from service metadata.

6. OTel test infrastructure (apiml/.../acceptance/OpenTelemetryTestConfig.java, otel/ directory)

   • In-memory metric reader, log record exporter for acceptance tests.
   • Docker-based OTel Collector + Golden Tester for integration validation.

7. Existing attribute names in OtelRequestContext:

   • http.request.method, url.scheme, url.path, service.response_code (note: underscore, not dot as in the issue), service.id, service.instance.id, auth.service.auth.method, auth.method, auth.status, auth.error.type, auth.error.message, user.id, user.distributed.id

Scope

In scope:

• When OtelServiceFilterFactory applies to a bypass/anonymous route, set user.id = "anonymous" on the OtelRequestContext.
• Set auth.status = "OK" for successfully routed anonymous requests (currently absent for bypass routes).
• Ensure the error.type and error.message attributes can be set if errors occur during anonymous routing (these are listed in the issue's attributes but currently not set for bypass routes).
• Add/update unit tests for OtelServiceFilterFactory and OtelRequestContext to cover the anonymous user.id case.
• Add acceptance tests verifying the full log signal is emitted with correct attributes for anonymous requests.
• Update documentation to describe the OTel log signal for anonymous/bypass requests.

Out of scope:

• Issue OTel - Log signal for routed requests to unknown service id #4705 (log signal for unknown/unroutable service IDs) — that is a separate, parallel feature.
• Changing the existing attribute names (e.g., service.response_code vs. service.response.status_code as listed in the issue) — this should be clarified but is likely a separate concern.
• Changes to the OTel Collector or Golden Tester configuration.
• New OTel signals (metrics, traces) — this issue is about log signals only.

Acceptance Criteria

1. When APIML successfully routes an anonymous/bypass request to a downstream service, the OTel log signal SHALL include user.id = "anonymous".
2. When APIML successfully routes an anonymous/bypass request, the OTel log signal SHALL include auth.status = "OK" (consistent with authenticated successful routes).
3. The log signal for anonymous requests SHALL include all currently supported attributes: http.request.method, url.scheme, url.path, service.id, service.instance.id, auth.service.auth.method (set to bypass), service.response_code, user.id, and auth.status.
4. If an error occurs during routing of an anonymous request, error.type and error.message SHALL be populated (aligned with the existing auth.error.type/auth.error.message pattern, or the generic error.type/error.message per OTel semantic conventions).
5. Unit tests verify user.id = "anonymous" is set for bypass routes in OtelServiceFilterFactory.
6. Acceptance tests verify the complete OTel log signal for anonymous requests.
7. Documentation is updated to describe the OTel log signal attributes for anonymous/bypass requests.

Design Questions for the Architect

1. Attribute naming discrepancy: The issue lists error.type, error.message, and service.response.status_code, but the existing codebase uses auth.error.type, auth.error.message, and service.response_code. Should the anonymous-request attributes follow the existing naming convention (prefixed with auth.), or should new top-level error.type/error.message attributes be introduced? The latter would align with OpenTelemetry semantic conventions but diverge from the existing pattern.

2. Where to set user.id = "anonymous": The simplest approach is in OtelServiceFilterFactory.apply() — add .userId("anonymous") alongside the existing .authMethod(BYPASS). Alternatively, should this be set in OtelRequestFilter.setDefaults() as a default that gets overwritten by authenticated filters? The trade-off is: setting it in the factory is explicit and localized; setting it as a default ensures no request ever has a missing user.id but may require more careful ordering.

3. auth.status for bypass routes: Should OtelServiceFilterFactory also call .authenticationSuccess() (setting auth.status = "OK"), or is the authMethod(BYPASS) sufficient to indicate success? If auth.status is expected in the signal for consistency with authenticated routes, this needs to be explicit.

4. Should OtelRequestContext have a dedicated anonymousUserId() method that sets user.id = "anonymous" (potentially without the uppercasing that userId() applies), or should we just call userId("anonymous") which would produce "ANONYMOUS"? The issue says "anonymous" (lowercase), but the existing convention uppercases user IDs.

5. Integration with issue OTel - Log signal for routed requests to unknown service id #4705: The sibling issue for unknown service IDs also needs user.id = "anonymous" handling. Should the anonymous user-id logic be shared between both features (e.g., a shared constant or utility method)?

6. Log signal timing: Currently OtelRequestFilter emits the log in doFinally(). For bypass routes where the OtelServiceFilterFactory is a per-route filter, the attributes are populated after the global filter initializes the context but before doFinally() fires. Is this ordering guaranteed and sufficient, or does the architect need to consider race conditions?

Constraints

• Backward compatibility: Existing OTel log signals for authenticated requests must not change. The user.id attribute for authenticated requests must continue to be uppercased.
• OTel SDK configuration: The feature is gated by otel.sdk.disabled (default: true). When disabled, no OTel code paths execute.
• OpenTelemetry semantic conventions: Where possible, attribute names should align with OTel semantic conventions (e.g., user.id, error.type, http.request.method are already OTel-convention-aligned).
• Performance: The OTel context is per-request and uses Attributes.builder(). Adding one more attribute (user.id = "anonymous") has negligible overhead.
• Testing: Acceptance tests use the @AcceptanceTest annotation with InMemoryLogRecordExporter (see OpenTelemetryTestConfig). New tests should follow this pattern.

[balhar-jakub]

Solution Design — OTel Log signal for successfully routed anonymous requests (#4704)

High-Level Architecture

The change is localized to two files in gateway-service. The core insight: OtelServiceFilterFactory already handles bypass routes (setting authMethod(BYPASS), serviceId, instanceId) but omits user.id and auth.status. We close that gap with two targeted additions.

Data flow for anonymous requests:

Request → OtelRequestFilter (global: set defaults) → Route match → OtelServiceFilterFactory (per-route: set BYPASS + anonymous user + OK status) → Downstream → doFinally (issue log)

Tech Choices with Rationale

1. Add anonymousUserId() to OtelRequestContext — dedicated method that sets user.id = "anonymous" WITHOUT uppercasing. Rationale: the existing userId() uppercases via StringUtils.upperCase(), which would produce "ANONYMOUS". The issue specifies lowercase "anonymous", and this is semantically a sentinel value, not a real user ID. A dedicated method makes intent explicit and avoids backward-incompatible changes to userId().

2. Shared constant ANONYMOUS_USER_ID — public static final String ANONYMOUS_USER_ID = "anonymous" in OtelRequestContext. This serves both issue OTel - Log signal for successfully routed anonymous requests #4704 and OTel - Log signal for routed requests to unknown service id #4705 (unknown service IDs).

3. Set auth.status = "OK" in OtelServiceFilterFactory — add .authenticationSuccess() to the builder chain. Bypass = no auth required = the request proceeds normally. This is consistent with how authenticated flows call authenticationSuccess() in cleanHeadersOnAuthSuccess().

4. No changes to OtelRequestFilter.setDefaults() — defaults should remain untouched. Setting user.id = "anonymous" as a default would create a confusing two-phase pattern where authenticated filters have to overwrite it.

Design Questions Answered

Q1: Attribute naming discrepancy
Follow the existing naming convention in OtelRequestContext: auth.error.type, auth.error.message, service.response_code. The issue lists bare error.type/error.message but the codebase already prefixes auth-related attributes with auth.. The user.id attribute is already correct per OTel semantic conventions. These attributes are already defined in OtelRequestContext and used by authenticated flows — anonymous requests will reuse them.

Q2: Where to set user.id = "anonymous"
In OtelServiceFilterFactory.apply(). Adding .anonymousUserId() alongside .authMethod(BYPASS) keeps the logic localized to the bypass case only. Setting it in setDefaults() would require authenticated filters to overwrite it, creating unnecessary complexity.

Q3: auth.status for bypass routes
Yes, call .authenticationSuccess() in OtelServiceFilterFactory. This sets auth.status = "OK", making the signal shape consistent with authenticated successful routes. Bypass = request proceeds normally = success.

Q4: anonymousUserId() vs. userId("anonymous")
Add a dedicated anonymousUserId() method that sets user.id = "anonymous" directly (no uppercasing). This:

• Preserves backward compatibility of userId() (uppercased for real user IDs)
• Makes the sentinel semantics explicit
• Enables a shared constant for OTel - Log signal for routed requests to unknown service id #4705

Q5: Integration with #4705
Yes. The ANONYMOUS_USER_ID constant and anonymousUserId() method in OtelRequestContext can be reused by #4705 (unknown service IDs). A single constant ensures both features produce the same user.id value.

Q6: Log signal timing
The ordering is guaranteed. Spring Cloud Gateway's filter chain ensures:

1. OtelRequestFilter (global, Ordered.HIGHEST_PRECEDENCE + 2) fires first → setDefaults() initializes context
2. chain.filter(exchange) proceeds through route-specific filters
3. OtelServiceFilterFactory (per-route) fires synchronously, setting auth attributes
4. doFinally() fires after the full chain completes → issue() emits the log

Per-route filters always execute between setDefaults() and doFinally(). No race conditions exist — the reactive pipeline guarantees sequential execution.

Files to Create/Modify

File	Action	Summary
gateway-service/.../OtelRequestContext.java	Modify	Add ANONYMOUS_USER_ID constant and anonymousUserId() method
gateway-service/.../OtelServiceFilterFactory.java	Modify	Add .anonymousUserId() and .authenticationSuccess() to apply()
gateway-service/.../OtelRequestContextTest.java	Modify	Add tests for anonymousUserId() and constant
gateway-service/.../OtelServiceFilterFactoryTest.java	Modify	Verify user.id="anonymous" and auth.status="OK" are set
apiml/.../acceptance/	Create	New acceptance test class for anonymous OTel signals
docs-site/...	Modify	Document OTel log attributes for anonymous requests

Independent Areas (for PM decomposition)

1. Area 1: OtelRequestContext changes — Add constant + anonymousUserId() method. File: OtelRequestContext.java in gateway-service. Self-contained, no dependencies on other areas.

2. Area 2: OtelServiceFilterFactory changes — Add .anonymousUserId() and .authenticationSuccess() calls. File: OtelServiceFilterFactory.java in gateway-service. Depends on Area 1 being done first (needs the anonymousUserId() method).

3. Area 3: Unit tests — Update both test files: OtelRequestContextTest and OtelServiceFilterFactoryTest. Can be done together with Areas 1 and 2 in a single engineer task (tests for both production changes in one test run).

4. Area 4: Acceptance tests — New test class in apiml/src/test/java/.../acceptance/. Verifies end-to-end OTel log signal for anonymous requests. Independent of areas 1-3 (tests against the compiled artifact using InMemoryLogRecordExporter pattern from OpenTelemetryTestConfig).

5. Area 5: Documentation — Update docs-site to describe user.id = "anonymous" and the full OTel log attribute set for anonymous/bypass requests.

Edge Cases and Risks

• OTel SDK disabled: When otel.sdk.disabled=true (default), the OtelServiceFilterFactory component is not created (@ConditionalOnProperty). No risk — the feature is gated.
• No route match: If no route matches, OtelServiceFilterFactory never fires and user.id remains unset. This is existing behavior and out of scope (OTel - Log signal for routed requests to unknown service id #4705 covers this).
• Distributed user ID: Anonymous requests have no distributed identity. The distributedIds method is never called for bypass routes — this is correct.
• Error during anonymous routing: If the downstream service returns an error, auth.error.type/auth.error.message attributes exist in OtelRequestContext and can be set by error handlers (e.g., RoutingConfigurationErrorFilterFactory). No new code needed — the attributes already exist.
• Backward compatibility: Authenticated flows are unchanged. The userId() method still uppercases. The user.id attribute for authenticated users remains UPPERCASE_USER_ID.

Constraints Verification

• Backward compatibility: ✓ — authenticated request signals unchanged
• Feature gated by otel.sdk.disabled: ✓ — no code paths execute when OTel is disabled
• OTel semantic conventions: ✓ — user.id attribute name matches OTel conventions
• Performance: ✓ — adding one string attribute to Attributes builder has negligible overhead
• Acceptance tests use InMemoryLogRecordExporter: ✓ — will follow existing OpenTelemetryTestConfig pattern

[balhar-jakub]

Implementation Decomposition — OTel Log signal for successfully routed anonymous requests (#4704)

Based on the architect's solution design, the implementation is decomposed into 3 sequential engineer tasks (chained, one at a time):

Task 1: [Step 4a] Production code + unit tests (Areas 1, 2, 3)

• OtelRequestContext.java: Add ANONYMOUS_USER_ID = "anonymous" constant + anonymousUserId() method
• OtelServiceFilterFactory.java: Add .anonymousUserId() and .authenticationSuccess() to the apply() builder chain for bypass routes
• Unit tests: Update OtelRequestContextTest and OtelServiceFilterFactoryTest to verify user.id="anonymous" and auth.status="OK"

Task 2: [Step 4b] Acceptance tests (Area 4)

• New acceptance test class using InMemoryLogRecordExporter pattern
• Verify full attributed log signal for anonymous requests (all attributes: method, scheme, path, serviceId, instanceId, authMethod=bypass, responseCode, userId, authStatus)
• Runs after 4a to test against compiled artifact

Task 3: [Step 4c] Documentation (Area 5)

• Update docs-site with OTel log attribute documentation for anonymous/bypass requests
• Document user.id = "anonymous" and full attribute set
• Runs last to ensure docs match actual implementation

Chaining rationale

Areas 1+2+3 are combined because Area 2 depends on Area 1 (needs anonymousUserId() method), and Area 3 tests both. A single engineer task avoids partial-build handoff issues. Area 4 (acceptance tests) requires the compiled artifact from 4a. Area 5 (docs) runs last to match the final implementation.