OTel - Log signal for routed requests to unknown service id

[balhar-jakub]

When APIML could not route a request due to unknown service id or there is no instance available, produce a log signal with attributes:

• time of event (timestamp)
• apiml url path (url.path)
• apiml url scheme (url.scheme)
• apiml http method (http.request.method)
• apiml response code (http.response.status_code)
• apiml error type (error.type)
• apiml error message (error.message)
• user id (user.id)
• service id (service.id)

Note: check the feature for attributes meaning.

Note: Requires implementing a WebFilter because if the route does not match, GlobalFilter is not executed

Acceptance criteria:

• When a request is made to a valid gateway URL but an unknown Service ID (or a service that is down):
  • The log is generated.
  • http.response.status_code records 404 or 503.
  • service.id records the attempted service name (if parsable) or "unknown".
  • error.type:
    • Service not onboarded
    • Service instance not available
• Acceptance tests
• Documentation updated

[balhar-jakub]

Problem Analysis: OTel Log Signal for Routed Requests to Unknown Service ID (Issue #4705)

Refined Problem Statement

When API Mediation Layer (APIML) receives a request that cannot be routed — either because the Service ID in the path is unknown (not registered in Discovery) or because the service has no available instances — the gateway currently returns HTTP 404 or 503 to the client, but does not produce a standardized OpenTelemetry log signal with the error context. This creates an observability gap: operators cannot distinguish between legitimate 404s (resource not found within a valid service) and infrastructure 404s (no such service exists). Similarly, the circumstances of a 503 (no instance available vs. service crashed mid-request) are invisible.

Real-world scenario: A client calls https://gateway:10010/oldservice/api/v1/data. The oldservice was decommissioned but clients still have it in their configuration. The gateway returns a generic 404. Operations teams have no way to detect this misconfiguration short of parsing access logs manually. With OTel logging enabled, each such failure should produce a structured log signal with all relevant attributes for monitoring dashboards and alert systems.

Current Architecture Touchpoints

1. OtelRequestFilter (apiml/src/main/java/org/zowe/apiml/filter/OtelRequestFilter.java)

• Implements both WebFilter and GlobalFilter (order: HIGHEST_PRECEDENCE + 2).
• The WebFilter interface ensures it runs even when no route matches (Spring Cloud Gateway GlobalFilters are bypassed on unmatched routes).
• setDefaults() already sets serviceId to the first path segment when it cannot be determined from the known base-path map (line 123: serviceId = paths.get(1).value()), with the comment: "it is an unknown serviceId."
• filterInternal() issues the OTel log in .doFinally() — runs regardless of success/error.
• Bug note: .doFinally(issue) runs before .then(responseCode) — so the response code is currently NOT included in the emitted log.

2. OtelServiceFilterFactory (gateway-service/.../filters/OtelServiceFilterFactory.java)

• Per-route GatewayFilter — only executes for matched routes.
• Overrides serviceId and instanceId with actual values from the Discovery service.
• This is why unknown services lose instance information: the route never matches, so this filter never runs.

3. OtelRequestContext (apiml-common/.../opentelemetry/OtelRequestContext.java)

• Existing OTel attributes: http.request.method, url.scheme, url.path, service.response_code, service.id, service.instance.id, auth.*, user.id, user.distributed.id.
• Missing attributes (required by this issue): http.response.status_code, error.type, error.message.
• Uses AtomicBoolean.mark() to ensure single execution through dual-interface (WebFilter + GlobalFilter) pattern.

4. GatewayExceptionHandler (gateway-service/.../controllers/GatewayExceptionHandler.java)

• @RestControllerAdvice that catches exceptions and produces error responses.
• Handles NoResourceFoundException → 404, ServiceNotAccessibleException → 503.
• No OTel integration in the exception handler — errors are logged via standard SLF4J, not OTel.

5. RouteLocator (gateway-service/.../service/RouteLocator.java)

• Generates route definitions from Discovery service instances.
• Adds OtelServiceFilterFactory to each route's post-routing filters when otel.sdk.disabled=false.

Scope

In scope:

• Add error.type, error.message, and http.response.status_code attributes to OtelRequestContext
• Modify OtelRequestFilter (or create a new WebFilter) to populate these attributes when a route cannot be matched
• Ensure http.response.status_code is populated with values 404 (unknown service ID) and 503 (service instance not available)
• Ensure service.id records the attempted service name, or "unknown" if not parsable
• Provide acceptance tests demonstrating the OTel log signal for both unknown-service and down-service cases
• Update OTel documentation (if any exists) to document the new attributes

Out of scope:

• Changes to the OTel trace/signal pipeline or exporter configuration
• Modifications to GatewayExceptionHandler beyond ensuring it surfaces needed error info
• Instrumentation of happy-path routing (already handled by OtelServiceFilterFactory)
• Non-OTel logging changes (SLF4J)

Acceptance Criteria (Numbered, Measurable)

1. When a request is made to https://gateway/{unknownServiceId}/... where unknownServiceId is not registered in Discovery:

   • An OTel log signal is emitted via org.zowe.apiml.opentelemetry logger
   • http.response.status_code = 404
   • service.id = the attempted {unknownServiceId} value or "unknown" if not parsable
   • error.type = "Service not onboarded"
   • error.message = descriptive message (e.g., "Service {serviceId} is not registered in the API ML")
   • Other required attributes populated: url.path, url.scheme, http.request.method, user.id

2. When a request targets a known service that has no available instances (all instances down):

   • An OTel log signal is emitted
   • http.response.status_code = 503
   • service.id = the known service ID
   • error.type = "Service instance not available"
   • error.message = descriptive message

3. Acceptance tests for both scenarios exist at the appropriate test layer (unit tests for the WebFilter/context, integration or acceptance tests for end-to-end signal verification)

4. OTel documentation is updated with the new attribute definitions

Design Questions for the Architect

1. New WebFilter vs. extending OtelRequestFilter? The issue states "Requires implementing a WebFilter." Should this be a new, separate WebFilter (e.g., UnknownServiceOtelWebFilter) or should we extend the existing OtelRequestFilter which already runs as a WebFilter and already detects unknown service IDs?

   • Context: OtelRequestFilter already runs on all requests via its WebFilter interface. Adding the error attributes to its existing flow would be simpler. A separate filter adds modularity but risks ordering conflicts.

2. How do we distinguish "unknown service" (404) from "known service, no instances" (503)? The OtelRequestFilter runs before route matching. At filter time, we can determine if the first path segment matches a known base path but we don't know yet whether Discovery has instances. Where should the 503-vs-404 detection happen?

   • Option A: In OtelRequestFilter, set preliminary error state; finalize in an error handler
   • Option B: New dedicated WebFilter that runs AFTER route matching
   • Option C: Enhance GatewayExceptionHandler to populate OTel attributes before issuing the log

3. Fixing the response code timing bug? The current OtelRequestFilter issues the OTel log in .doFinally() BEFORE updating the response code in .then(). Should we fix this ordering as part of this issue, or treat it as a separate concern?

   • Note: If we fix it here, the http.response.status_code attribute will be correct for all requests (not just error cases). If we don't, we need a different mechanism to get the status code into the error-case log.

4. Where should "Service not onboarded" vs. "Service instance not available" detection logic live? Given the reactive architecture and the fact that route matching happens after WebFilter execution:

   • Should we hook into Spring Cloud Gateway's RoutePredicateHandlerMapping failure path?
   • Should we use a reactive error handler (WebExceptionHandler)?
   • Should we modify GatewayExceptionHandler to set OTel attributes before delegating?

5. User ID availability for unknown services? For unknown service IDs, authentication may or may not have run yet. The user.id attribute may be empty or the value from a prior authenticated session. Is it acceptable for user.id to be null/empty for unknown-service-404 cases?

Constraints

• Spring Cloud Gateway reactive architecture: Must use reactive (WebFlux/Reactor) patterns. Blocking calls are not permitted.
• Dual filter interface pattern: Existing filters (OtelRequestFilter, RequestAttributesProvider) implement both WebFilter and GlobalFilter with deduplication. New filter should follow the same pattern if applicable, or be a pure WebFilter if GlobalFilter is never needed.
• Conditional on otel.sdk.disabled=false: All OTel instrumentation must respect this property.
• No ZAAS dependency: ZAAS was removed as a separate component; authentication now flows through gateway. Must not rely on ZAAS-specific attributes for user ID.
• Backward compatibility: New attributes must not break existing OTel consumers (e.g., log parsers, dashboards) that expect the current attribute set.
• Java 17 / Spring Boot 3: Code must use Jakarta namespace, Java 17+ features acceptable.
• EPL-2.0 license header required on all new files.

[balhar-jakub]

Solution Design: OTel Log Signal for Unrouted Requests

Architecture Decision: Extend OtelRequestFilter (No New WebFilter)

Answer to Q1: We extend the existing OtelRequestFilter rather than creating a new WebFilter.

Rationale:

• OtelRequestFilter already implements WebFilter (runs on ALL requests including unrouted ones) and already detects unknown service IDs via setDefaults() line 123.
• A separate WebFilter would introduce ordering complexity and potential conflicts with the existing deduplication mechanism (AtomicBoolean.mark()).
• The filter already owns the OtelRequestContext lifecycle (setDefaults → issue). Adding error attributes here is the most cohesive approach.

How Error Detection Works (Q2 + Q4)

Answer to Q2 & Q4: Detection happens in OtelRequestFilter.filterInternal() using Reactor .doOnError() hooks — NOT in GatewayExceptionHandler.

Why not GatewayExceptionHandler:

• GatewayExceptionHandler.setBodyResponse() creates a new DefaultServerWebExchange, meaning the original exchange attributes (where OtelRequestContext lives) are separated from the response-writing exchange.
• OtelRequestFilter.doFinally() (which calls issue()) fires BEFORE GatewayExceptionHandler finishes writing the response body. By the time the exception handler sets OTel attributes, the log has already been emitted.

Reactive chain design:

filter.apply(exchange)          // downstream chain: route matching → routing → service call
  .doOnError(NoResourceFoundException.class, e -> {
      otelContext.statusCode(404);
      otelContext.errorType("Service not onboarded");
      otelContext.errorMessage("Service  + attemptedService +  is not registered in the API ML");
  })
  .doOnError(ServiceNotAccessibleException.class, e -> {
      otelContext.statusCode(503);
      otelContext.errorType("Service instance not available");
      otelContext.errorMessage(e.getMessage());
  })
  .doFinally(signalType -> {
      // Fix timing bug: set responseCode BEFORE issue()
      // (only for successful completions — error cases already set via doOnError)
      if (signalType == SignalType.ON_COMPLETE) {
          Optional.ofNullable(exchange.getResponse())
              .map(ServerHttpResponse::getStatusCode)
              .map(HttpStatusCode::value)
              .ifPresent(otelContext::responseCode);
      }
      otelContext.issue();      // OTel log signal emitted with ALL attributes
  });

Scenario analysis:

Path	Route Match?	Exception	error.type	http.response.status_code
/unknownservice/api/v1/data	No	NoResourceFoundException	"Service not onboarded"	404
/known-service/api/v1/data (all instances down)	Yes	ServiceNotAccessibleException	"Service instance not available"	503
/apicatalog/valid-path	Yes	None (success)	(not set)	From downstream

Response Code Timing Bug Fix (Q3)

Answer to Q3: Fixed as part of this issue — it is directly required for correctness.

Current bug: .doFinally(issue) runs before .then(responseCode), so service.response_code is always empty in the emitted log.

Fix: Move responseCode setting into the doFinally block BEFORE issue(). For error cases, http.response.status_code is already set via doOnError (new attribute). For success cases, read from exchange.getResponse().getStatusCode() as before, but now BEFORE issue().

Attribute Model Changes (OtelRequestContext.java)

New attributes added:

Attribute	Method	Key	Purpose
HTTP response status code	statusCode(int)	http.response.status_code	The actual HTTP status sent to the client
Error type	errorType(String)	error.type	"Service not onboarded" or "Service instance not available"
Error message	errorMessage(String)	error.message	Human-readable error description

Existing responseCode(int) → service.response_code is preserved for backward compatibility.

User ID for Unknown Services (Q5)

Answer to Q5: user.id will be null/empty for unknown-service-404 cases. This is correct behavior — authentication filters run as GlobalFilters (route-scoped) and never execute on unmatched routes. The absence of user.id is a factual signal: "we could not authenticate this request because no service route was found."

Files to Modify

File	Module	Changes
OtelRequestContext.java	gateway-service	Add statusCode(), errorType(), errorMessage() methods + constants
OtelRequestFilter.java	apiml	Add doOnError hooks, fix responseCode timing, import SignalType
OtelRequestContextTest.java	gateway-service	Add tests for 3 new attribute methods
OtelRequestFilterTest.java	apiml	Add tests: unknown-service→404, service-down→503, verify OTel attributes

Edge Cases & Risks

1. Core service path mismatch: /apicatalog/nonexistent-resource — route matches for apicatalog, downstream returns 404. This is NOT a NoResourceFoundException at the gateway level, so no OTel error attributes are set. Correct behavior: this is a valid service returning a resource-404, not an infrastructure 404.
2. First path segment empty: / (homepage) — handled by existing setDefaults() → serviceId=gateway, route matches.
3. Race condition: Multiple parallel requests to same unknown service — each gets its own OtelRequestContext via exchange.getAttributes(), thread-safe.
4. OTel disabled: @ConditionalOnProperty(value = "otel.sdk.disabled", havingValue = "false") ensures entire filter is skipped when OTel is off.

Independent Areas (for engineer decomposition)

1. Area A — OtelRequestContext attributes: Add statusCode(), errorType(), errorMessage() methods with corresponding OTel attribute keys. Update OtelRequestContextTest.java.
2. Area B — OtelRequestFilter error handling: Add doOnError hooks + fix responseCode timing bug. Update OtelRequestFilterTest.java with error-scenario tests.
3. Area C — Documentation: Document new attributes in otel/README.md.

[balhar-jakub]

Decomposition Plan: #4705 Implementation

The architect's design has been decomposed into three chained engineer tasks. Each runs sequentially to ensure dependencies are respected.

Task Chain (4a → 4b → 4c)

Step	Area	Module	Key Changes
4a	OtelRequestContext attributes	gateway-service	Add statusCode(), errorType(), errorMessage() methods
4b	OtelRequestFilter error handling	apiml	Add doOnError hooks + fix responseCode timing bug
4c	Documentation	otel/	Document new attributes in README

Why Sequential

• 4b depends on 4a producing the new OtelRequestContext methods (statusCode(), errorType(), errorMessage())
• 4c documents the final implementation and should run after code is complete

Files Changed Per Area

Area A (gateway-service):

• OtelRequestContext.java — 3 new attribute methods
• OtelRequestContextTest.java — unit tests

Area B (apiml):

• OtelRequestFilter.java — doOnError hooks + timing fix
• OtelRequestFilterTest.java — error scenario tests

Area C (otel/):

• README.md — attribute documentation