diff --git a/ATTRIBUTIONS-Rust.md b/ATTRIBUTIONS-Rust.md index 2821c2c15..24cc63cc2 100644 --- a/ATTRIBUTIONS-Rust.md +++ b/ATTRIBUTIONS-Rust.md @@ -28687,6 +28687,87 @@ limitations under the License. ``` +## opentelemetry-semantic-conventions - 0.31.0 +**Repository URL**: https://github.com/open-telemetry/opentelemetry-rust/tree/main/opentelemetry-semantic-conventions +**License Type(s)**: Apache-2.0 +### License: https://spdx.org/licenses/Apache-2.0.html +``` +Apache License +Version 2.0, January 2004 +http://www.apache.org/licenses/ + +TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + +1. Definitions. + +"License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. + +"Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. + +"Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. + +"You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. + +"Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. + +"Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. + +"Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). + +"Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. + +"Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." + +"Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. + +2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. + +3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. + +4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: + + (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. + + You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. + +5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. + +6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. + +7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. + +8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. + +9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. + +END OF TERMS AND CONDITIONS + +APPENDIX: How to apply the Apache License to your work. + +To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. + +Copyright [yyyy] [name of copyright owner] + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + +``` + ## opentelemetry_sdk - 0.31.0 **Repository URL**: https://github.com/open-telemetry/opentelemetry-rust/tree/main/opentelemetry-sdk **License Type(s)**: Apache-2.0 diff --git a/Cargo.lock b/Cargo.lock index 2b8f07df3..aff26b13c 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1616,6 +1616,7 @@ dependencies = [ "openinference-semantic-conventions", "opentelemetry", "opentelemetry-otlp", + "opentelemetry-semantic-conventions", "opentelemetry_sdk", "reqwest", "schemars", @@ -2072,6 +2073,12 @@ dependencies = [ "tonic-prost", ] +[[package]] +name = "opentelemetry-semantic-conventions" +version = "0.31.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e62e29dfe041afb8ed2a6c9737ab57db4907285d999ef8ad3a59092a36bdc846" + [[package]] name = "opentelemetry_sdk" version = "0.31.0" diff --git a/Cargo.toml b/Cargo.toml index c77e6e40c..0d6032fcb 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -39,6 +39,7 @@ switchyard-translation = "0.1.0" nemo-relay-ffi = { version = "0.7.0", path = "crates/ffi" } nemo-relay-cli = { version = "0.7.0", path = "crates/cli" } opentelemetry = { version = "0.31", default-features = false } +opentelemetry-semantic-conventions = { version = "0.31", default-features = false, features = ["semconv_experimental"] } opentelemetry_sdk = { version = "0.31", default-features = false } uuid = "=1.18.1" diff --git a/README.md b/README.md index 5a9f5facc..3a87e789f 100644 --- a/README.md +++ b/README.md @@ -29,7 +29,7 @@ shared runtime for scopes, policy, plugins, and lifecycle events. | Instrument app-owned LLM or tool calls | [Quick Start Application](https://docs.nvidia.com/nemo/relay/getting-started/quick-start) | | Use LangChain, LangGraph, Deep Agents, or OpenClaw | [Supported Integrations](https://docs.nvidia.com/nemo/relay/supported-integrations/about) | | Build a framework or provider integration | [Integrate into Frameworks](https://docs.nvidia.com/nemo/relay/integrate-into-frameworks/about) | -| Export ATOF, ATIF, OpenTelemetry, or OpenInference | [Observability Plugin](https://docs.nvidia.com/nemo/relay/configure-plugins/observability/about) | +| Export ATOF, ATIF, OpenTelemetry GenAI, generic OpenTelemetry, or OpenInference | [Observability Plugin](https://docs.nvidia.com/nemo/relay/configure-plugins/observability/about) | | Package reusable middleware or exporters | [Build Plugins](https://docs.nvidia.com/nemo/relay/v0.5.0/build-plugins/about) | | Develop or test this repository from source | [CONTRIBUTING.md](CONTRIBUTING.md) | diff --git a/crates/core/Cargo.toml b/crates/core/Cargo.toml index 1e091e2d8..8928eaf3f 100644 --- a/crates/core/Cargo.toml +++ b/crates/core/Cargo.toml @@ -42,6 +42,7 @@ object-store = [ otel = [ "dep:opentelemetry", "dep:opentelemetry-otlp", + "dep:opentelemetry-semantic-conventions", "dep:opentelemetry_sdk", "dep:reqwest", "dep:tonic", @@ -87,6 +88,7 @@ tokio-stream = { version = "0.1", default-features = false, features = ["sync"] typed-builder = "0.23.2" futures-util = { version = "0.3", optional = true } opentelemetry = { workspace = true, features = ["trace"], optional = true } +opentelemetry-semantic-conventions = { workspace = true, optional = true } opentelemetry_sdk = { workspace = true, features = ["trace"], optional = true } openinference-semantic-conventions = { version = "0.1.1", optional = true } libloading = "0.8" diff --git a/crates/core/README.md b/crates/core/README.md index 83ce2286e..5e743e673 100644 --- a/crates/core/README.md +++ b/crates/core/README.md @@ -47,7 +47,8 @@ Node.js bindings mirror the semantics exposed by this crate. one shared plugin system. - **Built-in observability plugin**: Configure first-party Agent Trajectory Observability Format (ATOF), Agent Trajectory Interchange Format (ATIF), - OpenTelemetry, and OpenInference exporters from the core crate. + generic or GenAI-semantic OpenTelemetry, and OpenInference exporters from the + core crate. - **Codec and typed helpers**: Normalize provider requests and responses for framework integrations. - **Binding source of truth**: Use the runtime semantics mirrored by the diff --git a/crates/core/src/observability/mod.rs b/crates/core/src/observability/mod.rs index b55db2253..91ec5f86f 100644 --- a/crates/core/src/observability/mod.rs +++ b/crates/core/src/observability/mod.rs @@ -44,6 +44,8 @@ pub(crate) mod manual; pub mod openinference; #[cfg(feature = "otel")] pub mod otel; +#[cfg(feature = "otel")] +mod otel_genai; pub mod plugin_component; /// Export representation for point-in-time mark events. diff --git a/crates/core/src/observability/otel.rs b/crates/core/src/observability/otel.rs index 5fab7868a..35417b786 100644 --- a/crates/core/src/observability/otel.rs +++ b/crates/core/src/observability/otel.rs @@ -20,6 +20,7 @@ use std::collections::{HashMap, VecDeque}; use std::sync::{Arc, Mutex}; use std::time::{Duration, SystemTime, UNIX_EPOCH}; +use super::otel_genai; use super::{ MarkProjection, OtlpAttributeMapping, apply_attribute_mappings, attribute_mapping_aliases, attribute_mapping_inputs, default_mark_exclude_names, effective_mark_projection, @@ -42,6 +43,7 @@ use opentelemetry::{Context, KeyValue}; use opentelemetry_otlp::{Protocol, SpanExporter, WithExportConfig, WithHttpConfig}; use opentelemetry_sdk::Resource; use opentelemetry_sdk::trace::{SdkTracer, SdkTracerProvider, Span}; +use serde::{Deserialize, Serialize}; use uuid::Uuid; const COMPLETED_SPAN_CONTEXT_LIMIT: usize = 4096; @@ -91,6 +93,18 @@ pub enum OtlpTransport { Grpc, } +/// Semantic convention projection used for OpenTelemetry spans. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Default, Serialize, Deserialize)] +#[cfg_attr(feature = "schema", derive(schemars::JsonSchema))] +#[serde(rename_all = "snake_case")] +pub enum OpenTelemetrySemanticConvention { + /// Preserve Relay's generic OpenTelemetry span names and attributes. + #[default] + Generic, + /// Project supported scopes using OpenTelemetry GenAI semantic conventions 1.37 or newer. + GenAi, +} + /// Configuration for the OpenTelemetry subscriber. #[derive(Debug, Clone)] pub struct OpenTelemetryConfig { @@ -104,6 +118,8 @@ pub struct OpenTelemetryConfig { mark_projection: MarkProjection, mark_exclude_names: Vec, attribute_mappings: Vec, + semantic_convention: OpenTelemetrySemanticConvention, + capture_content: bool, timeout: Duration, transport: OtlpTransport, } @@ -121,6 +137,8 @@ impl Default for OpenTelemetryConfig { mark_projection: MarkProjection::default(), mark_exclude_names: default_mark_exclude_names(), attribute_mappings: Vec::new(), + semantic_convention: OpenTelemetrySemanticConvention::Generic, + capture_content: false, timeout: Duration::from_secs(3), transport: OtlpTransport::HttpBinary, } @@ -229,6 +247,24 @@ impl OpenTelemetryConfig { self.attribute_mappings = mappings.into_iter().collect(); self } + + /// Selects the span semantic convention projection. + pub fn with_semantic_convention( + mut self, + semantic_convention: OpenTelemetrySemanticConvention, + ) -> Self { + self.semantic_convention = semantic_convention; + self + } + + /// Enables sensitive GenAI content attributes after Relay sanitization. + /// + /// Content capture is disabled by default. This setting only affects the + /// [`OpenTelemetrySemanticConvention::GenAi`] projection. + pub fn with_content_capture(mut self, capture_content: bool) -> Self { + self.capture_content = capture_content; + self + } } /// OpenTelemetry-backed NeMo Relay subscriber. @@ -280,6 +316,8 @@ impl OpenTelemetrySubscriber { config.mark_projection, config.mark_exclude_names, config.attribute_mappings, + config.semantic_convention, + config.capture_content, )) } @@ -294,6 +332,8 @@ impl OpenTelemetrySubscriber { MarkProjection::default(), default_mark_exclude_names(), Vec::new(), + OpenTelemetrySemanticConvention::Generic, + false, ) } @@ -309,6 +349,8 @@ impl OpenTelemetrySubscriber { mark_projection, default_mark_exclude_names(), Vec::new(), + OpenTelemetrySemanticConvention::Generic, + false, ) } @@ -329,6 +371,8 @@ impl OpenTelemetrySubscriber { mark_projection, mark_exclude_names.into_iter().map(Into::into).collect(), Vec::new(), + OpenTelemetrySemanticConvention::Generic, + false, ) } @@ -366,6 +410,29 @@ impl OpenTelemetrySubscriber { options.mark_projection, options.mark_exclude_names, options.attribute_mappings, + OpenTelemetrySemanticConvention::Generic, + false, + )) + } + + /// Builds a subscriber from a tracer provider with generic and semantic projection options. + pub fn from_tracer_provider_with_semantic_convention( + provider: SdkTracerProvider, + instrumentation_scope: impl Into, + options: OpenTelemetrySubscriberOptions, + semantic_convention: OpenTelemetrySemanticConvention, + capture_content: bool, + ) -> Result { + validate_attribute_mappings(&options.attribute_mappings) + .map_err(OpenTelemetryError::InvalidAttributeMappings)?; + Ok(Self::from_tracer_provider_with_scope( + provider, + instrumentation_scope.into(), + options.mark_projection, + options.mark_exclude_names, + options.attribute_mappings, + semantic_convention, + capture_content, )) } @@ -375,6 +442,8 @@ impl OpenTelemetrySubscriber { mark_projection: MarkProjection, mark_exclude_names: Vec, attribute_mappings: Vec, + semantic_convention: OpenTelemetrySemanticConvention, + capture_content: bool, ) -> Self { let processor = Arc::new(Mutex::new( OtelEventProcessor::new_with_mark_projection_and_exclusions_and_mappings( @@ -383,6 +452,8 @@ impl OpenTelemetrySubscriber { mark_projection, mark_exclude_names, attribute_mappings, + semantic_convention, + capture_content, ), )); let processor_for_callback = Arc::clone(&processor); @@ -567,6 +638,8 @@ struct OtelEventProcessor { mark_projection: MarkProjection, mark_exclude_names: Vec, attribute_mappings: Vec, + semantic_convention: OpenTelemetrySemanticConvention, + capture_content: bool, } impl OtelEventProcessor { @@ -602,6 +675,8 @@ impl OtelEventProcessor { mark_projection, mark_exclude_names, Vec::new(), + OpenTelemetrySemanticConvention::Generic, + false, ) } @@ -611,6 +686,8 @@ impl OtelEventProcessor { mark_projection: MarkProjection, mark_exclude_names: Vec, attribute_mappings: Vec, + semantic_convention: OpenTelemetrySemanticConvention, + capture_content: bool, ) -> Self { let tracer = provider.tracer(instrumentation_scope); Self { @@ -622,6 +699,8 @@ impl OtelEventProcessor { mark_projection, mark_exclude_names, attribute_mappings, + semantic_convention, + capture_content, } } @@ -645,19 +724,61 @@ impl OtelEventProcessor { .map_err(|e| OpenTelemetryError::Provider(e.to_string())) } + fn span_name(&self, event: &Event) -> String { + if self.semantic_convention == OpenTelemetrySemanticConvention::GenAi + && otel_genai::supports(event) + { + otel_genai::span_name(event) + } else { + span_name(event) + } + } + + fn span_kind(&self, event: &Event) -> SpanKind { + if self.semantic_convention == OpenTelemetrySemanticConvention::GenAi + && otel_genai::supports(event) + { + otel_genai::span_kind(event) + } else { + span_kind(event) + } + } + + fn start_attributes(&self, event: &Event) -> Vec { + if self.semantic_convention == OpenTelemetrySemanticConvention::GenAi + && otel_genai::supports(event) + { + otel_genai::start_attributes(event, self.capture_content) + } else { + start_attributes(event) + } + } + + fn end_attributes(&self, event: &Event) -> Vec { + if self.semantic_convention == OpenTelemetrySemanticConvention::GenAi + && otel_genai::supports(event) + { + let mut attributes = otel_genai::end_attributes(event, self.capture_content); + push_end_accounting_attributes(&mut attributes, event); + attributes + } else { + end_attributes(event) + } + } + fn process_start(&mut self, event: &Event) { self.remove_completed_span_context(event.uuid()); let parent_context = self.parent_context(event); let is_trace_root = !parent_context.span().span_context().is_valid(); let mut span = self .tracer - .span_builder(span_name(event)) - .with_kind(span_kind(event)) + .span_builder(self.span_name(event)) + .with_kind(self.span_kind(event)) .with_start_time(to_system_time(*event.timestamp())) .with_trace_id(relay_trace_id(event.uuid())) .with_span_id(relay_span_id(event.uuid())) .start_with_context(&self.tracer, &parent_context); - let mut attributes = start_attributes(event); + let mut attributes = self.start_attributes(event); if is_trace_root { push_session_identity_attributes(&mut attributes, event); } @@ -681,7 +802,7 @@ impl OtelEventProcessor { self.record_completed_span_context(event.uuid(), active_span.span_context.clone()); super::set_span_status_from_event_metadata(&mut active_span.span, event); - let mut attributes = end_attributes(event); + let mut attributes = self.end_attributes(event); if !self.attribute_mappings.is_empty() { let mut projected_attributes = active_span.projected_attributes; projected_attributes.extend(attributes.iter().cloned()); @@ -866,6 +987,11 @@ fn end_attributes(event: &Event) -> Vec { push_top_level_json_attributes(&mut attributes, "nemo_relay.end.data", event.data()); push_top_level_json_attributes(&mut attributes, "nemo_relay.end.metadata", event.metadata()); push_top_level_json_attributes(&mut attributes, "nemo_relay.end.output", event.output()); + push_end_accounting_attributes(&mut attributes, event); + attributes +} + +fn push_end_accounting_attributes(attributes: &mut Vec, event: &Event) { if event .category() .is_some_and(|category| category.as_str() == "llm") @@ -877,9 +1003,8 @@ fn end_attributes(event: &Event) -> Vec { if let Some(response) = event.annotated_response() && let Some(summary) = response.optimization_summary.as_ref() { - push_optimization_attributes(&mut attributes, summary); + push_optimization_attributes(attributes, summary); } - attributes } fn push_optimization_attributes( @@ -960,7 +1085,7 @@ fn mark_attributes(event: &Event) -> Vec { attributes } -fn common_attributes(event: &Event) -> Vec { +pub(super) fn common_attributes(event: &Event) -> Vec { let mut attributes = vec![ KeyValue::new("nemo_relay.uuid", event.uuid().to_string()), KeyValue::new( diff --git a/crates/core/src/observability/otel_genai.rs b/crates/core/src/observability/otel_genai.rs new file mode 100644 index 000000000..004150e71 --- /dev/null +++ b/crates/core/src/observability/otel_genai.rs @@ -0,0 +1,759 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! OpenTelemetry GenAI semantic-convention projection. + +use crate::api::event::{Event, EventNormalizationExt}; +use crate::api::scope::ScopeType; +use crate::codec::request::{ContentPart, Message, MessageContent}; +use crate::codec::response::{AnnotatedLlmResponse, FinishReason}; +use crate::json::Json; +use opentelemetry::KeyValue; +use opentelemetry::trace::SpanKind; +use opentelemetry_semantic_conventions::attribute as semconv; +use serde_json::{Map, Value, json}; + +const SEMANTIC_CONVENTION_VERSION: &str = "1.37+"; +const OPERATION_CHAT: &str = "chat"; +const OPERATION_EMBEDDINGS: &str = "embeddings"; +const OPERATION_EXECUTE_TOOL: &str = "execute_tool"; +const OPERATION_GENERATE_CONTENT: &str = "generate_content"; +const OPERATION_INVOKE_AGENT: &str = "invoke_agent"; +const OPERATION_RERANK: &str = "rerank"; +const OPERATION_RETRIEVAL: &str = "retrieval"; +const OPERATION_TEXT_COMPLETION: &str = "text_completion"; + +// OpenTelemetry Rust 0.31 matches Relay's SDK version but predates generated +// constants for these 1.37+ development attributes. Keep the missing keys in +// one projection-local block until the generated crate exposes them without a +// major SDK upgrade. +const GEN_AI_AGENT_VERSION: &str = "gen_ai.agent.version"; +const GEN_AI_INPUT_MESSAGES: &str = "gen_ai.input.messages"; +const GEN_AI_OUTPUT_MESSAGES: &str = "gen_ai.output.messages"; +const GEN_AI_PROVIDER_NAME: &str = "gen_ai.provider.name"; +const GEN_AI_RETRIEVAL_DOCUMENTS: &str = "gen_ai.retrieval.documents"; +const GEN_AI_RETRIEVAL_QUERY_TEXT: &str = "gen_ai.retrieval.query.text"; +const GEN_AI_SYSTEM_INSTRUCTIONS: &str = "gen_ai.system_instructions"; +const GEN_AI_TOOL_CALL_ARGUMENTS: &str = "gen_ai.tool.call.arguments"; +const GEN_AI_TOOL_CALL_RESULT: &str = "gen_ai.tool.call.result"; +const GEN_AI_TOOL_DEFINITIONS: &str = "gen_ai.tool.definitions"; +const GEN_AI_USAGE_CACHE_CREATION_INPUT_TOKENS: &str = "gen_ai.usage.cache_creation.input_tokens"; +const GEN_AI_USAGE_CACHE_READ_INPUT_TOKENS: &str = "gen_ai.usage.cache_read.input_tokens"; + +pub(super) fn supports(event: &Event) -> bool { + matches!( + event.scope_type(), + Some( + ScopeType::Agent + | ScopeType::Llm + | ScopeType::Tool + | ScopeType::Embedder + | ScopeType::Retriever + | ScopeType::Reranker + ) + ) +} + +pub(super) fn span_name(event: &Event) -> String { + let operation = operation_name(event); + let qualifier = match event.scope_type() { + Some(ScopeType::Agent | ScopeType::Tool) => Some(event.name().to_string()), + Some(ScopeType::Retriever) => data_source_id(event), + Some(ScopeType::Llm | ScopeType::Embedder | ScopeType::Reranker) => request_model(event), + _ => None, + }; + qualifier.filter(|value| !value.is_empty()).map_or_else( + || operation.to_string(), + |value| format!("{operation} {value}"), + ) +} + +pub(super) fn span_kind(event: &Event) -> SpanKind { + match event.scope_type() { + Some(ScopeType::Agent | ScopeType::Tool) => SpanKind::Internal, + _ => SpanKind::Client, + } +} + +pub(super) fn start_attributes(event: &Event, capture_content: bool) -> Vec { + let mut attributes = super::otel::common_attributes(event); + super::push_serialized_top_level_attributes( + &mut attributes, + "nemo_relay.handle_attributes", + event.attributes(), + ); + attributes.push(KeyValue::new( + "nemo_relay.otel.semantic_convention", + SEMANTIC_CONVENTION_VERSION, + )); + attributes.push(KeyValue::new( + semconv::GEN_AI_OPERATION_NAME, + operation_name(event), + )); + + push_common_attributes(&mut attributes, event); + match event.scope_type() { + Some(ScopeType::Agent) => push_agent_attributes(&mut attributes, event, capture_content), + Some(ScopeType::Llm) => { + push_llm_request_attributes(&mut attributes, event, capture_content) + } + Some(ScopeType::Tool) => push_tool_attributes(&mut attributes, event, capture_content), + Some(ScopeType::Retriever | ScopeType::Reranker) => { + push_retrieval_attributes(&mut attributes, event, capture_content) + } + Some(ScopeType::Embedder) => push_model_attribute(&mut attributes, event), + _ => {} + } + attributes +} + +pub(super) fn end_attributes(event: &Event, capture_content: bool) -> Vec { + let mut attributes = Vec::new(); + push_error_attributes(&mut attributes, event); + match event.scope_type() { + Some(ScopeType::Llm) => { + push_llm_response_attributes(&mut attributes, event, capture_content) + } + Some(ScopeType::Tool) if capture_content => { + if let Some(result) = wrapped_content_or_value(event.output(), &["result", "output"]) { + attributes.push(KeyValue::new(GEN_AI_TOOL_CALL_RESULT, json_string(result))); + } + } + Some(ScopeType::Embedder | ScopeType::Reranker) => { + push_non_llm_response_attributes(&mut attributes, event); + if capture_content && event.scope_type() == Some(ScopeType::Reranker) { + push_retrieval_content(&mut attributes, event); + } + } + Some(ScopeType::Retriever) if capture_content => { + push_retrieval_content(&mut attributes, event); + } + _ => {} + } + attributes +} + +fn push_non_llm_response_attributes(attributes: &mut Vec, event: &Event) { + if let Some(value) = scalar_string( + event, + &[semconv::GEN_AI_RESPONSE_MODEL, "response_model", "model"], + ) { + attributes.push(KeyValue::new(semconv::GEN_AI_RESPONSE_MODEL, value)); + } + if let Some(value) = scalar_i64( + event, + &[ + semconv::GEN_AI_USAGE_INPUT_TOKENS, + "input_tokens", + "prompt_tokens", + ], + ) { + attributes.push(KeyValue::new(semconv::GEN_AI_USAGE_INPUT_TOKENS, value)); + } + if let Some(value) = scalar_i64( + event, + &[ + semconv::GEN_AI_USAGE_OUTPUT_TOKENS, + "output_tokens", + "completion_tokens", + ], + ) { + attributes.push(KeyValue::new(semconv::GEN_AI_USAGE_OUTPUT_TOKENS, value)); + } +} + +fn operation_name(event: &Event) -> &'static str { + match event.scope_type() { + Some(ScopeType::Agent) => OPERATION_INVOKE_AGENT, + Some(ScopeType::Tool) => OPERATION_EXECUTE_TOOL, + Some(ScopeType::Embedder) => OPERATION_EMBEDDINGS, + Some(ScopeType::Retriever) => OPERATION_RETRIEVAL, + Some(ScopeType::Reranker) => OPERATION_RERANK, + Some(ScopeType::Llm) => llm_operation_name(event), + _ => OPERATION_CHAT, + } +} + +fn llm_operation_name(event: &Event) -> &'static str { + let name = event.name().to_ascii_lowercase(); + if name.contains("generate_content") || name.contains("generatecontent") { + OPERATION_GENERATE_CONTENT + } else if name.contains("completion") && !name.contains("chat") { + OPERATION_TEXT_COMPLETION + } else { + OPERATION_CHAT + } +} + +fn push_common_attributes(attributes: &mut Vec, event: &Event) { + if let Some(provider) = provider_name(event) { + attributes.push(KeyValue::new(GEN_AI_PROVIDER_NAME, provider)); + } + if let Some(conversation_id) = scalar_string( + event, + &[ + semconv::GEN_AI_CONVERSATION_ID, + "conversation_id", + "session_id", + "thread_id", + ], + ) { + attributes.push(KeyValue::new( + semconv::GEN_AI_CONVERSATION_ID, + conversation_id, + )); + } + if let Some(address) = scalar_string(event, &[semconv::SERVER_ADDRESS, "server_address"]) { + attributes.push(KeyValue::new(semconv::SERVER_ADDRESS, address)); + } + if let Some(port) = scalar_i64(event, &[semconv::SERVER_PORT, "server_port"]) { + attributes.push(KeyValue::new(semconv::SERVER_PORT, port)); + } +} + +fn push_agent_attributes(attributes: &mut Vec, event: &Event, capture_content: bool) { + attributes.push(KeyValue::new( + semconv::GEN_AI_AGENT_NAME, + event.name().to_string(), + )); + for (key, candidates) in [ + ( + semconv::GEN_AI_AGENT_ID, + &["gen_ai.agent.id", "agent_id"][..], + ), + ( + GEN_AI_AGENT_VERSION, + &["gen_ai.agent.version", "agent_version"][..], + ), + ] { + if let Some(value) = scalar_string(event, candidates) { + attributes.push(KeyValue::new(key, value)); + } + } + if capture_content + && let Some(value) = + scalar_string(event, &["gen_ai.agent.description", "agent_description"]) + { + attributes.push(KeyValue::new(semconv::GEN_AI_AGENT_DESCRIPTION, value)); + } + push_model_attribute(attributes, event); + if capture_content { + push_tool_definitions(attributes, event); + } +} + +fn push_model_attribute(attributes: &mut Vec, event: &Event) { + if let Some(model) = request_model(event) { + attributes.push(KeyValue::new(semconv::GEN_AI_REQUEST_MODEL, model)); + } +} + +fn push_llm_request_attributes( + attributes: &mut Vec, + event: &Event, + capture_content: bool, +) { + let Some(request) = event.normalized_llm_request() else { + push_model_attribute(attributes, event); + return; + }; + let request = request.as_ref(); + if let Some(model) = request + .model + .clone() + .or_else(|| event.model_name().map(ToOwned::to_owned)) + { + attributes.push(KeyValue::new(semconv::GEN_AI_REQUEST_MODEL, model)); + } + if let Some(params) = request.params.as_ref() { + if let Some(value) = params.temperature { + attributes.push(KeyValue::new(semconv::GEN_AI_REQUEST_TEMPERATURE, value)); + } + if request.max_output_tokens.is_none() + && let Some(value) = params.max_tokens.and_then(to_i64) + { + attributes.push(KeyValue::new(semconv::GEN_AI_REQUEST_MAX_TOKENS, value)); + } + if let Some(value) = params.top_p { + attributes.push(KeyValue::new(semconv::GEN_AI_REQUEST_TOP_P, value)); + } + if let Some(value) = params.stop.as_ref() { + attributes.push(KeyValue::new( + semconv::GEN_AI_REQUEST_STOP_SEQUENCES, + string_array(value.iter().cloned()), + )); + } + } + if let Some(value) = request.max_output_tokens.and_then(to_i64) { + attributes.push(KeyValue::new(semconv::GEN_AI_REQUEST_MAX_TOKENS, value)); + } + if !capture_content { + return; + } + let (instructions, messages) = request_messages(&request.messages); + if !instructions.is_empty() { + attributes.push(KeyValue::new( + GEN_AI_SYSTEM_INSTRUCTIONS, + json_string(&Value::Array(instructions)), + )); + } + if !messages.is_empty() { + attributes.push(KeyValue::new( + GEN_AI_INPUT_MESSAGES, + json_string(&Value::Array(messages)), + )); + } + push_tool_definitions(attributes, event); +} + +fn push_llm_response_attributes( + attributes: &mut Vec, + event: &Event, + capture_content: bool, +) { + let Some(response) = event.normalized_llm_response() else { + return; + }; + let response = response.as_ref(); + if let Some(value) = response.id.as_ref() { + attributes.push(KeyValue::new(semconv::GEN_AI_RESPONSE_ID, value.clone())); + } + if let Some(value) = response.model.as_ref() { + attributes.push(KeyValue::new(semconv::GEN_AI_RESPONSE_MODEL, value.clone())); + } + if let Some(value) = response.finish_reason.as_ref() { + attributes.push(KeyValue::new( + semconv::GEN_AI_RESPONSE_FINISH_REASONS, + string_array([finish_reason(value).to_string()]), + )); + } + if let Some(usage) = response.usage.as_ref() { + if let Some(value) = usage.prompt_tokens.and_then(to_i64) { + attributes.push(KeyValue::new(semconv::GEN_AI_USAGE_INPUT_TOKENS, value)); + } + if let Some(value) = usage.completion_tokens.and_then(to_i64) { + attributes.push(KeyValue::new(semconv::GEN_AI_USAGE_OUTPUT_TOKENS, value)); + } + if let Some(value) = usage.cache_read_tokens.and_then(to_i64) { + attributes.push(KeyValue::new(GEN_AI_USAGE_CACHE_READ_INPUT_TOKENS, value)); + } + if let Some(value) = usage.cache_write_tokens.and_then(to_i64) { + attributes.push(KeyValue::new( + GEN_AI_USAGE_CACHE_CREATION_INPUT_TOKENS, + value, + )); + } + } + if capture_content && let Some(message) = output_message(response) { + attributes.push(KeyValue::new( + GEN_AI_OUTPUT_MESSAGES, + json_string(&json!([message])), + )); + } +} + +fn push_tool_attributes(attributes: &mut Vec, event: &Event, capture_content: bool) { + attributes.push(KeyValue::new( + semconv::GEN_AI_TOOL_NAME, + event.name().to_string(), + )); + if let Some(value) = scalar_string(event, &[semconv::GEN_AI_TOOL_TYPE, "tool_type"]) { + attributes.push(KeyValue::new(semconv::GEN_AI_TOOL_TYPE, value)); + } + if let Some(value) = event + .tool_call_id() + .map(ToOwned::to_owned) + .or_else(|| scalar_string(event, &[semconv::GEN_AI_TOOL_CALL_ID, "tool_call_id"])) + { + attributes.push(KeyValue::new(semconv::GEN_AI_TOOL_CALL_ID, value)); + } + if capture_content { + if let Some(value) = scalar_string( + event, + &[ + semconv::GEN_AI_TOOL_DESCRIPTION, + "tool_description", + "description", + ], + ) { + attributes.push(KeyValue::new(semconv::GEN_AI_TOOL_DESCRIPTION, value)); + } + if let Some(arguments) = wrapped_content_or_value(event.input(), &["arguments", "input"]) { + attributes.push(KeyValue::new( + GEN_AI_TOOL_CALL_ARGUMENTS, + json_string(arguments), + )); + } + } +} + +fn push_retrieval_attributes(attributes: &mut Vec, event: &Event, capture_content: bool) { + if let Some(value) = data_source_id(event) { + attributes.push(KeyValue::new(semconv::GEN_AI_DATA_SOURCE_ID, value)); + } + push_model_attribute(attributes, event); + if let Some(value) = scalar_f64(event, &[semconv::GEN_AI_REQUEST_TOP_K, "top_k"]) { + attributes.push(KeyValue::new(semconv::GEN_AI_REQUEST_TOP_K, value)); + } + if capture_content { + push_retrieval_content(attributes, event); + } +} + +fn push_retrieval_content(attributes: &mut Vec, event: &Event) { + if let Some(value) = content_value(event.data(), &["query", "query_text"]) { + let query = value + .as_str() + .map(ToOwned::to_owned) + .unwrap_or_else(|| json_string(value)); + attributes.push(KeyValue::new(GEN_AI_RETRIEVAL_QUERY_TEXT, query)); + } + if let Some(value) = content_value(event.data(), &["documents", "results"]) { + attributes.push(KeyValue::new( + GEN_AI_RETRIEVAL_DOCUMENTS, + json_string(value), + )); + } +} + +fn push_tool_definitions(attributes: &mut Vec, event: &Event) { + let definitions = if let Some(request) = event.normalized_llm_request() + && let Some(tools) = request.tools.as_ref() + { + tools + .iter() + .map(|tool| { + let mut definition = Map::new(); + definition.insert("type".to_string(), Value::String(tool.tool_type.clone())); + definition.insert( + "name".to_string(), + Value::String(tool.function.name.clone()), + ); + if let Some(value) = tool.function.description.as_ref() { + definition.insert("description".to_string(), Value::String(value.clone())); + } + if let Some(value) = tool.function.parameters.as_ref() { + definition.insert("parameters".to_string(), value.clone()); + } + Value::Object(definition) + }) + .collect::>() + } else if let Some(value) = content_value(event.data(), &["tool_definitions", "tools"]) + && let Some(values) = value.as_array() + { + values.clone() + } else { + return; + }; + attributes.push(KeyValue::new( + GEN_AI_TOOL_DEFINITIONS, + json_string(&Value::Array(definitions)), + )); +} + +fn request_messages(messages: &[Message]) -> (Vec, Vec) { + let mut instructions = Vec::new(); + let mut input = Vec::new(); + for message in messages { + match message { + Message::System { content, .. } => instructions.extend(content_parts(content)), + Message::User { content, name } => input.push(message_value( + "user", + name.as_deref(), + content_parts(content), + )), + Message::Assistant { + content, + tool_calls, + name, + } => { + let mut parts = content.as_ref().map(content_parts).unwrap_or_default(); + if let Some(tool_calls) = tool_calls { + parts.extend(tool_calls.iter().map(|call| { + let arguments = serde_json::from_str(&call.function.arguments) + .unwrap_or_else(|_| Value::String(call.function.arguments.clone())); + json!({ + "type": "tool_call", + "id": call.id, + "name": call.function.name, + "arguments": arguments, + }) + })); + } + input.push(message_value("assistant", name.as_deref(), parts)); + } + Message::Tool { + content, + tool_call_id, + } => input.push(json!({ + "role": "tool", + "parts": [{ + "type": "tool_call_response", + "id": tool_call_id, + "result": content_result(content), + }], + })), + } + } + (instructions, input) +} + +fn output_message(response: &AnnotatedLlmResponse) -> Option { + let mut parts = response + .message + .as_ref() + .map(content_parts) + .unwrap_or_default(); + if let Some(tool_calls) = response.tool_calls.as_ref() { + parts.extend(tool_calls.iter().map(|call| { + json!({ + "type": "tool_call", + "id": call.id, + "name": call.name, + "arguments": call.arguments, + }) + })); + } + if parts.is_empty() { + return None; + } + let mut message = Map::new(); + message.insert("role".to_string(), Value::String("assistant".to_string())); + message.insert("parts".to_string(), Value::Array(parts)); + if let Some(reason) = response.finish_reason.as_ref() { + message.insert( + "finish_reason".to_string(), + Value::String(finish_reason(reason).to_string()), + ); + } + Some(Value::Object(message)) +} + +fn message_value(role: &str, name: Option<&str>, parts: Vec) -> Value { + let mut message = Map::new(); + message.insert("role".to_string(), Value::String(role.to_string())); + message.insert("parts".to_string(), Value::Array(parts)); + if let Some(name) = name { + message.insert("name".to_string(), Value::String(name.to_string())); + } + Value::Object(message) +} + +fn content_parts(content: &MessageContent) -> Vec { + match content { + MessageContent::Text(content) => vec![json!({"type": "text", "content": content})], + MessageContent::Parts(parts) => parts + .iter() + .map(|part| match part { + ContentPart::Text { text } => json!({"type": "text", "content": text}), + ContentPart::ImageUrl { image_url } => { + json!({"type": "uri", "uri": image_url.url}) + } + }) + .collect(), + } +} + +fn content_result(content: &MessageContent) -> Value { + match content { + MessageContent::Text(value) => Value::String(value.clone()), + MessageContent::Parts(_) => Value::Array(content_parts(content)), + } +} + +fn finish_reason(reason: &FinishReason) -> &str { + match reason { + FinishReason::Complete => "stop", + FinishReason::Length => "length", + FinishReason::ToolUse => "tool_calls", + FinishReason::ContentFilter => "content_filter", + FinishReason::Unknown(value) => value, + } +} + +fn request_model(event: &Event) -> Option { + event + .normalized_llm_request() + .and_then(|request| request.as_ref().model.clone()) + .or_else(|| event.model_name().map(ToOwned::to_owned)) + .or_else(|| { + scalar_string( + event, + &[semconv::GEN_AI_REQUEST_MODEL, "model", "model_name"], + ) + }) +} + +fn provider_name(event: &Event) -> Option { + scalar_string(event, &[GEN_AI_PROVIDER_NAME, "provider_name", "provider"]).or_else(|| { + let name = event.name().to_ascii_lowercase(); + [ + ("azure", "azure.ai.openai"), + ("anthropic", "anthropic"), + ("bedrock", "aws.bedrock"), + ("cohere", "cohere"), + ("deepseek", "deepseek"), + ("gemini", "gcp.gemini"), + ("vertex", "gcp.vertex_ai"), + ("groq", "groq"), + ("mistral", "mistral_ai"), + ("openai", "openai"), + ("perplexity", "perplexity"), + ] + .into_iter() + .find_map(|(needle, provider)| name.contains(needle).then(|| provider.to_string())) + }) +} + +fn data_source_id(event: &Event) -> Option { + scalar_string( + event, + &[ + semconv::GEN_AI_DATA_SOURCE_ID, + "data_source_id", + "index_name", + "collection_name", + ], + ) +} + +fn push_error_attributes(attributes: &mut Vec, event: &Event) { + let is_error = event + .metadata() + .and_then(Value::as_object) + .and_then(|metadata| metadata.get("otel.status_code")) + .and_then(Value::as_str) + .is_some_and(|status| status.eq_ignore_ascii_case("ERROR")); + if !is_error { + return; + } + let error_type = scalar_string(event, &[semconv::ERROR_TYPE, "error_type"]) + .unwrap_or_else(|| "_OTHER".to_string()); + attributes.push(KeyValue::new(semconv::ERROR_TYPE, error_type)); +} + +fn scalar_string(event: &Event, keys: &[&str]) -> Option { + if let Some(profile) = event.category_profile() { + for key in keys { + if let Some(value) = profile.extra.get(*key) { + if let Some(value) = value.as_str() { + return Some(value.to_string()); + } + if value.is_number() || value.is_boolean() { + return Some(value.to_string()); + } + } + } + } + for object in event_objects(event) { + for key in keys { + if let Some(value) = object_value(object, key) { + if let Some(value) = value.as_str() { + return Some(value.to_string()); + } + if value.is_number() || value.is_boolean() { + return Some(value.to_string()); + } + } + } + } + None +} + +fn scalar_i64(event: &Event, keys: &[&str]) -> Option { + if let Some(profile) = event.category_profile() { + for key in keys { + if let Some(value) = profile.extra.get(*key) { + if let Some(value) = value.as_i64() { + return Some(value); + } + if let Some(value) = value.as_u64().and_then(to_i64) { + return Some(value); + } + } + } + } + for object in event_objects(event) { + for key in keys { + if let Some(value) = object_value(object, key) { + if let Some(value) = value.as_i64() { + return Some(value); + } + if let Some(value) = value.as_u64().and_then(to_i64) { + return Some(value); + } + } + } + } + None +} + +fn scalar_f64(event: &Event, keys: &[&str]) -> Option { + if let Some(profile) = event.category_profile() { + for key in keys { + if let Some(value) = profile.extra.get(*key).and_then(Value::as_f64) { + return Some(value); + } + } + } + for object in event_objects(event) { + for key in keys { + if let Some(value) = object_value(object, key).and_then(Value::as_f64) { + return Some(value); + } + } + } + None +} + +fn object_value<'a>(object: &'a Map, key: &str) -> Option<&'a Json> { + object.get(key).or_else(|| { + ["usage", "request", "response"] + .into_iter() + .filter_map(|container| object.get(container).and_then(Value::as_object)) + .find_map(|nested| nested.get(key)) + }) +} + +fn event_objects(event: &Event) -> Vec<&Map> { + let mut objects = Vec::new(); + if let Some(value) = event.metadata().and_then(Value::as_object) { + objects.push(value); + } + if let Some(value) = event.data().and_then(Value::as_object) { + objects.push(value); + } + objects +} + +fn content_value<'a>(value: Option<&'a Json>, keys: &[&str]) -> Option<&'a Json> { + let value = value?; + if let Some(object) = value.as_object() { + for key in keys { + if let Some(value) = object.get(*key) { + return Some(value); + } + } + return None; + } + Some(value) +} + +fn wrapped_content_or_value<'a>(value: Option<&'a Json>, keys: &[&str]) -> Option<&'a Json> { + let value = value?; + content_value(Some(value), keys).or(Some(value)) +} + +fn json_string(value: &Value) -> String { + serde_json::to_string(value).expect("serializing a JSON value cannot fail") +} + +fn string_array(values: impl IntoIterator) -> opentelemetry::Value { + opentelemetry::Value::Array(opentelemetry::Array::String( + values.into_iter().map(Into::into).collect(), + )) +} + +fn to_i64(value: u64) -> Option { + i64::try_from(value).ok() +} diff --git a/crates/core/src/observability/plugin_component.rs b/crates/core/src/observability/plugin_component.rs index 8a52fb6e6..ed133e455 100644 --- a/crates/core/src/observability/plugin_component.rs +++ b/crates/core/src/observability/plugin_component.rs @@ -54,7 +54,8 @@ use crate::observability::openinference::{ }; #[cfg(feature = "otel")] use crate::observability::otel::{ - OpenTelemetryConfig as CoreOpenTelemetryConfig, OpenTelemetrySubscriber, + OpenTelemetryConfig as CoreOpenTelemetryConfig, OpenTelemetrySemanticConvention, + OpenTelemetrySubscriber, }; use crate::observability::{ MarkProjection, OtlpAttributeMapping, default_mark_exclude_names, validate_attribute_mappings, @@ -402,6 +403,23 @@ pub struct OtlpSectionConfig { /// Typed projected attributes copied to aliases. #[serde(default, skip_serializing_if = "Vec::is_empty")] pub attribute_mappings: Vec, + /// OpenTelemetry semantic projection: `generic` or `gen_ai`. + /// + /// The `openinference` section only supports `generic`. + #[serde( + default = "default_semantic_convention", + skip_serializing_if = "is_default_semantic_convention" + )] + #[cfg_attr( + feature = "schema", + schemars(schema_with = "semantic_convention_schema") + )] + pub semantic_convention: String, + /// Whether the OpenTelemetry GenAI projection includes sanitized content. + /// + /// The `openinference` section does not use this setting. + #[serde(default, skip_serializing_if = "is_false")] + pub capture_content: bool, /// OTLP transport: `http_binary` or `grpc`. #[serde(default = "default_otlp_transport")] #[cfg_attr(feature = "schema", schemars(schema_with = "otlp_transport_schema"))] @@ -439,6 +457,8 @@ impl Default for OtlpSectionConfig { mark_projection: MarkProjection::default(), mark_exclude_names: default_mark_exclude_names(), attribute_mappings: Vec::new(), + semantic_convention: default_semantic_convention(), + capture_content: false, transport: default_otlp_transport(), endpoint: None, headers: HashMap::new(), @@ -607,6 +627,8 @@ crate::editor_config! { mark_projection => { label: "mark_projection", kind: Enum, values: ["inherit", "event", "tool"] }, mark_exclude_names => { label: "mark_exclude_names", kind: Json }, attribute_mappings => { label: "attribute_mappings", kind: List, list: &OTLP_ATTRIBUTE_MAPPING_LIST_ITEM }, + semantic_convention => { label: "semantic_convention", kind: Enum, values: ["generic", "gen_ai"] }, + capture_content => { label: "capture_content", kind: Boolean }, transport => { label: "transport", kind: Enum, values: ["http_binary", "grpc"] }, endpoint => { label: "endpoint", kind: String, optional: true }, headers => { label: "headers", kind: StringMap }, @@ -702,6 +724,13 @@ fn mark_projection_schema( string_enum_schema(generator, &["inherit", "event", "tool"], Some("inherit")) } +#[cfg(feature = "schema")] +fn semantic_convention_schema( + generator: &mut schemars::r#gen::SchemaGenerator, +) -> schemars::schema::Schema { + string_enum_schema(generator, &["generic", "gen_ai"], Some("generic")) +} + #[cfg(feature = "schema")] fn string_enum_schema( generator: &mut schemars::r#gen::SchemaGenerator, @@ -1630,7 +1659,17 @@ fn build_otel_config(section: OtlpSectionConfig) -> PluginResult OpenTelemetrySemanticConvention::Generic, + "gen_ai" => OpenTelemetrySemanticConvention::GenAi, + other => { + return Err(PluginError::InvalidConfig(format!( + "OpenTelemetry semantic_convention must be 'generic' or 'gen_ai', got {other:?}" + ))); + } + }) + .with_content_capture(section.capture_content); if let Some(endpoint) = section.endpoint { config = config.with_endpoint(endpoint); @@ -1803,6 +1842,8 @@ fn validate_observability_section_fields( "mark_projection", "mark_exclude_names", "attribute_mappings", + "semantic_convention", + "capture_content", "transport", "endpoint", "headers", @@ -1824,6 +1865,8 @@ fn validate_observability_section_fields( "mark_projection", "mark_exclude_names", "attribute_mappings", + "semantic_convention", + "capture_content", "transport", "endpoint", "headers", @@ -1979,6 +2022,27 @@ fn validate_openinference_section( section: &OtlpSectionConfig, ) { validate_otlp_values(diagnostics, policy, "openinference", section); + if section.semantic_convention != "generic" { + push_policy_diag( + diagnostics, + policy.unsupported_value, + "observability.unsupported_value", + Some("openinference".to_string()), + Some("semantic_convention".to_string()), + "openinference semantic_convention must be 'generic'".to_string(), + ); + } + if section.capture_content { + push_policy_diag( + diagnostics, + policy.unsupported_value, + "observability.unsupported_value", + Some("openinference".to_string()), + Some("capture_content".to_string()), + "openinference does not support capture_content; use its existing content policy" + .to_string(), + ); + } validate_openinference_feature_support(diagnostics, policy, section); } @@ -2604,6 +2668,16 @@ fn validate_otlp_values( format!("{section_name} transport must be 'http_binary' or 'grpc'"), ); } + if !matches!(section.semantic_convention.as_str(), "generic" | "gen_ai") { + push_policy_diag( + diagnostics, + policy.unsupported_value, + "observability.unsupported_value", + Some(section_name.to_string()), + Some("semantic_convention".to_string()), + format!("{section_name} semantic_convention must be 'generic' or 'gen_ai'"), + ); + } if let Err(message) = validate_attribute_mappings(§ion.attribute_mappings) { push_policy_diag( diagnostics, @@ -2703,6 +2777,18 @@ fn default_otlp_transport() -> String { "http_binary".to_string() } +fn default_semantic_convention() -> String { + "generic".to_string() +} + +fn is_default_semantic_convention(value: &str) -> bool { + value == "generic" +} + +fn is_false(value: &bool) -> bool { + !value +} + fn default_service_name() -> String { "nemo-relay".to_string() } diff --git a/crates/core/tests/unit/observability/otel_tests.rs b/crates/core/tests/unit/observability/otel_tests.rs index cec581fca..11f920b75 100644 --- a/crates/core/tests/unit/observability/otel_tests.rs +++ b/crates/core/tests/unit/observability/otel_tests.rs @@ -14,6 +14,7 @@ use crate::api::scope::ScopeType; use crate::api::scope::{event, pop_scope, push_scope}; use crate::api::tool::ToolAttributes; use crate::codec::model_pricing::pricing_test_mutex; +use crate::codec::request::AnnotatedLlmRequest; use crate::codec::response::{ AnnotatedLlmResponse, CostEstimate, CostSource, PricingCatalog, PricingResolver, Usage, reset_active_pricing_resolver, set_active_pricing_resolver, @@ -25,6 +26,7 @@ use serde_json::json; use std::collections::HashMap; use std::io::{Read, Write}; use std::net::TcpListener; +use std::sync::Arc; use std::sync::mpsc; use std::thread; use uuid::Uuid; @@ -413,6 +415,41 @@ fn make_scope_event_with_profile( )) } +fn make_scope_event_with_metadata_and_profile( + scope_category: ScopeCategory, + uuid: Uuid, + name: &str, + scope_type: ScopeType, + data: Option, + metadata: Option, + category_profile: Option, +) -> Event { + Event::Scope(ScopeEvent::new( + BaseEvent::builder() + .uuid(uuid) + .name(name) + .data_opt(data) + .metadata_opt(metadata) + .build(), + scope_category, + Vec::new(), + EventCategory::from(scope_type), + category_profile, + )) +} + +fn make_genai_processor(provider: SdkTracerProvider, capture_content: bool) -> OtelEventProcessor { + OtelEventProcessor::new_with_mark_projection_and_exclusions_and_mappings( + provider, + "test-genai".to_string(), + MarkProjection::default(), + default_mark_exclude_names(), + Vec::new(), + OpenTelemetrySemanticConvention::GenAi, + capture_content, + ) +} + fn make_scope_event_with_attributes( scope_category: ScopeCategory, uuid: Uuid, @@ -549,6 +586,8 @@ fn config_defaults_and_builder_overrides_are_applied() { .with_instrumentation_scope("demo-scope") .with_mark_projection(MarkProjection::Tool) .with_mark_exclude_names(["notification", "hook_mark"]) + .with_semantic_convention(OpenTelemetrySemanticConvention::GenAi) + .with_content_capture(true) .with_timeout(Duration::from_millis(1250)); assert_eq!(config.transport, OtlpTransport::HttpBinary); @@ -570,6 +609,11 @@ fn config_defaults_and_builder_overrides_are_applied() { assert_eq!(config.instrumentation_scope, "demo-scope"); assert_eq!(config.mark_projection, MarkProjection::Tool); assert_eq!(config.mark_exclude_names, vec!["notification", "hook_mark"]); + assert_eq!( + config.semantic_convention, + OpenTelemetrySemanticConvention::GenAi + ); + assert!(config.capture_content); assert_eq!(config.timeout, Duration::from_millis(1250)); let defaults = OpenTelemetryConfig::default(); @@ -578,6 +622,11 @@ fn config_defaults_and_builder_overrides_are_applied() { assert_eq!(defaults.instrumentation_scope, "nemo-relay-otel"); assert_eq!(defaults.mark_projection, MarkProjection::Inherit); assert_eq!(defaults.mark_exclude_names, vec!["llm.chunk"]); + assert_eq!( + defaults.semantic_convention, + OpenTelemetrySemanticConvention::Generic + ); + assert!(!defaults.capture_content); assert_eq!(defaults.timeout, Duration::from_secs(3)); assert!(defaults.headers.is_empty()); assert!(defaults.resource_attributes.is_empty()); @@ -1668,6 +1717,403 @@ fn llm_end_emits_cost_only_no_token_or_gen_ai_attributes() { .all(|k| !k.starts_with("llm.token") && !k.starts_with("gen_ai")), "no token attributes expected on the LLM span: {keys:?}" ); + assert_eq!(spans[0].name.as_ref(), "other"); + assert_eq!(spans[0].span_kind, SpanKind::Client); +} + +#[test] +fn genai_llm_projection_emits_datadog_compatible_attributes() { + let request: AnnotatedLlmRequest = serde_json::from_value(json!({ + "messages": [ + {"role": "system", "content": "Answer precisely."}, + {"role": "user", "content": "What is 2+2?"} + ], + "model": "gpt-4.1-mini", + "params": { + "temperature": 0.2, + "max_tokens": 64, + "top_p": 0.9, + "stop": ["DONE"] + }, + "tools": [{ + "type": "function", + "function": { + "name": "calculator", + "description": "Evaluate an expression", + "parameters": {"type": "object"} + } + }] + })) + .unwrap(); + let response: AnnotatedLlmResponse = serde_json::from_value(json!({ + "id": "chatcmpl-123", + "model": "gpt-4.1-mini-2026-01-01", + "message": "Four.", + "finish_reason": "complete", + "usage": { + "prompt_tokens": 17, + "completion_tokens": 3, + "total_tokens": 20, + "cache_read_tokens": 5 + } + })) + .unwrap(); + let uuid = Uuid::now_v7(); + let start = make_scope_event_with_metadata_and_profile( + ScopeCategory::Start, + uuid, + "openai.chat", + ScopeType::Llm, + None, + Some(json!({ + "conversation_id": "conversation-42", + "server.address": "api.openai.com" + })), + Some( + CategoryProfile::builder() + .model_name("gpt-4.1-mini") + .annotated_request(Arc::new(request)) + .build(), + ), + ); + let end = make_scope_event_with_metadata_and_profile( + ScopeCategory::End, + uuid, + "openai.chat", + ScopeType::Llm, + None, + None, + Some( + CategoryProfile::builder() + .model_name("gpt-4.1-mini") + .annotated_response(Arc::new(response)) + .build(), + ), + ); + + let (provider, exporter) = make_provider(); + let mut processor = make_genai_processor(provider, true); + processor.process(&start); + processor.process(&end); + processor.force_flush().unwrap(); + + let spans = exporter.get_finished_spans().unwrap(); + assert_eq!(spans.len(), 1); + let span = &spans[0]; + assert_eq!(span.name.as_ref(), "chat gpt-4.1-mini"); + assert_eq!(span.span_kind, SpanKind::Client); + let attributes = attr_map(&span.attributes); + assert_eq!(attributes["gen_ai.operation.name"], "chat"); + assert_eq!(attributes["gen_ai.provider.name"], "openai"); + assert_eq!(attributes["gen_ai.request.model"], "gpt-4.1-mini"); + assert_eq!( + attributes["gen_ai.response.model"], + "gpt-4.1-mini-2026-01-01" + ); + assert_eq!(attributes["gen_ai.response.id"], "chatcmpl-123"); + assert_eq!(attributes["gen_ai.usage.input_tokens"], "17"); + assert_eq!(attributes["gen_ai.usage.output_tokens"], "3"); + assert_eq!(attributes["gen_ai.usage.cache_read.input_tokens"], "5"); + assert_eq!(attributes["gen_ai.conversation.id"], "conversation-42"); + assert_eq!(attributes["server.address"], "api.openai.com"); + assert!(attributes["gen_ai.system_instructions"].contains("Answer precisely.")); + assert!(attributes["gen_ai.input.messages"].contains("What is 2+2?")); + assert!(attributes["gen_ai.output.messages"].contains("Four.")); + assert!(attributes["gen_ai.tool.definitions"].contains("calculator")); + assert_eq!(attributes["nemo_relay.scope_type"], "llm"); + assert_eq!(attributes["nemo_relay.model_name"], "gpt-4.1-mini"); + assert_eq!(attributes["nemo_relay.otel.semantic_convention"], "1.37+"); +} + +#[test] +fn genai_content_capture_is_disabled_by_default_and_uses_sanitized_values() { + let request: AnnotatedLlmRequest = serde_json::from_value(json!({ + "messages": [ + {"role": "system", "content": "[REDACTED]"}, + {"role": "user", "content": "[REDACTED]"} + ], + "model": "safe-model", + "tools": [{ + "type": "function", + "function": { + "name": "safe-tool", + "description": "[REDACTED]", + "parameters": {"secret": "[REDACTED]"} + } + }] + })) + .unwrap(); + let response: AnnotatedLlmResponse = serde_json::from_value(json!({ + "model": "safe-model", + "message": "[REDACTED]", + "finish_reason": "complete", + "usage": {"prompt_tokens": 2, "completion_tokens": 1} + })) + .unwrap(); + let uuid = Uuid::now_v7(); + let start = make_scope_event_with_metadata_and_profile( + ScopeCategory::Start, + uuid, + "openai.chat", + ScopeType::Llm, + Some(json!({"raw_secret": "must-not-export"})), + None, + Some( + CategoryProfile::builder() + .annotated_request(Arc::new(request.clone())) + .build(), + ), + ); + let end = make_scope_event_with_metadata_and_profile( + ScopeCategory::End, + uuid, + "openai.chat", + ScopeType::Llm, + Some(json!({"raw_secret": "must-not-export"})), + None, + Some( + CategoryProfile::builder() + .annotated_response(Arc::new(response.clone())) + .build(), + ), + ); + + let (provider, exporter) = make_provider(); + let mut processor = make_genai_processor(provider, false); + processor.process(&start); + processor.process(&end); + processor.force_flush().unwrap(); + let spans = exporter.get_finished_spans().unwrap(); + let attributes = attr_map(&spans[0].attributes); + assert!(!attributes.contains_key("gen_ai.system_instructions")); + assert!(!attributes.contains_key("gen_ai.input.messages")); + assert!(!attributes.contains_key("gen_ai.output.messages")); + assert!(!attributes.contains_key("gen_ai.tool.definitions")); + assert!( + attributes + .values() + .all(|value| !value.contains("must-not-export")) + ); + assert_eq!(attributes["gen_ai.usage.input_tokens"], "2"); + + let tool_uuid = Uuid::now_v7(); + processor.process(&make_scope_event_with_metadata_and_profile( + ScopeCategory::Start, + tool_uuid, + "private-tool", + ScopeType::Tool, + Some(json!({"arguments": {"secret": "must-not-export"}})), + Some(json!({"tool_description": "must-not-export"})), + None, + )); + processor.process(&make_scope_event_with_metadata_and_profile( + ScopeCategory::End, + tool_uuid, + "private-tool", + ScopeType::Tool, + Some(json!({"result": "must-not-export"})), + None, + None, + )); + processor.force_flush().unwrap(); + let spans = exporter.get_finished_spans().unwrap(); + let tool = spans + .iter() + .find(|span| span.name.as_ref() == "execute_tool private-tool") + .unwrap(); + let tool_attributes = attr_map(&tool.attributes); + assert!(!tool_attributes.contains_key("gen_ai.tool.description")); + assert!(!tool_attributes.contains_key("gen_ai.tool.call.arguments")); + assert!(!tool_attributes.contains_key("gen_ai.tool.call.result")); + assert!( + tool_attributes + .values() + .all(|value| !value.contains("must-not-export")) + ); + + let (provider, exporter) = make_provider(); + let mut processor = make_genai_processor(provider, true); + processor.process(&start); + processor.process(&end); + processor.force_flush().unwrap(); + let spans = exporter.get_finished_spans().unwrap(); + let attributes = attr_map(&spans[0].attributes); + assert!(attributes["gen_ai.input.messages"].contains("[REDACTED]")); + assert!(attributes["gen_ai.output.messages"].contains("[REDACTED]")); + assert!( + attributes + .values() + .all(|value| !value.contains("must-not-export")) + ); +} + +#[test] +fn genai_llm_span_names_follow_the_inference_operation() { + let (provider, _) = make_provider(); + let processor = make_genai_processor(provider, false); + for (name, operation) in [ + ("openai.chat", "chat"), + ("gemini.generateContent", "generate_content"), + ("openai.completions", "text_completion"), + ] { + let event = make_scope_event_with_metadata_and_profile( + ScopeCategory::Start, + Uuid::now_v7(), + name, + ScopeType::Llm, + None, + Some(json!({"model": "test-model"})), + None, + ); + assert_eq!( + processor.span_name(&event), + format!("{operation} test-model") + ); + let attributes = attr_map(&processor.start_attributes(&event)); + assert_eq!(attributes["gen_ai.operation.name"], operation); + } +} + +#[test] +fn genai_projection_names_supported_scope_types_and_records_errors() { + let cases = [ + ( + ScopeType::Agent, + "planner", + "invoke_agent planner", + SpanKind::Internal, + ), + ( + ScopeType::Tool, + "search", + "execute_tool search", + SpanKind::Internal, + ), + ( + ScopeType::Embedder, + "embed", + "embeddings text-embed-3", + SpanKind::Client, + ), + ( + ScopeType::Retriever, + "search-index", + "retrieval products", + SpanKind::Client, + ), + ( + ScopeType::Reranker, + "rerank", + "rerank cohere-rerank", + SpanKind::Client, + ), + ]; + let (provider, exporter) = make_provider(); + let mut processor = make_genai_processor(provider, true); + for (scope_type, name, _, _) in &cases { + let uuid = Uuid::now_v7(); + let metadata = match scope_type { + ScopeType::Embedder => json!({"model": "text-embed-3"}), + ScopeType::Retriever => json!({"data_source_id": "products", "top_k": 7.0}), + ScopeType::Reranker => json!({"model": "cohere-rerank"}), + ScopeType::Tool => json!({"tool_call_id": "call-7", "tool_type": "function"}), + _ => json!({"agent_id": "agent-7", "agent_version": "1.0"}), + }; + processor.process(&make_scope_event_with_metadata_and_profile( + ScopeCategory::Start, + uuid, + name, + *scope_type, + Some(if *scope_type == ScopeType::Tool { + json!({"x": 1}) + } else { + json!({"query": "private query"}) + }), + Some(metadata), + None, + )); + let end_data = match scope_type { + ScopeType::Tool => json!({"value": 2}), + ScopeType::Embedder => json!({ + "model": "text-embed-3-2026", + "usage": {"input_tokens": 9} + }), + ScopeType::Retriever | ScopeType::Reranker => { + json!({"results": [{"id": "doc-1"}]}) + } + _ => json!({"status": "failed"}), + }; + processor.process(&make_scope_event_with_metadata_and_profile( + ScopeCategory::End, + uuid, + name, + *scope_type, + Some(end_data), + Some(json!({"otel.status_code": "ERROR", "error.type": "timeout"})), + None, + )); + } + processor.force_flush().unwrap(); + + let spans = exporter.get_finished_spans().unwrap(); + assert_eq!(spans.len(), cases.len()); + for (_, _, expected_name, expected_kind) in cases { + let span = spans + .iter() + .find(|span| span.name.as_ref() == expected_name) + .unwrap_or_else(|| panic!("missing span {expected_name}")); + assert_eq!(span.span_kind, expected_kind); + let attributes = attr_map(&span.attributes); + assert_eq!(attributes["error.type"], "timeout"); + assert!(attributes.contains_key("gen_ai.operation.name")); + assert!(attributes.contains_key("nemo_relay.uuid")); + assert!(matches!( + span.status, + opentelemetry::trace::Status::Error { .. } + )); + } + let tool = spans + .iter() + .find(|span| span.name.as_ref() == "execute_tool search") + .unwrap(); + let tool_attributes = attr_map(&tool.attributes); + assert_eq!(tool_attributes["gen_ai.tool.name"], "search"); + assert_eq!(tool_attributes["gen_ai.tool.call.id"], "call-7"); + assert!(tool_attributes["gen_ai.tool.call.arguments"].contains("\"x\":1")); + assert!(tool_attributes["gen_ai.tool.call.result"].contains("\"value\":2")); + + let agent = spans + .iter() + .find(|span| span.name.as_ref() == "invoke_agent planner") + .unwrap(); + let agent_attributes = attr_map(&agent.attributes); + assert_eq!(agent_attributes["gen_ai.agent.name"], "planner"); + assert_eq!(agent_attributes["gen_ai.agent.id"], "agent-7"); + assert_eq!(agent_attributes["gen_ai.agent.version"], "1.0"); + + let embedding = spans + .iter() + .find(|span| span.name.as_ref() == "embeddings text-embed-3") + .unwrap(); + let embedding_attributes = attr_map(&embedding.attributes); + assert_eq!( + embedding_attributes["gen_ai.response.model"], + "text-embed-3-2026" + ); + assert_eq!(embedding_attributes["gen_ai.usage.input_tokens"], "9"); + + let retrieval = spans + .iter() + .find(|span| span.name.as_ref() == "retrieval products") + .unwrap(); + let retrieval_attributes = attr_map(&retrieval.attributes); + assert_eq!(retrieval_attributes["gen_ai.data_source.id"], "products"); + assert!(retrieval.attributes.iter().any(|attribute| { + attribute.key.as_str() == "gen_ai.request.top_k" + && matches!(&attribute.value, opentelemetry::Value::F64(value) if *value == 7.0) + })); + assert!(retrieval_attributes["gen_ai.retrieval.query.text"].contains("private query")); + assert!(retrieval_attributes["gen_ai.retrieval.documents"].contains("doc-1")); } #[test] diff --git a/crates/core/tests/unit/observability/plugin_component_tests.rs b/crates/core/tests/unit/observability/plugin_component_tests.rs index 128d473aa..40291dc06 100644 --- a/crates/core/tests/unit/observability/plugin_component_tests.rs +++ b/crates/core/tests/unit/observability/plugin_component_tests.rs @@ -212,6 +212,14 @@ fn editor_schema_tracks_observability_config_types() { .expect("openinference editor schema"); let headers = otlp.field("headers").expect("headers field"); assert_eq!(headers.kind, EditorFieldKind::StringMap); + let semantic_convention = otlp + .field("semantic_convention") + .expect("semantic convention field"); + assert_eq!(semantic_convention.kind, EditorFieldKind::Enum); + let capture_content = otlp + .field("capture_content") + .expect("content capture field"); + assert_eq!(capture_content.kind, EditorFieldKind::Boolean); let attribute_mappings = otlp .field("attribute_mappings") @@ -368,6 +376,8 @@ fn default_config_and_component_conversion_cover_public_shape() { assert_eq!(otlp.mark_projection, MarkProjection::Inherit); assert_eq!(otlp.mark_exclude_names, vec!["llm.chunk"]); assert!(otlp.attribute_mappings.is_empty()); + assert_eq!(otlp.semantic_convention, "generic"); + assert!(!otlp.capture_content); assert_eq!(otlp.transport, "http_binary"); assert_eq!(otlp.service_name, "nemo-relay"); assert_eq!(otlp.timeout_millis, 3_000); @@ -397,6 +407,14 @@ fn mark_projection_parses_for_otlp_and_rejects_unknown_values() { .unwrap(); assert_eq!(mappings.attribute_mappings.len(), 1); + let genai: OtlpSectionConfig = serde_json::from_value(json!({ + "semantic_convention": "gen_ai", + "capture_content": true + })) + .unwrap(); + assert_eq!(genai.semantic_convention, "gen_ai"); + assert!(genai.capture_content); + let otlp: OtlpSectionConfig = serde_json::from_value(json!({ "mark_projection": "tool" })) @@ -435,6 +453,29 @@ fn mark_projection_parses_for_otlp_and_rejects_unknown_values() { == "observability.invalid_plugin_config" && diagnostic.message.contains("unknown variant `span`"))); + let report = validate_plugin_config(&plugin_config(json!({ + "opentelemetry": {"semantic_convention": "future"} + }))); + assert!(report.has_errors()); + assert!(report.diagnostics.iter().any(|diagnostic| { + diagnostic.field.as_deref() == Some("semantic_convention") + && diagnostic.message.contains("generic") + && diagnostic.message.contains("gen_ai") + })); + + let report = validate_plugin_config(&plugin_config(json!({ + "openinference": {"semantic_convention": "gen_ai", "capture_content": true} + }))); + assert!(report.has_errors()); + assert!(report.diagnostics.iter().any(|diagnostic| { + diagnostic.component.as_deref() == Some("openinference") + && diagnostic.field.as_deref() == Some("semantic_convention") + })); + assert!(report.diagnostics.iter().any(|diagnostic| { + diagnostic.component.as_deref() == Some("openinference") + && diagnostic.field.as_deref() == Some("capture_content") + })); + let report = validate_plugin_config(&plugin_config(json!({ "atif": {"mark_projection": "tool"} }))); @@ -508,6 +549,8 @@ fn schema_contains_every_supported_observability_option() { "mark_projection", "mark_exclude_names", "attribute_mappings", + "semantic_convention", + "capture_content", "tool_definitions", "extra", "filename_template", @@ -545,6 +588,11 @@ fn schema_contains_every_supported_observability_option() { "mark_projection", &["inherit", "event", "tool"] )); + assert!(schema_property_has_enum( + &schema, + "semantic_convention", + &["generic", "gen_ai"] + )); assert!(schema_property_has_default( &schema, "mode", diff --git a/crates/ffi/nemo_relay.h b/crates/ffi/nemo_relay.h index af178bac7..27f9407c3 100644 --- a/crates/ffi/nemo_relay.h +++ b/crates/ffi/nemo_relay.h @@ -1361,6 +1361,29 @@ NemoRelayStatus nemo_relay_otel_subscriber_create_with_attribute_mappings(const const char *attribute_mappings_json, struct FfiOpenTelemetrySubscriber **out); +/** + * Creates a new OpenTelemetry subscriber with projection and content-capture options. + * + * `semantic_convention` accepts `generic` (the default) or `gen_ai`. + * `capture_content` only affects the `gen_ai` projection and operates on sanitized events. + * + * # Safety + * Any non-null C strings must be valid and `out` must be non-null. + */ +NemoRelayStatus nemo_relay_otel_subscriber_create_with_options(const char *transport, + const char *endpoint, + const char *headers_json, + const char *resource_attributes_json, + const char *service_name, + const char *service_namespace, + const char *service_version, + const char *instrumentation_scope, + uint64_t timeout_millis, + const char *attribute_mappings_json, + const char *semantic_convention, + bool capture_content, + struct FfiOpenTelemetrySubscriber **out); + /** * Registers the OpenTelemetry subscriber as an event subscriber. * diff --git a/crates/ffi/src/api/observability.rs b/crates/ffi/src/api/observability.rs index 989b4f127..499544c4e 100644 --- a/crates/ffi/src/api/observability.rs +++ b/crates/ffi/src/api/observability.rs @@ -13,6 +13,8 @@ type AtofExporterConfig = nemo_relay::observability::atof::AtofExporterConfig; type AtofExporterError = nemo_relay::observability::atof::AtofExporterError; type AtofExporterMode = nemo_relay::observability::atof::AtofExporterMode; type OpenTelemetryConfig = nemo_relay::observability::otel::OpenTelemetryConfig; +type OpenTelemetrySemanticConvention = + nemo_relay::observability::otel::OpenTelemetrySemanticConvention; type OpenTelemetrySubscriber = nemo_relay::observability::otel::OpenTelemetrySubscriber; type OpenInferenceConfig = nemo_relay::observability::openinference::OpenInferenceConfig; type OpenInferenceSubscriber = nemo_relay::observability::openinference::OpenInferenceSubscriber; @@ -730,6 +732,48 @@ pub unsafe extern "C" fn nemo_relay_otel_subscriber_create_with_attribute_mappin timeout_millis: u64, attribute_mappings_json: *const c_char, out: *mut *mut FfiOpenTelemetrySubscriber, +) -> NemoRelayStatus { + unsafe { + nemo_relay_otel_subscriber_create_with_options( + transport, + endpoint, + headers_json, + resource_attributes_json, + service_name, + service_namespace, + service_version, + instrumentation_scope, + timeout_millis, + attribute_mappings_json, + std::ptr::null(), + false, + out, + ) + } +} + +/// Creates a new OpenTelemetry subscriber with projection and content-capture options. +/// +/// `semantic_convention` accepts `generic` (the default) or `gen_ai`. +/// `capture_content` only affects the `gen_ai` projection and operates on sanitized events. +/// +/// # Safety +/// Any non-null C strings must be valid and `out` must be non-null. +#[unsafe(no_mangle)] +pub unsafe extern "C" fn nemo_relay_otel_subscriber_create_with_options( + transport: *const c_char, + endpoint: *const c_char, + headers_json: *const c_char, + resource_attributes_json: *const c_char, + service_name: *const c_char, + service_namespace: *const c_char, + service_version: *const c_char, + instrumentation_scope: *const c_char, + timeout_millis: u64, + attribute_mappings_json: *const c_char, + semantic_convention: *const c_char, + capture_content: bool, + out: *mut *mut FfiOpenTelemetrySubscriber, ) -> NemoRelayStatus { clear_last_error(); if let Err(status) = required_out_ptr(out) { @@ -749,6 +793,20 @@ pub unsafe extern "C" fn nemo_relay_otel_subscriber_create_with_attribute_mappin Ok(config) => config, Err(status) => return status, }; + let semantic_convention = match parse_string_or_default(semantic_convention, "generic") { + Ok(value) if value == "generic" => OpenTelemetrySemanticConvention::Generic, + Ok(value) if value == "gen_ai" => OpenTelemetrySemanticConvention::GenAi, + Ok(value) => { + set_last_error(&format!( + "semantic_convention must be 'generic' or 'gen_ai', got {value:?}" + )); + return NemoRelayStatus::InvalidArg; + } + Err(status) => return status, + }; + config = config + .with_semantic_convention(semantic_convention) + .with_content_capture(capture_content); config = match apply_optional_string(config, endpoint, OpenTelemetryConfig::with_endpoint) { Ok(config) => config, Err(status) => return status, diff --git a/crates/ffi/tests/unit/api/plugin_tests.rs b/crates/ffi/tests/unit/api/plugin_tests.rs index ebf6a4217..c3068646e 100644 --- a/crates/ffi/tests/unit/api/plugin_tests.rs +++ b/crates/ffi/tests/unit/api/plugin_tests.rs @@ -1510,6 +1510,46 @@ fn test_ffi_typed_attribute_mapping_constructors_validate_and_accept_mappings() NemoRelayStatus::InvalidArg ); + let gen_ai = cstring("gen_ai"); + let invalid_semantic_convention = cstring("future"); + assert_eq!( + nemo_relay_otel_subscriber_create_with_options( + ptr::null(), + ptr::null(), + ptr::null(), + ptr::null(), + ptr::null(), + ptr::null(), + ptr::null(), + ptr::null(), + 0, + valid.as_ptr(), + gen_ai.as_ptr(), + true, + &mut otel, + ), + NemoRelayStatus::Ok + ); + nemo_relay_otel_subscriber_free(otel); + assert_eq!( + nemo_relay_otel_subscriber_create_with_options( + ptr::null(), + ptr::null(), + ptr::null(), + ptr::null(), + ptr::null(), + ptr::null(), + ptr::null(), + ptr::null(), + 0, + ptr::null(), + invalid_semantic_convention.as_ptr(), + false, + &mut otel, + ), + NemoRelayStatus::InvalidArg + ); + let mut openinference = ptr::null_mut(); assert_eq!( nemo_relay_openinference_subscriber_create_with_attribute_mappings( diff --git a/crates/node/README.md b/crates/node/README.md index c3820a132..87bc1768d 100644 --- a/crates/node/README.md +++ b/crates/node/README.md @@ -49,7 +49,8 @@ The Node.js package provides the following capabilities: plus mark and scope event sanitizers for `data`, `categoryProfile`, and `metadata`. - **Observability exporters**: Subscriber and exporter support for common - runtime telemetry flows. + runtime telemetry flows, including generic OpenTelemetry and the opt-in + OpenTelemetry GenAI 1.37 or newer projection. - **Additional entry points**: `nemo-relay-node/typed`, `nemo-relay-node/plugin`, `nemo-relay-node/adaptive`, and `nemo-relay-node/observability`. diff --git a/crates/node/observability.d.ts b/crates/node/observability.d.ts index caddfb316..d89a2c2c2 100644 --- a/crates/node/observability.d.ts +++ b/crates/node/observability.d.ts @@ -71,6 +71,10 @@ export interface OtlpConfig { mark_projection?: 'inherit' | 'event' | 'tool'; mark_exclude_names?: string[]; attribute_mappings?: OtlpAttributeMapping[]; + /** OpenTelemetry-only semantic projection. OpenInference supports `generic`. */ + semantic_convention?: 'generic' | 'gen_ai'; + /** OpenTelemetry-only capture of sanitized GenAI content. */ + capture_content?: boolean; transport?: 'http_binary' | 'grpc' | string; endpoint?: string; headers?: Record; diff --git a/crates/node/observability.js b/crates/node/observability.js index a43d69e9c..8c95217b6 100644 --- a/crates/node/observability.js +++ b/crates/node/observability.js @@ -59,6 +59,8 @@ function otlpConfig(config = {}) { mark_projection: 'inherit', mark_exclude_names: ['llm.chunk'], attribute_mappings: [], + semantic_convention: 'generic', + capture_content: false, transport: 'http_binary', headers: {}, resource_attributes: {}, diff --git a/crates/node/src/api/mod.rs b/crates/node/src/api/mod.rs index 1a7cc8a5e..96c618e59 100644 --- a/crates/node/src/api/mod.rs +++ b/crates/node/src/api/mod.rs @@ -175,6 +175,16 @@ fn build_otel_config( .instrumentation_scope .unwrap_or_else(|| "nemo-relay-otel".to_string()); let timeout_millis = options.timeout_millis.unwrap_or(3_000); + let semantic_convention = match options.semantic_convention.as_deref().unwrap_or("generic") { + "generic" => nemo_relay::observability::otel::OpenTelemetrySemanticConvention::Generic, + "gen_ai" => nemo_relay::observability::otel::OpenTelemetrySemanticConvention::GenAi, + other => { + return Err(napi::Error::from_reason(format!( + "semanticConvention must be 'generic' or 'gen_ai', got {other:?}", + ))); + } + }; + let capture_content = options.capture_content.unwrap_or(false); let mut config = match transport.as_str() { "http_binary" => { @@ -188,7 +198,9 @@ fn build_otel_config( } } .with_instrumentation_scope(instrumentation_scope) - .with_timeout(std::time::Duration::from_millis(timeout_millis.into())); + .with_timeout(std::time::Duration::from_millis(timeout_millis.into())) + .with_semantic_convention(semantic_convention) + .with_content_capture(capture_content); if let Some(endpoint) = options.endpoint { config = config.with_endpoint(endpoint); @@ -3598,6 +3610,10 @@ pub struct OpenTelemetryConfig { pub timeout_millis: Option, /// Typed projected attributes copied to aliases. pub attribute_mappings: Option>, + /// `"generic"` (default) or the OpenTelemetry GenAI `"gen_ai"` projection. + pub semantic_convention: Option, + /// Emit sanitized message and tool content in GenAI attributes. Defaults to `false`. + pub capture_content: Option, } /// Typed projected attribute copy configuration. diff --git a/crates/node/tests/observability_plugin_tests.mjs b/crates/node/tests/observability_plugin_tests.mjs index 99b8d9f84..92a470c87 100644 --- a/crates/node/tests/observability_plugin_tests.mjs +++ b/crates/node/tests/observability_plugin_tests.mjs @@ -32,6 +32,8 @@ describe('observability plugin helpers', () => { mark_projection: 'inherit', mark_exclude_names: ['llm.chunk'], attribute_mappings: [], + semantic_convention: 'generic', + capture_content: false, transport: 'http_binary', headers: {}, resource_attributes: {}, diff --git a/crates/node/tests/otel_tests.mjs b/crates/node/tests/otel_tests.mjs index 9144178b6..57fa253d9 100644 --- a/crates/node/tests/otel_tests.mjs +++ b/crates/node/tests/otel_tests.mjs @@ -33,6 +33,8 @@ describe('OpenTelemetrySubscriber', () => { 'deployment.environment': 'test', }, attributeMappings: [{ key: 'nemo_relay.start.data.tenant', alias: 'tenant.id' }], + semanticConvention: 'gen_ai', + captureContent: true, }); const name = uniqueId('node_otel'); @@ -44,6 +46,13 @@ describe('OpenTelemetrySubscriber', () => { }); it('rejects invalid config values', () => { + assert.throws( + () => + new OpenTelemetrySubscriber({ + semanticConvention: 'future', + }), + /semanticConvention must be/i, + ); assert.throws( () => new OpenTelemetrySubscriber({ @@ -114,4 +123,30 @@ describe('OpenTelemetrySubscriber', () => { await collector.close(); } }); + + it('exports the opt-in GenAI projection through the existing OTLP transport', async () => { + const collector = await startCollector(); + const subscriber = new OpenTelemetrySubscriber({ + endpoint: collector.endpoint, + serviceName: 'node-genai-agent', + semanticConvention: 'gen_ai', + }); + + const name = uniqueId('node_otel_genai'); + subscriber.register(name); + try { + const scope = pushScope('planner', ScopeType.Agent, null, null, null, null); + popScope(scope); + subscriber.forceFlush(); + + const request = await collector.nextRequest(); + assertOtlpStringAttribute(request.body, 'gen_ai.operation.name', 'invoke_agent'); + assertOtlpStringAttribute(request.body, 'gen_ai.agent.name', 'planner'); + assertOtlpStringAttribute(request.body, 'nemo_relay.otel.semantic_convention', '1.37+'); + } finally { + subscriber.deregister(name); + subscriber.shutdown(); + await collector.close(); + } + }); }); diff --git a/crates/python/src/py_types/observability.rs b/crates/python/src/py_types/observability.rs index 85a24bb23..4fa1e9836 100644 --- a/crates/python/src/py_types/observability.rs +++ b/crates/python/src/py_types/observability.rs @@ -440,6 +440,10 @@ pub struct PyOpenTelemetryConfig { pub(crate) instrumentation_scope: String, #[pyo3(get, set)] pub(crate) timeout_millis: u64, + #[pyo3(get, set)] + pub(crate) semantic_convention: String, + #[pyo3(get, set)] + pub(crate) capture_content: bool, pub(crate) headers: HashMap, pub(crate) resource_attributes: HashMap, pub(crate) attribute_mappings: Vec, @@ -463,7 +467,17 @@ impl PyOpenTelemetryConfig { } } .with_instrumentation_scope(self.instrumentation_scope.clone()) - .with_timeout(Duration::from_millis(self.timeout_millis)); + .with_timeout(Duration::from_millis(self.timeout_millis)) + .with_semantic_convention(match self.semantic_convention.as_str() { + "generic" => nemo_relay::observability::otel::OpenTelemetrySemanticConvention::Generic, + "gen_ai" => nemo_relay::observability::otel::OpenTelemetrySemanticConvention::GenAi, + other => { + return Err(pyo3::exceptions::PyValueError::new_err(format!( + "semantic_convention must be 'generic' or 'gen_ai', got {other:?}" + ))); + } + }) + .with_content_capture(self.capture_content); if let Some(endpoint) = &self.endpoint { config = config.with_endpoint(endpoint.clone()); @@ -497,6 +511,8 @@ impl PyOpenTelemetryConfig { service_version: None, instrumentation_scope: "nemo-relay-otel".to_string(), timeout_millis: 3_000, + semantic_convention: "generic".to_string(), + capture_content: false, headers: HashMap::new(), resource_attributes: HashMap::new(), attribute_mappings: Vec::new(), @@ -558,8 +574,8 @@ impl PyOpenTelemetryConfig { pub(crate) fn __repr__(&self) -> String { format!( - "", - self.transport, self.endpoint + "", + self.transport, self.semantic_convention, self.endpoint ) } } diff --git a/docs/about-nemo-relay/concepts/subscribers.mdx b/docs/about-nemo-relay/concepts/subscribers.mdx index b794a7966..0b8ea2646 100644 --- a/docs/about-nemo-relay/concepts/subscribers.mdx +++ b/docs/about-nemo-relay/concepts/subscribers.mdx @@ -171,7 +171,8 @@ event per line. ### OpenTelemetry Subscriber The OpenTelemetry subscriber maps runtime events into OTLP traces for tracing -backends. +backends. It preserves generic Relay semantics by default and can opt in to an +OpenTelemetry GenAI 1.37 or newer projection for supported AI scopes. ### OpenInference Subscriber diff --git a/docs/configure-plugins/observability/about.mdx b/docs/configure-plugins/observability/about.mdx index 85ef99e54..818e2ac15 100644 --- a/docs/configure-plugins/observability/about.mdx +++ b/docs/configure-plugins/observability/about.mdx @@ -90,7 +90,7 @@ Choose the exporter based on the downstream system: |---|---| | Raw canonical event stream | [Agent Trajectory Observability Format (ATOF)](/configure-plugins/observability/atof) | | Offline analysis, replay, or evaluation trajectories | [Agent Trajectory Interchange Format (ATIF)](/configure-plugins/observability/atif) | -| Generic OTLP traces | [OpenTelemetry](/configure-plugins/observability/opentelemetry) | +| Generic OTLP traces or opt-in OpenTelemetry GenAI 1.37+ spans | [OpenTelemetry](/configure-plugins/observability/opentelemetry) | | OpenInference-oriented agent and LLM spans | [OpenInference](/configure-plugins/observability/openinference) | Start with in-process event inspection before exporting externally. Add sanitize @@ -140,6 +140,6 @@ IDs and exposes the same logical values under `extra.nemo_relay`. - [Agent Trajectory Interchange Format (ATIF)](/configure-plugins/observability/atif) covers trajectory export for top-level Agent scopes and supported coding-agent turn scopes. -- [OpenTelemetry](/configure-plugins/observability/opentelemetry) covers generic OTLP trace export. +- [OpenTelemetry](/configure-plugins/observability/opentelemetry) covers generic OTLP trace export and the opt-in GenAI semantic projection. - [OpenInference](/configure-plugins/observability/openinference) covers OpenInference-oriented OTLP trace export. diff --git a/docs/configure-plugins/observability/configuration.mdx b/docs/configure-plugins/observability/configuration.mdx index 90ba3167f..e57adce8b 100644 --- a/docs/configure-plugins/observability/configuration.mdx +++ b/docs/configure-plugins/observability/configuration.mdx @@ -92,6 +92,8 @@ filename_template = "trajectory-{session_id}.json" [components.config.opentelemetry] enabled = true +semantic_convention = "gen_ai" +capture_content = false transport = "http_binary" endpoint = "http://localhost:4318/v1/traces" service_name = "nemo-relay" diff --git a/docs/configure-plugins/observability/opentelemetry.mdx b/docs/configure-plugins/observability/opentelemetry.mdx index 6c434c9cd..1a00aa369 100644 --- a/docs/configure-plugins/observability/opentelemetry.mdx +++ b/docs/configure-plugins/observability/opentelemetry.mdx @@ -8,7 +8,9 @@ SPDX-License-Identifier: Apache-2.0 */} Use the `opentelemetry` section when you want NeMo Relay lifecycle events -exported as generic OpenTelemetry Protocol (OTLP) trace spans. +exported as OpenTelemetry Protocol (OTLP) trace spans. Generic Relay semantics +remain the default. Set `semantic_convention = "gen_ai"` to opt in to the +OpenTelemetry GenAI 1.37 or newer projection. OpenTelemetry export is a good fit when your tracing backend already expects OTLP spans and you want NeMo Relay scopes, tool calls, LLM calls, and marks to @@ -30,6 +32,8 @@ version = 2 [components.config.opentelemetry] enabled = true +semantic_convention = "gen_ai" +capture_content = false transport = "http_binary" endpoint = "http://localhost:4318/v1/traces" service_name = "agent-service" @@ -58,6 +62,8 @@ The following table describes OpenTelemetry exporter settings: | `mark_projection` | `inherit` | `inherit` uses exporter-native handling; `event` forces span events; `tool` emits zero-duration spans, parented as children when context is available, for trace-tree visibility. | | `mark_exclude_names` | `["llm.chunk"]` | Mark names excluded from `tool` projection; excluded marks retain exporter-native handling. Metadata `hook_event_name` aliases are also matched. | | `attribute_mappings` | `[]` | Copies each fully qualified projected `key` to its `alias` without changing the OTLP type. Set both values to nonblank strings, and use each alias only once. | +| `semantic_convention` | `generic` | `generic` preserves the existing Relay span projection. `gen_ai` uses OpenTelemetry GenAI 1.37 or newer semantics for supported scopes. | +| `capture_content` | `false` | When `semantic_convention = "gen_ai"`, includes sanitized messages, instructions, tool definitions and payloads, retrieval queries, and retrieved content. Leave disabled for metadata-only spans. | | `transport` | `http_binary` | `http_binary` or `grpc`. | | `endpoint` | Exporter default | OTLP endpoint. | | `headers` | `{}` | String-to-string exporter headers. | @@ -74,6 +80,36 @@ The collector should receive OTLP trace export requests. The tracing backend should show spans for NeMo Relay scopes, tools, LLM calls, and marks grouped by root scope. +### GenAI Semantic Projection + +The opt-in `gen_ai` projection uses these operation names and span kinds: + +| Relay Scope | OpenTelemetry Operation | Span Kind | +|---|---|---| +| Agent | `invoke_agent` | `INTERNAL` | +| LLM | `chat`, `generate_content`, or `text_completion` | `CLIENT` | +| Tool | `execute_tool` | `INTERNAL` | +| Embedder | `embeddings` | `CLIENT` | +| Retriever | `retrieval` | `CLIENT` | +| Reranker | `rerank` | `CLIENT` | + +OpenTelemetry 1.37 does not define a first-class reranking operation, so +`rerank` is a Relay extension used by compatible AI-observability backends. +Unsupported scope types and marks retain the generic projection. + +Where normalized Relay data is available, GenAI spans include standardized +`gen_ai.*` attributes for provider and model identity, request parameters, +token usage, conversations, agents, tools, data sources, status, and errors. +They continue to include `nemo_relay.*` identity, ancestry, accounting, and +correlation attributes. + +Content capture is independent of semantic selection and defaults to disabled. +When `capture_content = false`, Relay does not emit messages, system +instructions, descriptions, tool definitions, tool arguments, tool results, +retrieval queries, or retrieved content in GenAI attributes. When enabled, +Relay projects only the event after sanitize guardrails and normalization; it +does not read the original callback arguments or return value. + The default `inherit` projection follows exporter-native handling: a mark with an active parent span is a span event, while an orphan mark is a standalone zero-duration `mark:` span. `mark_projection = "event"` explicitly selects @@ -95,8 +131,9 @@ attributes. These values match ATIF `step.extra.ancestry.function_id` and the trajectory-root span's `nemo_relay.uuid` also matches the ATIF `session_id`. Backend-native `trace_id` and `span_id` values are not written into ATIF. -NeMo Relay projects top-level lifecycle payload fields to typed OTLP attributes -with dotted names. For example, `metadata = { tenant = "acme" }` becomes +In the default `generic` mode, NeMo Relay projects top-level lifecycle payload +fields to typed OTLP attributes with dotted names. For example, +`metadata = { tenant = "acme" }` becomes `nemo_relay.start.metadata.tenant = "acme"`. - Start events use the `nemo_relay.start.data`, `nemo_relay.start.metadata`, and @@ -143,8 +180,10 @@ it is a span event when a parent is active and a zero-duration root span when it is orphaned. For LLM end spans, cost is emitted as `nemo_relay.llm.cost.total` and -`nemo_relay.llm.cost.currency` (any currency). Token counts are not emitted as -discrete attributes. Refer to +`nemo_relay.llm.cost.currency` (any currency). Generic mode does not emit token +counts as discrete attributes. GenAI mode additionally emits normalized input, +output, cache-read, and cache-creation token counts as `gen_ai.usage.*` +attributes when they are available. Refer to [Token and Cost Field Semantics](/integrate-into-frameworks/provider-response-codecs#token-and-cost-field-semantics) for the full mapping. @@ -177,6 +216,8 @@ config = plugin.PluginConfig( ObservabilityConfig( opentelemetry=OtlpConfig( enabled=True, + semantic_convention="gen_ai", + capture_content=False, transport="http_binary", endpoint="http://localhost:4318/v1/traces", service_name="agent-service", @@ -221,6 +262,8 @@ void (async () => { version: 2, opentelemetry: observability.otlpConfig({ enabled: true, + semantic_convention: "gen_ai", + capture_content: false, transport: "http_binary", endpoint: "http://localhost:4318/v1/traces", service_name: "agent-service", @@ -265,6 +308,8 @@ use nemo_relay::plugin::{ let component = ComponentSpec::new(ObservabilityConfig { opentelemetry: Some(OtlpSectionConfig { enabled: true, + semantic_convention: "gen_ai".into(), + capture_content: false, transport: "http_binary".into(), endpoint: Some("http://localhost:4318/v1/traces".into()), service_name: "agent-service".into(), @@ -314,6 +359,8 @@ config = OpenTelemetryConfig() config.transport = "http_binary" config.endpoint = "http://localhost:4318/v1/traces" config.service_name = "agent-service" +config.semantic_convention = "gen_ai" +config.capture_content = False config.set_resource_attribute("deployment.environment", "dev") subscriber = OpenTelemetrySubscriber(config) @@ -336,6 +383,8 @@ const subscriber = new OpenTelemetrySubscriber({ transport: "http_binary", endpoint: "http://localhost:4318/v1/traces", serviceName: "agent-service", + semanticConvention: "gen_ai", + captureContent: false, resourceAttributes: { "deployment.environment": "dev", }, @@ -356,11 +405,15 @@ try { ```rust -use nemo_relay::observability::otel::{OpenTelemetryConfig, OpenTelemetrySubscriber}; +use nemo_relay::observability::otel::{ + OpenTelemetryConfig, OpenTelemetrySemanticConvention, OpenTelemetrySubscriber, +}; fn main() -> Result<(), Box> { let config = OpenTelemetryConfig::http_binary("agent-service") .with_endpoint("http://localhost:4318/v1/traces") + .with_semantic_convention(OpenTelemetrySemanticConvention::GenAi) + .with_content_capture(false) .with_resource_attribute("deployment.environment", "dev"); let subscriber = OpenTelemetrySubscriber::new(config)?; subscriber.register("otel-exporter")?; @@ -381,6 +434,8 @@ fn main() -> Result<(), Box> { ## Common Configuration and Runtime Issues - `transport` is not `http_binary` or `grpc`. +- `semantic_convention` is not `generic` or `gen_ai`. +- GenAI-only options are configured under the `openinference` section. - Headers or resource attributes are not string-to-string maps. - The exporter feature is unavailable in the current build or target. - The endpoint is unreachable at runtime. diff --git a/docs/integrate-into-frameworks/provider-response-codecs.mdx b/docs/integrate-into-frameworks/provider-response-codecs.mdx index 5111e201e..839e1c3ba 100644 --- a/docs/integrate-into-frameworks/provider-response-codecs.mdx +++ b/docs/integrate-into-frameworks/provider-response-codecs.mdx @@ -389,12 +389,12 @@ currencies. Each exporter projects `usage`/`cost` differently. Projections do not change the canonical fields above. -| | ATOF | ATIF step / `final_metrics` | OpenInference | OpenTelemetry | +| | ATOF | ATIF step / `final_metrics` | OpenInference | OpenTelemetry (`generic` / `gen_ai`) | |---|---|---|---|---| -| Prompt tokens | full `usage` preserved | `prompt_tokens` / `total_prompt_tokens` | `llm.token_count.prompt` | not emitted | -| Completion tokens | preserved | `completion_tokens` / `total_completion_tokens` | `llm.token_count.completion` | not emitted | +| Prompt tokens | full `usage` preserved | `prompt_tokens` / `total_prompt_tokens` | `llm.token_count.prompt` | not emitted / `gen_ai.usage.input_tokens` | +| Completion tokens | preserved | `completion_tokens` / `total_completion_tokens` | `llm.token_count.completion` | not emitted / `gen_ai.usage.output_tokens` | | Total tokens | preserved | no first-class field | `llm.token_count.total` | not emitted | -| Cache read / write | preserved | summed into `cached_tokens` / `total_cached_tokens` | `llm.token_count.prompt_details.cache_read` / `…cache_write` | not emitted | +| Cache read / write | preserved | summed into `cached_tokens` / `total_cached_tokens` | `llm.token_count.prompt_details.cache_read` / `…cache_write` | not emitted / `gen_ai.usage.cache_read.input_tokens` and `gen_ai.usage.cache_creation.input_tokens` | | Cost | full `cost` preserved | `cost_usd` / `total_cost_usd` (USD only) | `llm.cost.total` (USD only) | `nemo_relay.llm.cost.total` + `nemo_relay.llm.cost.currency` (any currency) | OpenTelemetry carries cost in any currency, while ATIF and OpenInference report cost @@ -421,7 +421,8 @@ consumers. The following behaviors are intentional in this release but can change later: -- OpenTelemetry emits cost only, not token counts. +- Generic OpenTelemetry emits cost only. The opt-in GenAI projection also emits + normalized input, output, cache-read, and cache-creation token counts. - ATIF and OpenInference report cost only in USD. - Reasoning tokens are not a first-class `Usage` field. - Bindings expose `usage`/`cost` as snake_case JSON rather than typed objects. diff --git a/docs/supported-integrations/openclaw-plugin.mdx b/docs/supported-integrations/openclaw-plugin.mdx index 5c079ab3b..e5630eb4e 100644 --- a/docs/supported-integrations/openclaw-plugin.mdx +++ b/docs/supported-integrations/openclaw-plugin.mdx @@ -162,8 +162,10 @@ do not use, or set their `enabled` fields to `false`. `adaptive`. - `config.plugins.components[].config.atif` writes ATIF trajectory JSON files. Set `output_directory` to the directory where OpenClaw should write files. -- `config.plugins.components[].config.opentelemetry` sends generic OTLP spans to - an OpenTelemetry collector when `enabled` is `true`. +- `config.plugins.components[].config.opentelemetry` sends OTLP spans to an + OpenTelemetry collector when `enabled` is `true`. Generic Relay semantics are + the default; set `semantic_convention` to `gen_ai` for the OpenTelemetry GenAI + 1.37 or newer projection. - `config.plugins.components[].config.openinference` sends OpenInference OTLP spans to Phoenix or another OpenInference-compatible collector when `enabled` is `true`. diff --git a/go/nemo_relay/nemo_relay.go b/go/nemo_relay/nemo_relay.go index b8b5280dd..f7a0918f8 100644 --- a/go/nemo_relay/nemo_relay.go +++ b/go/nemo_relay/nemo_relay.go @@ -252,6 +252,7 @@ extern void nemo_relay_atof_exporter_free(void*); // OpenTelemetry subscriber extern int32_t nemo_relay_otel_subscriber_create(const char*, const char*, const char*, const char*, const char*, const char*, const char*, const char*, uint64_t, void**); extern int32_t nemo_relay_otel_subscriber_create_with_attribute_mappings(const char*, const char*, const char*, const char*, const char*, const char*, const char*, const char*, uint64_t, const char*, void**); +extern int32_t nemo_relay_otel_subscriber_create_with_options(const char*, const char*, const char*, const char*, const char*, const char*, const char*, const char*, uint64_t, const char*, const char*, _Bool, void**); extern int32_t nemo_relay_otel_subscriber_register(const void*, const char*); extern int32_t nemo_relay_otel_subscriber_deregister(const char*); extern int32_t nemo_relay_otel_subscriber_force_flush(const void*); @@ -1861,6 +1862,16 @@ const ( OpenTelemetryTransportGrpc OpenTelemetryTransport = "grpc" ) +// OpenTelemetrySemanticConvention selects the span attribute projection. +type OpenTelemetrySemanticConvention string + +const ( + // OpenTelemetrySemanticConventionGeneric preserves Relay's generic OpenTelemetry output. + OpenTelemetrySemanticConventionGeneric OpenTelemetrySemanticConvention = "generic" + // OpenTelemetrySemanticConventionGenAI emits OpenTelemetry GenAI 1.37+ conventions. + OpenTelemetrySemanticConventionGenAI OpenTelemetrySemanticConvention = "gen_ai" +) + // OpenTelemetryConfig configures the OpenTelemetry subscriber. // // Create it with [NewOpenTelemetryConfig], then mutate fields as needed before @@ -1876,6 +1887,8 @@ type OpenTelemetryConfig struct { InstrumentationScope string Timeout time.Duration AttributeMappings []OtlpAttributeMapping + SemanticConvention OpenTelemetrySemanticConvention + CaptureContent bool } // OtlpAttributeMapping copies a projected OTLP attribute to an alias. @@ -1893,6 +1906,7 @@ func NewOpenTelemetryConfig() OpenTelemetryConfig { ServiceName: defaultServiceName, InstrumentationScope: "nemo-relay-otel", Timeout: 3 * time.Second, + SemanticConvention: OpenTelemetrySemanticConventionGeneric, } } @@ -1921,6 +1935,9 @@ func NewOpenTelemetrySubscriber(config OpenTelemetryConfig) (*OpenTelemetrySubsc if config.ResourceAttributes == nil { config.ResourceAttributes = map[string]string{} } + if config.SemanticConvention == "" { + config.SemanticConvention = OpenTelemetrySemanticConventionGeneric + } cTransport := C.CString(string(config.Transport)) defer C.free(unsafe.Pointer(cTransport)) @@ -1972,9 +1989,12 @@ func NewOpenTelemetrySubscriber(config OpenTelemetryConfig) (*OpenTelemetrySubsc cInstrumentationScope := C.CString(config.InstrumentationScope) defer C.free(unsafe.Pointer(cInstrumentationScope)) + cSemanticConvention := C.CString(string(config.SemanticConvention)) + defer C.free(unsafe.Pointer(cSemanticConvention)) + cCaptureContent := C.bool(config.CaptureContent) var ptr unsafe.Pointer - status := C.nemo_relay_otel_subscriber_create_with_attribute_mappings( + status := C.nemo_relay_otel_subscriber_create_with_options( cTransport, cEndpoint, cHeadersJSON, @@ -1985,6 +2005,8 @@ func NewOpenTelemetrySubscriber(config OpenTelemetryConfig) (*OpenTelemetrySubsc cInstrumentationScope, C.uint64_t(config.Timeout/time.Millisecond), cAttributeMappingsJSON, + cSemanticConvention, + cCaptureContent, &ptr, ) if err := checkStatus(status); err != nil { diff --git a/go/nemo_relay/observability_plugin.go b/go/nemo_relay/observability_plugin.go index 0fb179114..a1586a386 100644 --- a/go/nemo_relay/observability_plugin.go +++ b/go/nemo_relay/observability_plugin.go @@ -181,19 +181,21 @@ func (config ObservabilityHttpStorageConfig) MarshalJSON() ([]byte, error) { // ObservabilityOtlpConfig configures OpenTelemetry or OpenInference OTLP export. type ObservabilityOtlpConfig struct { - Enabled bool `json:"enabled,omitempty"` - MarkProjection ObservabilityMarkProjection `json:"mark_projection,omitempty"` - MarkExcludeNames []string `json:"mark_exclude_names,omitempty"` - AttributeMappings []OtlpAttributeMapping `json:"attribute_mappings,omitempty"` - Transport string `json:"transport,omitempty"` - Endpoint string `json:"endpoint,omitempty"` - Headers map[string]string `json:"headers,omitempty"` - ResourceAttributes map[string]string `json:"resource_attributes,omitempty"` - ServiceName string `json:"service_name,omitempty"` - ServiceNamespace string `json:"service_namespace,omitempty"` - ServiceVersion string `json:"service_version,omitempty"` - InstrumentationScope string `json:"instrumentation_scope,omitempty"` - TimeoutMillis uint64 `json:"timeout_millis,omitempty"` + Enabled bool `json:"enabled,omitempty"` + MarkProjection ObservabilityMarkProjection `json:"mark_projection,omitempty"` + MarkExcludeNames []string `json:"mark_exclude_names,omitempty"` + AttributeMappings []OtlpAttributeMapping `json:"attribute_mappings,omitempty"` + SemanticConvention OpenTelemetrySemanticConvention `json:"semantic_convention,omitempty"` + CaptureContent bool `json:"capture_content,omitempty"` + Transport string `json:"transport,omitempty"` + Endpoint string `json:"endpoint,omitempty"` + Headers map[string]string `json:"headers,omitempty"` + ResourceAttributes map[string]string `json:"resource_attributes,omitempty"` + ServiceName string `json:"service_name,omitempty"` + ServiceNamespace string `json:"service_namespace,omitempty"` + ServiceVersion string `json:"service_version,omitempty"` + InstrumentationScope string `json:"instrumentation_scope,omitempty"` + TimeoutMillis uint64 `json:"timeout_millis,omitempty"` } // MarshalJSON preserves the distinction between a nil exclusion list, which @@ -271,6 +273,7 @@ func NewObservabilityOtlpConfig() ObservabilityOtlpConfig { Transport: "http_binary", MarkProjection: ObservabilityMarkProjectionInherit, MarkExcludeNames: []string{"llm.chunk"}, + SemanticConvention: OpenTelemetrySemanticConventionGeneric, Headers: map[string]string{}, ResourceAttributes: map[string]string{}, ServiceName: "nemo-relay", diff --git a/go/nemo_relay/observability_plugin_test.go b/go/nemo_relay/observability_plugin_test.go index 6a0eea564..756ae7c60 100644 --- a/go/nemo_relay/observability_plugin_test.go +++ b/go/nemo_relay/observability_plugin_test.go @@ -66,7 +66,7 @@ func TestObservabilityConfigHelpers(t *testing.T) { httpStorage, } otlp := NewObservabilityOtlpConfig() - if otlp.Enabled || otlp.MarkProjection != ObservabilityMarkProjectionInherit || len(otlp.MarkExcludeNames) != 1 || otlp.MarkExcludeNames[0] != "llm.chunk" || otlp.Transport != "http_binary" || otlp.ServiceName != "nemo-relay" || otlp.TimeoutMillis != 3000 { + if otlp.Enabled || otlp.MarkProjection != ObservabilityMarkProjectionInherit || len(otlp.MarkExcludeNames) != 1 || otlp.MarkExcludeNames[0] != "llm.chunk" || otlp.SemanticConvention != OpenTelemetrySemanticConventionGeneric || otlp.CaptureContent || otlp.Transport != "http_binary" || otlp.ServiceName != "nemo-relay" || otlp.TimeoutMillis != 3000 { t.Fatalf("unexpected OTLP defaults: %#v", otlp) } otlp.MarkProjection = ObservabilityMarkProjectionTool diff --git a/go/nemo_relay/otel_test.go b/go/nemo_relay/otel_test.go index e3d78edab..8fd87487d 100644 --- a/go/nemo_relay/otel_test.go +++ b/go/nemo_relay/otel_test.go @@ -45,6 +45,12 @@ func TestNewOpenTelemetryConfigDefaults(t *testing.T) { if config.Timeout != 3*time.Second { t.Fatalf("expected default timeout 3s, got %v", config.Timeout) } + if config.SemanticConvention != OpenTelemetrySemanticConventionGeneric { + t.Fatalf("expected generic semantic convention, got %q", config.SemanticConvention) + } + if config.CaptureContent { + t.Fatal("expected content capture to be disabled") + } if config.Headers == nil || len(config.Headers) != 0 { t.Fatalf("expected empty headers map, got %#v", config.Headers) } @@ -67,6 +73,8 @@ func TestOpenTelemetrySubscriberLifecycle(t *testing.T) { Key: "nemo_relay.start.data.tenant", Alias: otelTenantAttributeAlias, }} + config.SemanticConvention = OpenTelemetrySemanticConventionGenAI + config.CaptureContent = true subscriber, err := NewOpenTelemetrySubscriber(config) if err != nil { @@ -111,6 +119,15 @@ func TestOpenTelemetrySubscriberRejectsInvalidAttributeMapping(t *testing.T) { } } +func TestOpenTelemetrySubscriberRejectsInvalidSemanticConvention(t *testing.T) { + config := NewOpenTelemetryConfig() + config.SemanticConvention = OpenTelemetrySemanticConvention("future") + + if _, err := NewOpenTelemetrySubscriber(config); err == nil { + t.Fatal("expected invalid semantic convention error") + } +} + func TestOpenTelemetrySubscriberExportsScopeLifecycleAndMarks(t *testing.T) { type otelRequest struct { Path string diff --git a/python/nemo_relay/README.md b/python/nemo_relay/README.md index 0927b4f04..5e4df9bd7 100644 --- a/python/nemo_relay/README.md +++ b/python/nemo_relay/README.md @@ -48,7 +48,8 @@ The Python package provides the following capabilities: responses, and execution, plus mark and scope event sanitizers for `data`, `category_profile`, and `metadata`. - **Subscribers and exporters**: Event consumers for observability and - diagnostics. + diagnostics, including generic OpenTelemetry and the opt-in OpenTelemetry + GenAI 1.37 or newer projection. - **Plugin and typed helpers**: Public modules for plugins, codecs, typed wrappers, adaptive runtime behavior, and observability plugin configuration. - **Shared Rust runtime semantics**: Python behavior aligned with the Rust diff --git a/python/nemo_relay/_native.pyi b/python/nemo_relay/_native.pyi index fac3ed23d..3c6884bf0 100644 --- a/python/nemo_relay/_native.pyi +++ b/python/nemo_relay/_native.pyi @@ -955,6 +955,8 @@ class OpenTelemetryConfig: service_version: Optional[str] instrumentation_scope: str timeout_millis: int + semantic_convention: Literal["generic", "gen_ai"] + capture_content: bool def __init__(self) -> None: """Create an OpenTelemetry config with native defaults.""" diff --git a/python/nemo_relay/observability.py b/python/nemo_relay/observability.py index 5f9bc8837..f3df6c878 100644 --- a/python/nemo_relay/observability.py +++ b/python/nemo_relay/observability.py @@ -215,7 +215,11 @@ def to_dict(self) -> JsonObject: @dataclass(slots=True) class OtlpConfig: - """Shared OpenTelemetry/OpenInference OTLP export settings.""" + """Shared OpenTelemetry/OpenInference OTLP export settings. + + ``semantic_convention`` and ``capture_content`` apply only to the + OpenTelemetry section. OpenInference supports the generic value only. + """ enabled: bool = False mark_projection: MarkProjection = "inherit" @@ -230,6 +234,8 @@ class OtlpConfig: instrumentation_scope: str | None = None timeout_millis: int = 3000 attribute_mappings: list[dict[str, str]] = field(default_factory=list) + semantic_convention: Literal["generic", "gen_ai"] = "generic" + capture_content: bool = False def to_dict(self) -> JsonObject: """Serialize this OTLP config to the canonical JSON object shape.""" @@ -239,6 +245,8 @@ def to_dict(self) -> JsonObject: "mark_projection": self.mark_projection, "mark_exclude_names": self.mark_exclude_names, "attribute_mappings": self.attribute_mappings, + "semantic_convention": self.semantic_convention, + "capture_content": self.capture_content, "transport": self.transport, "endpoint": self.endpoint, "headers": self.headers, diff --git a/python/nemo_relay/observability.pyi b/python/nemo_relay/observability.pyi index d7b24807d..9521ad4cd 100644 --- a/python/nemo_relay/observability.pyi +++ b/python/nemo_relay/observability.pyi @@ -93,6 +93,8 @@ class OtlpConfig: instrumentation_scope: str | None = ... timeout_millis: int = ... attribute_mappings: list[dict[str, str]] = field(default_factory=list) + semantic_convention: Literal["generic", "gen_ai"] = ... + capture_content: bool = ... def to_dict(self) -> JsonObject: ... @dataclass(slots=True) diff --git a/python/tests/test_observability_plugin.py b/python/tests/test_observability_plugin.py index c82626fdf..95325f9f1 100644 --- a/python/tests/test_observability_plugin.py +++ b/python/tests/test_observability_plugin.py @@ -96,6 +96,8 @@ def test_defaults_and_component_wrapper(self): "mark_projection": "inherit", "mark_exclude_names": ["llm.chunk"], "attribute_mappings": [], + "semantic_convention": "generic", + "capture_content": False, "transport": "http_binary", "headers": {}, "resource_attributes": {}, @@ -103,6 +105,9 @@ def test_defaults_and_component_wrapper(self): "timeout_millis": 3000, } assert OtlpConfig(mark_projection="tool").to_dict()["mark_projection"] == "tool" + assert ( + OtlpConfig(semantic_convention="gen_ai", capture_content=True).to_dict()["semantic_convention"] == "gen_ai" + ) wrapped = ComponentSpec(ObservabilityConfig(atof=AtofConfig())).to_dict() assert wrapped["kind"] == OBSERVABILITY_PLUGIN_KIND diff --git a/python/tests/test_types.py b/python/tests/test_types.py index f537216b1..af5603944 100644 --- a/python/tests/test_types.py +++ b/python/tests/test_types.py @@ -574,6 +574,8 @@ def test_config_defaults_mutation_and_repr(self): assert config.service_name == "nemo-relay" assert config.instrumentation_scope == "nemo-relay-otel" assert config.timeout_millis == 3000 + assert config.semantic_convention == "generic" + assert config.capture_content is False assert config.headers == {} assert config.resource_attributes == {} @@ -583,6 +585,8 @@ def test_config_defaults_mutation_and_repr(self): config.service_version = "1.0.0" config.instrumentation_scope = "py-tests" config.timeout_millis = 1250 + config.semantic_convention = "gen_ai" + config.capture_content = True config.set_header("authorization", "Bearer token") config.set_resource_attribute("deployment.environment", "test") @@ -636,6 +640,11 @@ def test_subscriber_lifecycle_and_invalid_transport(self): with pytest.raises(ValueError, match="transport must be"): OpenTelemetrySubscriber(bad) + bad_semantic_convention = OpenTelemetryConfig() + setattr(bad_semantic_convention, "semantic_convention", "future") + with pytest.raises(ValueError, match="semantic_convention must be"): + OpenTelemetrySubscriber(bad_semantic_convention) + def test_subscriber_exports_scope_and_mark_events_end_to_end(self): with _OtelCollector() as collector: source = "python-é" * 20 @@ -670,6 +679,29 @@ def test_subscriber_exports_scope_and_mark_events_end_to_end(self): subscriber.deregister(subscriber_name) subscriber.shutdown() + def test_subscriber_exports_opt_in_genai_projection(self): + with _OtelCollector() as collector: + config = OpenTelemetryConfig() + config.endpoint = collector.endpoint + config.service_name = "py-genai-agent" + config.semantic_convention = "gen_ai" + + subscriber = OpenTelemetrySubscriber(config) + subscriber_name = f"py_otel_genai_{uuid4().hex}" + subscriber.register(subscriber_name) + try: + handle = scope.push("planner", ScopeType.Agent) + scope.pop(handle) + subscriber.force_flush() + + request = collector.wait_for_request() + assert _otlp_string_attribute("gen_ai.operation.name", "invoke_agent") in request["body"] + assert _otlp_string_attribute("gen_ai.agent.name", "planner") in request["body"] + assert _otlp_string_attribute("nemo_relay.otel.semantic_convention", "1.37+") in request["body"] + finally: + subscriber.deregister(subscriber_name) + subscriber.shutdown() + class TestOpenInferenceTypes: def test_config_defaults_mutation_and_repr(self): diff --git a/skills/nemo-relay-plugin-observability/references/opentelemetry.md b/skills/nemo-relay-plugin-observability/references/opentelemetry.md index dffc7cead..bcfc8f772 100644 --- a/skills/nemo-relay-plugin-observability/references/opentelemetry.md +++ b/skills/nemo-relay-plugin-observability/references/opentelemetry.md @@ -30,6 +30,25 @@ OpenTelemetry Collector, Jaeger, Tempo, or Honeycomb. - NeMo Relay emits a top-level object or array field as a JSON string, omits a top-level `null` field, and no longer emits the old aggregate `*_json` payload attributes. +- Generic projection remains the default. Set `semantic_convention` to + `gen_ai` to use OpenTelemetry GenAI semantic conventions 1.37 or newer for + agent, LLM, tool, embedding, retrieval, and reranking scopes. The exporter, + transport, authentication, resource, mark, and flush settings remain the + same. +- In GenAI projection, Relay emits standardized `gen_ai.*`, `error.type`, and + `server.*` attributes alongside `nemo_relay.*` correlation and accounting + attributes. Reranking uses the documented Relay extension operation name + `rerank` because OpenTelemetry does not define a first-class reranking + operation. +- GenAI content capture is disabled by default. Set `capture_content` to `true` + only after configuring Relay sanitization. When disabled, Relay omits system + instructions, input and output messages, tool descriptions, tool + definitions, tool arguments and results, retrieval queries and documents, + and agent descriptions. When enabled, the projection reads the sanitized + event representation rather than the callback's original values. +- The `semantic_convention` and `capture_content` settings apply only to the + OpenTelemetry section. OpenInference keeps its existing semantic and content + behavior and rejects GenAI-only settings. - Use `attribute_mappings` to copy a fully qualified projected attribute to a backend-specific alias without changing its OTLP type. - Start with `http_binary` transport and an OTLP traces endpoint such as a local @@ -56,8 +75,32 @@ OpenTelemetry Collector, Jaeger, Tempo, or Honeycomb. - Endpoint, TLS verification, and required authentication header names - Service naming and resource attributes - Whether deterministic flush-before-exit is required +- Whether the backend expects generic Relay attributes or the opt-in GenAI + projection +- Whether content capture is permitted and sanitization has been verified - Whether the chosen binding and target support the desired transport +## GenAI Configuration + +Use the same OTLP destination for generic and GenAI projection. For plugin +configuration, add these fields to the `opentelemetry` section: + +```json +{ + "enabled": true, + "semantic_convention": "gen_ai", + "capture_content": false, + "transport": "http_binary", + "endpoint": "http://127.0.0.1:4318/v1/traces", + "service_name": "my-agent" +} +``` + +For direct subscriber APIs, set `semantic_convention` and `capture_content` on +the Python config, `semanticConvention` and `captureContent` on the Node.js +config, `SemanticConvention` and `CaptureContent` on the Go config, or use +`with_semantic_convention(...)` and `with_content_capture(...)` in Rust. + ## Troubleshooting Focus - No spans visible