Repository navigation
Conversation
The OTLP specification lets a server accept part of an export request and reject the rest, reporting it in the partial_success field of the response. A partial_success holding a zero rejected count and an empty error message is equivalent to the field being absent, and the helper reports neither. A response that cannot be parsed is logged at debug level and otherwise ignored, so a malformed body never turns an export the server accepted into a failed one.
The HTTP client is shared by every OTLP/HTTP exporter and is the only place holding the transport result, so it is where the response body is read. A body that cannot be read is logged at debug level and replaced with an empty value, so an export the server accepted is never turned into a failure.
The value returned by Export was discarded, so a rejection was never seen. Reporting is best effort: OTLPExporterMixin is public and a subclass may export through a service of its own, so a response the helper cannot read is logged at debug level rather than raised at a caller whose request succeeded.
Any 2xx or 3xx counted as a full success and the body was never deserialized. Each of the three exporters now parses it with its own Export<signal>ServiceResponse type.
ocelotl
force-pushed
the
issue_2882_otlp_partial_success
branch
from
October 31, 2026 14:36
f6b1074 to
2c5ce22
Compare
Owner
Author
|
Opened upstream as open-telemetry#5871 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Handles the OTLP
partial_successresponse field, which every OTLP exporter in this repository ignored. Tracked upstream as open-telemetry#2882.This pull request targets
issue_2882_base, a branch pinned toopen-telemetry/opentelemetry-python@3462000f3, so the diff shows only the commits of this change. It is opened in the fork for review and is not proposed upstream yet.The problem
A collector that accepts part of an export request and rejects the rest reports that in the
partial_successfield of the export response. Neither transport read that field, so rejected spans, data points and log records were lost with nothing written to the logs.The gRPC exporter called
self._client.Export(...)without binding the return value. The three HTTP exporters returnedSUCCESSon any 2xx without deserializing the body. A repository-wide search forpartial_successacross the exporter sources returned nothing.The change
The change, in four parts:
opentelemetry-exporter-otlp-proto-common: a new_partial_successmodule with_partial_success_detail, which turns a parsed response into the detail logged as the last argument of the format string_warning_formatbuilds, and_log_partial_success, which parses a serialized response first. Per the specification, apartial_successholding a zero rejected count and an empty error message is equivalent to the field being absent, and neither helper reports it.opentelemetry-exporter-otlp-common:_ExportResultnow carriescontent, the raw body of a successful export. The HTTP client is shared across signals and is the only place holding the transport result, so it reads the body; a body that cannot be read is reported and replaced with an empty value, so an accepted export is never turned into a failure.opentelemetry-exporter-otlp-proto-grpc: bind the valueExportreturns and log the partial success it reports.opentelemetry-exporter-otlp-proto-http: parse the response body in the span, metric and log exporters, each passing its ownExport<signal>ServiceResponsetype.Messages the user sees:
The third form is the warning-only case the specification allows, where the rejected count is zero and the error message is not empty.
The two transports name the trace signal differently,
spansover OTLP/HTTP andtracesover gRPC. That is not new here: each message takes the word its own package already uses for the export failure messages beside it,kind="spans"in the OTLP/HTTP span exporter and_exportingin the gRPC mixin.Verification
otlp-proto-common, 35 inotlp-common, 123 inotlp-proto-http, 87 inotlp-proto-grpc. The 76 tests ofotlp-json-http, which shares the HTTP client, pass too.partial_successcases, the warning-only case, an unparsable body, a non-protobuf body, invalid UTF-8 in the error message, an unreadable response body, a gRPC stub returning nothing, that the loggedLogRecord.msgstays constant across repeats of one signal soDuplicateFiltercollapses them, that it differs between signals so one signal cannot mask another, and a count of one being reported in the singular.PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python, the runtime the PyPy environment uses.tox -e precommitpasses.tox -e typecheckreports no error in any changed file. The two errors it does report are in the Python 3.14 standard library and are present with this change reverted.Review
This branch has been through thirteen rounds of adversarial self-review, the last of which ran the change against real servers rather than fakes. The fixed findings were:
UnicodeDecodeErrorescaping from the parser under the pure-Python protobuf runtime, a warning on every export for any non-protobuf success body, the missingrejected_profilesfield name, four tests whose bodies serialized to empty and so never parsed anything, a gRPC stub returningNonebeing dereferenced, theUnicodeDecodeErrorbranch not being exercised by any runtime CI runs, the interpolated warning text defeatingDuplicateFilter, a count of one reported with a plural noun, two tests of the filter that depended on which side of a twenty second boundary the clock sat, a docstring that credited the OTLP/HTTP span and metric loggers with aDuplicateFilterthey do not carry, and two response shapes that made reporting raise out of an export the server had accepted: a message type that does not definepartial_successat all, and one whosepartial_successhas noerror_messagefield.One of those rounds moved the signal into the format string, so that each of the three gRPC signals would get its own
DuplicateFilterkey rather than the first one to report silencing the other two. The round after it reverted that:DuplicateFilterremembers only the most recent key, so two signals reporting partial success alternately passed it every time, andBatchProcessor.emitinopentelemetry-sdk/src/opentelemetry/sdk/_shared_internal/__init__.pydoes not consult_SUPPRESS_INSTRUMENTATION_KEY, so a gRPC log exporter turned each of its own exports into a new record to export. A later round reverted that revert, because running both designs against a real two-signal gRPC pipeline showed what the argument could not: the shared format string reported 2 of 50 rejections and the per-signal one reports 43 of 50, while the self generated records it costs neither grow nor persist once the application goes idle. The signal is in the format string at HEAD. The measurement and the open question it depends on are below.Still to decide before this goes upstream
herin049offered to take it on 2026-07-14 in Handle partial success responses from OTLP export services open-telemetry/opentelemetry-python#2882 and was never assigned; they should be asked whether they still want it before this is proposed upstream.OTEL_LOG_LEVELrather than adding a dedicated environment variable was made bymichaelsafyanon 2025-11-11 in Make it possible to opt-in to logging partial errors open-telemetry/opentelemetry-python#4803, and in the thread of 4805DylanRussellasked "Why put this log behind an env var ? Why not just always log when the field is set", after which theOTEL_LOG_LEVELconditioning was dropped. This change logs atWARNINGthrough the existing module loggers and adds no new environment variable, which matches where that discussion ended, but it has not been confirmed with the maintainers.DuplicateFilterinopentelemetry-sdk/src/opentelemetry/sdk/_shared_internal/__init__.pykeys onLogRecord.msgand remembers only the most recent key, and the three gRPC exporters share one logger. Measured against a realgrpc.serverreturning a populatedpartial_successon every export of two signals, driven for 25 seconds by aTracerProviderwith aBatchSpanProcessorand aLoggerProviderwith aBatchLogRecordProcessorplus the SDKLoggingHandleron the root logger: one format string shared by every signal reported 2 of 50 rejections, because whichever signal reached the filter first silenced the others for the whole window. Putting the signal in the string reports 43 of 50. The cost, measured the same way, is that the exporter's own warnings keep the log pipeline busy, 42 self generated records over those 25 seconds against 2. That number does not grow with time, and an idle application settles at about one record per twenty second window, because repeats of a single signal still collapse. Reporting 4 per cent of the rejections is the worse failure for a change whose whole purpose is to tell the user a collector is dropping telemetry, so the signal goes in the string.DuplicateFilterchanged. In the review of Implement partial success logging in OTLP exporters open-telemetry/opentelemetry-python#4805,DylanRussellproposed keying it on(record.module, record.pathname, record.lineno, time_boundary)and remembering the last ten rather than the last one. With a key that does not readLogRecord.msg, this choice stops mattering: each call site would be throttled on its own, no signal would mask another, and the self generated records would fall back to one per window. That is an SDK change this pull request does not make.otel.sdk.exporter.span.exportedand its metric and log counterparts, that rejected items MUST count as failed and SHOULD carry anerror.typeofrejected. Splitting the count needsExporterMetrics.export_operationinexporter/opentelemetry-exporter-otlp-common/src/opentelemetry/exporter/otlp/common/_exporter_metrics.pyto accept a partial outcome, which this change does not touch, so it is left as a follow-up. This is not a regression: before this change the rejected count was never parsed, so it could not have been split.opentelemetry-exporter-otlp-json-httpare untouched. They would need the JSON form of the same field, which is a separate change..changelogentries are named after the pull request number upstream, which does not exist yet.