Skip to content

Report OTLP partial success responses in all exporters - #53

Closed
ocelotl wants to merge 5 commits into
issue_2882_basefrom
issue_2882_otlp_partial_success
Closed

ocelotl wants to merge 5 commits into
issue_2882_basefrom
issue_2882_otlp_partial_success

Conversation

@ocelotl

@ocelotl ocelotl commented Oct 30, 2026 •

Copy link
Copy Markdown
Owner

Handles the OTLP partial_success response field, which every OTLP exporter in this repository ignored. Tracked upstream as open-telemetry#2882.

This pull request targets issue_2882_base, a branch pinned to open-telemetry/opentelemetry-python@3462000f3, so the diff shows only the commits of this change. It is opened in the fork for review and is not proposed upstream yet.

The problem

A collector that accepts part of an export request and rejects the rest reports that in the partial_success field of the export response. Neither transport read that field, so rejected spans, data points and log records were lost with nothing written to the logs.

The gRPC exporter called self._client.Export(...) without binding the return value. The three HTTP exporters returned SUCCESS on any 2xx without deserializing the body. A repository-wide search for partial_success across the exporter sources returned nothing.

The change

The change, in four parts:

  1. opentelemetry-exporter-otlp-proto-common: a new _partial_success module with _partial_success_detail, which turns a parsed response into the detail logged as the last argument of the format string _warning_format builds, and _log_partial_success, which parses a serialized response first. Per the specification, a partial_success holding a zero rejected count and an empty error message is equivalent to the field being absent, and neither helper reports it.
  2. opentelemetry-exporter-otlp-common: _ExportResult now carries content, the raw body of a successful export. The HTTP client is shared across signals and is the only place holding the transport result, so it reads the body; a body that cannot be read is reported and replaced with an empty value, so an accepted export is never turned into a failure.
  3. opentelemetry-exporter-otlp-proto-grpc: bind the value Export returns and log the partial success it reports.
  4. opentelemetry-exporter-otlp-proto-http: parse the response body in the span, metric and log exporters, each passing its own Export<signal>ServiceResponse type.

Messages the user sees:

OTLP partial success exporting spans to http://localhost:4318/v1/traces: the server rejected 2 spans: quota exceeded
OTLP partial success exporting metrics to http://localhost:4318/v1/metrics: the server rejected 7 data points
OTLP partial success exporting traces to localhost:4317: deprecated attribute in use

The third form is the warning-only case the specification allows, where the rejected count is zero and the error message is not empty.

The two transports name the trace signal differently, spans over OTLP/HTTP and traces over gRPC. That is not new here: each message takes the word its own package already uses for the export failure messages beside it, kind="spans" in the OTLP/HTTP span exporter and _exporting in the gRPC mixin.

Verification

  • 287 tests pass across the four packages: 42 in otlp-proto-common, 35 in otlp-common, 123 in otlp-proto-http, 87 in otlp-proto-grpc. The 76 tests of otlp-json-http, which shares the HTTP client, pass too.
  • 37 of those are new. They cover all four signals the protos define, both transports, the absent and zeroed partial_success cases, the warning-only case, an unparsable body, a non-protobuf body, invalid UTF-8 in the error message, an unreadable response body, a gRPC stub returning nothing, that the logged LogRecord.msg stays constant across repeats of one signal so DuplicateFilter collapses them, that it differs between signals so one signal cannot mask another, and a count of one being reported in the singular.
  • The suite also passes under PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python, the runtime the PyPy environment uses.
  • tox -e precommit passes.
  • pylint reports 10.00/10 on all four packages.
  • tox -e typecheck reports no error in any changed file. The two errors it does report are in the Python 3.14 standard library and are present with this change reverted.

Review

This branch has been through thirteen rounds of adversarial self-review, the last of which ran the change against real servers rather than fakes. The fixed findings were: UnicodeDecodeError escaping from the parser under the pure-Python protobuf runtime, a warning on every export for any non-protobuf success body, the missing rejected_profiles field name, four tests whose bodies serialized to empty and so never parsed anything, a gRPC stub returning None being dereferenced, the UnicodeDecodeError branch not being exercised by any runtime CI runs, the interpolated warning text defeating DuplicateFilter, a count of one reported with a plural noun, two tests of the filter that depended on which side of a twenty second boundary the clock sat, a docstring that credited the OTLP/HTTP span and metric loggers with a DuplicateFilter they do not carry, and two response shapes that made reporting raise out of an export the server had accepted: a message type that does not define partial_success at all, and one whose partial_success has no error_message field.

One of those rounds moved the signal into the format string, so that each of the three gRPC signals would get its own DuplicateFilter key rather than the first one to report silencing the other two. The round after it reverted that: DuplicateFilter remembers only the most recent key, so two signals reporting partial success alternately passed it every time, and BatchProcessor.emit in opentelemetry-sdk/src/opentelemetry/sdk/_shared_internal/__init__.py does not consult _SUPPRESS_INSTRUMENTATION_KEY, so a gRPC log exporter turned each of its own exports into a new record to export. A later round reverted that revert, because running both designs against a real two-signal gRPC pipeline showed what the argument could not: the shared format string reported 2 of 50 rejections and the per-signal one reports 43 of 50, while the self generated records it costs neither grow nor persist once the application goes idle. The signal is in the format string at HEAD. The measurement and the open question it depends on are below.

Still to decide before this goes upstream

  • Issue 2882 is assigned to me. herin049 offered to take it on 2026-07-14 in Handle partial success responses from OTLP export services open-telemetry/opentelemetry-python#2882 and was never assigned; they should be asked whether they still want it before this is proposed upstream.
  • A previous attempt, Implement partial success logging in OTLP exporters open-telemetry/opentelemetry-python#4805, was closed unmerged. The argument for respecting OTEL_LOG_LEVEL rather than adding a dedicated environment variable was made by michaelsafyan on 2025-11-11 in Make it possible to opt-in to logging partial errors open-telemetry/opentelemetry-python#4803, and in the thread of 4805 DylanRussell asked "Why put this log behind an env var ? Why not just always log when the field is set", after which the OTEL_LOG_LEVEL conditioning was dropped. This change logs at WARNING through the existing module loggers and adds no new environment variable, which matches where that discussion ended, but it has not been confirmed with the maintainers.
  • The per-signal format string is a choice a maintainer may want to revisit, and it is the one part of this change that was settled by measurement rather than by reading. DuplicateFilter in opentelemetry-sdk/src/opentelemetry/sdk/_shared_internal/__init__.py keys on LogRecord.msg and remembers only the most recent key, and the three gRPC exporters share one logger. Measured against a real grpc.server returning a populated partial_success on every export of two signals, driven for 25 seconds by a TracerProvider with a BatchSpanProcessor and a LoggerProvider with a BatchLogRecordProcessor plus the SDK LoggingHandler on the root logger: one format string shared by every signal reported 2 of 50 rejections, because whichever signal reached the filter first silenced the others for the whole window. Putting the signal in the string reports 43 of 50. The cost, measured the same way, is that the exporter's own warnings keep the log pipeline busy, 42 self generated records over those 25 seconds against 2. That number does not grow with time, and an idle application settles at about one record per twenty second window, because repeats of a single signal still collapse. Reporting 4 per cent of the rejections is the worse failure for a change whose whole purpose is to tell the user a collector is dropping telemetry, so the signal goes in the string.
  • Both of those numbers would improve if DuplicateFilter changed. In the review of Implement partial success logging in OTLP exporters open-telemetry/opentelemetry-python#4805, DylanRussell proposed keying it on (record.module, record.pathname, record.lineno, time_boundary) and remembering the last ten rather than the last one. With a key that does not read LogRecord.msg, this choice stops mattering: each call site would be throttled on its own, no signal would mask another, and the self generated records would fall back to one per window. That is an SDK change this pull request does not make.
  • The exporter self-metrics still count a rejected item as exported. The semantic convention vendored in this repository states, for otel.sdk.exporter.span.exported and its metric and log counterparts, that rejected items MUST count as failed and SHOULD carry an error.type of rejected. Splitting the count needs ExporterMetrics.export_operation in exporter/opentelemetry-exporter-otlp-common/src/opentelemetry/exporter/otlp/common/_exporter_metrics.py to accept a partial outcome, which this change does not touch, so it is left as a follow-up. This is not a regression: before this change the rejected count was never parsed, so it could not have been split.
  • The JSON exporters under opentelemetry-exporter-otlp-json-http are untouched. They would need the JSON form of the same field, which is a separate change.
  • No changelog fragment yet, because .changelog entries are named after the pull request number upstream, which does not exist yet.

The OTLP specification lets a server accept part of an export request and
reject the rest, reporting it in the partial_success field of the response. A
partial_success holding a zero rejected count and an empty error message is
equivalent to the field being absent, and the helper reports neither.

A response that cannot be parsed is logged at debug level and otherwise
ignored, so a malformed body never turns an export the server accepted into a
failed one.
The HTTP client is shared by every OTLP/HTTP exporter and is the only place
holding the transport result, so it is where the response body is read. A body
that cannot be read is logged at debug level and replaced with an empty value,
so an export the server accepted is never turned into a failure.
The value returned by Export was discarded, so a rejection was never seen.
Reporting is best effort: OTLPExporterMixin is public and a subclass may
export through a service of its own, so a response the helper cannot read is
logged at debug level rather than raised at a caller whose request succeeded.
Any 2xx or 3xx counted as a full success and the body was never deserialized.
Each of the three exporters now parses it with its own
Export<signal>ServiceResponse type.
@ocelotl
ocelotl force-pushed the issue_2882_otlp_partial_success branch from f6b1074 to 2c5ce22 Compare October 31, 2026 14:36
@ocelotl

ocelotl commented Oct 31, 2026

Copy link
Copy Markdown
Owner Author

Opened upstream as open-telemetry#5871

@ocelotl ocelotl closed this Oct 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant