Repository navigation
Open
Description
The test audit on the test/test-audit-campaign branch found these gaps. They existed before the audit, and the audit did not remove their proof.
Decisions:
- The second parameter of
telemetryDevin@telemetry-dev/ai-sdkand@telemetry-dev/tanstack-ai(overrides), andMediaWrapperOverrides, have no caller outside the tests. Two options: document them as public options, or change the tests to decode the OTLP requests from afetchspy and remove the parameter in a major release. postOtlpandmaybeGzipare public exports of@telemetry-dev/otel, but only the package itself uses them. A removal is a breaking change.
Coverage:
- No test runs the ai@6 integration through the real
ai-v6dispatcher. Every ai@6 test sends hand-built events. Done in test(ai): cover the AI SDK paths the test audit found #169. - These AI SDK paths have no test: the gateway relabel in the ai@7 and media paths, the ai@7 step without
onStepStart,executeTool,wrapGenerateVideo,wrapGetBatchStatus,wrapCancelBatch, the transcription duration, the video counts, and the wrapper passthrough without a key. Done in test(ai): cover the AI SDK paths the test audit found #169. - The
onConfigphase guard inpackages/tanstack-ai/src/middleware.tshas no test. Supported engines do not callonConfigafteronStart. Done in test(sdk-tanstack-ai): cover the onConfig phase guard #162. - No release check makes sure that each
exportsandpublishConfig.exportstarget is in the packed tarball. Done in ci(sdk): check that each npm tarball has the targets of its exports #155 and ci(sdk): fail npm tarballs with no export or bin targets, and print the counts #160. - The Python core has no wire test for
on_errorwith a server that returns an error. Done in fix(sdk-python): report a series of failed exports once #87. - In the ai@7 integration, the object and embed step ends set the provider on the step's duration and token metrics, and no test checks it. If either recorded the end event's raw provider, the metrics would say
gatewayinstead ofVercel AI Gatewayand every test would still pass. The gateway table that test(ai): cover the AI SDK paths the test audit found #169 added topackages/ai/tests/integration_v7.test.tscould check each metric'sgen_ai.provider.name. Done in test(ai): check the gateway relabel on the ai@7 step metrics #209. - In the ai@7 integration, the
execute_toolduration metrics take their provider from the root (metricBaseinrecordQueuedMetricsinpackages/ai/src/integration_v7.ts), and no test checks it. If that provider were dropped or recorded asgateway, every test would still pass. No row of the gateway table runs a tool, so a row with a tool call could check it.
Python compatibility code:
- The Bedrock, Google GenAI, LiteLLM, and OpenRouter integrations keep fallbacks for span handle methods (
capture_masked,record_output_chunk) that each supported core version has. The Anthropic and OpenAI integrations no longer have them. - CI tests each Python package against the core in this repository, also in the lowest-bounds job. If the
telemetry-devfloor of an integration goes below the version that has these handle methods, no test fails. - In the Python core,
CaptureBudget.from_clientand the publicmetrics_recorderparameter ofTelemetrySpanProcessorhave no caller.
Metadata
Metadata
Assignees
Labels
No labels
Activity
lofoneh commentedon Oct 31, 2026
I'd like to take the coverage part of this. I checked it against main today. The tarball check is covered now by #155 and #160, and the Python
on_errorwire test by #87. The other three are still open. For example, removing theonConfigphase guard in tanstack-ai still passes all its tests.My plan:
ai-v6dispatcheronConfigphase guardI'll leave the decisions and the Python compatibility items to you. I'd send the tanstack-ai test first as a small PR, then the AI SDK tests in a second one.
lofoneh commentedon Oct 31, 2026
Quick question before I send the AI SDK tests. They capture spans through the
overridesargument, like the rest of the ai-sdk and tanstack-ai tests (#162 too). Which way are you leaning on that decision?I'd lean toward keeping it. The SDKs send OTLP as protobuf, so a
fetchspy would mean decoding spans and metrics from the request bodies in every test. Each test file also captures through one helper, so a later switch would mostly touch those helpers. If you'd rather remove it, I can take that rewrite too.test(sdk-tanstack-ai): cover the onConfig phase guard (#162)
aikins01 commentedon Nov 4, 2026
@lofoneh sorry for the slow answer. Let's keep
overrides. Decoding protobuf in every test would cost more than the argument does, and it lets the tests check the real spans and metric points. #169 can keep capturing through it as it does now.I'll document it as a supported way to capture spans and metrics in tests, in the ai-sdk and tanstack-ai READMEs and on the type in
@telemetry-dev/otel, so it stops looking like a leftover.test(ai): cover the AI SDK paths the test audit found
test(ai): cover the AI SDK paths the test audit found (#169)
lofoneh commentedon Nov 5, 2026
Thanks for the reviews on #162 and #169. I'll take the last coverage item too, the provider on the step metrics in the gateway table.
test(ai): check the gateway relabel on the ai@7 step metrics (#209)
lofoneh commentedon Nov 6, 2026
Thanks for merging #209. I'll take the execute_tool metrics item too, by adding a tool call to the generateText row in the gateway table.