Traces not showing up on self-hosted 25.9.0 when the SDK sends transactions

[gravityvi]

How do you use Sentry?

Sentry Self Hosted v25.9.0

SDK version

0.47.0

Steps to reproduce

Summary

The Rust SDK (tested on 0.47.0) sends the old transaction envelope format,
routing data through the ingest-transactions Kafka topic. Self-hosted Sentry 25.x
has migrated performance storage to EAP (eap_items_1_local), and the pipeline
that sets the has_transactions project flag reads exclusively from ingest-spans
(fed by span v2 envelopes). Because the Rust SDK never writes to ingest-spans,
has_transactions is never set, and the Sentry UI permanently shows the onboarding
empty state instead of traces — even though the trace data arrives and is stored
correctly.

Data flow (current Rust SDK)

Rust SDK → transaction envelope
└─► ingest-transactions (Kafka)
└─► ingest-consumer → transactions topic
└─► transactions_group → snuba-items
└─► eap_items_group → eap_items_1_local ✅ (data stored correctly)

ingest-spans (Kafka) — EMPTY for Rust projects
└─► process-spans → buffered-segments — EMPTY
└─► process-segments → set has_transactions ← NEVER RUNS

Expected data flow (what JS/Python SDKs do)

SDK → span v2 envelope (type: "span")
└─► ingest-spans (Kafka)
├─► process-spans → buffered-segments
│ └─► process-segments → has_transactions = True ✅
└─► snuba-eap-items-consumer → eap_items_1_local ✅

Impact

On self-hosted Sentry 25.x, any project using the Rust SDK will:

1. Have project.flags.has_transactions = False permanently

2. Show the performance onboarding screen instead of traces
   (the UI gates on this flag before making any API call)

3. Require a manual one-time workaround per project:

   from sentry.models.project import Project
   p = Project.objects.get(slug='your-project')
   p.update(flags=Project.flags.has_transactions.mask | int(p.flags))

Projects using Python or JS SDKs are unaffected as those SDKs already send
span v2 envelopes.

Expected result

Implement the span v2 envelope format in the Rust SDK so spans are sent as:

envelope header: { dsn, sdk }
item header: { "type": "span", "content_type": "application/vnd.sentry.items.span.v2+json" }
payload: [ { trace_id, span_id, name, is_segment: true, ... }, ... ]
Reference: https://develop.sentry.dev/sdk/telemetry/spans/span-protocol/

This is the format already implemented in JS and Python SDKs as part of the
platform-wide transaction→spans migration (tracked internally as SDK-10).

References
Span v2 protocol spec: https://develop.sentry.dev/sdk/telemetry/spans/span-protocol/
Transactions→Spans FAQ: https://sentry.zendesk.com/hc/en-us/articles/40366087871515
Related: RUST-40 (OTel support — different concern but same underlying need)

Actual result

The performance UI at /organizations/{org}/explore/traces/?project={rust-project}
shows the empty onboarding state ("Set Up Tracing") and never queries for traces.

Server logs confirm the UI calls POST /api/0/projects/{org}/{slug}/create-sample-transaction/
(the onboarding "Create Sample Transaction" button) rather than
GET /api/0/organizations/{org}/traces/.

The trace data IS stored correctly in ClickHouse (eap_items_1_local):

• Spans arrive, are processed, and are queryable via the API
• GET /api/0/organizations/{org}/traces/?project={id} returns real traces
• The only broken piece is the project.flags.has_transactions bit, which controls
  the UI gate

[linear-code]

RUST-178

[szokeasaurusrex]

Hi @gravityvi, thanks for opening this issue.

I do not think the missing span v2 support in sentry-rust explains the self-hosted behavior you are seeing. Self-hosted Sentry should continue to support the existing transaction envelope format indefinitely.

Technical background

In 25.9.0, the standard self-hosted deployment still includes the old transaction ingestion path via transactions-consumer, and the Sentry application still sets project.flags.has_transactions from transaction ingestion in event_manager.py. The newer span/segment path also sets the same flag in process_segments/message.py. So the server should not require span v2 in order to ingest performance data or flip has_transactions.

Given that you can already query traces through the API, my current assessment is that this is more likely a self-hosted deployment/configuration problem, or a self-hosted/server bug in how the project flag is updated, rather than an SDK protocol incompatibility.

Next steps

Could you check a few configuration basics on the self-hosted side?

• confirm the DSN in the Rust app points to the correct self-hosted project
• confirm other telemetry, such as errors, also reaches the same Sentry deployment
• confirm transactions-consumer is enabled and healthy
• if you are using the default self-hosted compose setup, confirm the feature-complete profile is enabled so the performance consumers are present
• check whether Relay, Kafka, Snuba, or the transaction consumer show ingestion or processing errors

➡️ Please let us know what you find. If the issue is still reproducible after these checks, we can help investigate further. 🙏

I've opened a separate issue to track the span v2 work, but, as I mentioned, self-hosted should still support the old format.

[gravityvi]

@szokeasaurusrex Thanks for the guidance. We've checked everything you mentioned:

DSN points to the correct self-hosted project
Errors and performance data both arrive at the same deployment — confirmed via
the API and ClickHouse.

Other telemetry (errors) reaches the same Sentry deployment
Error events show up correctly in the issues view.

transactions-consumer is enabled and healthy
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
ingest-consumer ingest-transactions 0 37294368 37294373 5

feature-complete profile is enabled
All performance consumers are present and healthy: transactions-consumer,
snuba-transactions-consumer, process-spans, process-segments, taskworker.

No ingestion or processing errors
Checked Relay, Kafka, Snuba, transactions-consumer, snuba-transactions-consumer,
and worker logs — no errors, drops, or exceptions.

---

What we found

Despite all consumers running cleanly, transactions_local is frozen org-wide
since our upgrade to 25.9.0:

SELECT count(), max(finish_ts) FROM transactions_local;
-- 6,743,298 rows | latest: 2026-02-26

SELECT count() FROM transactions_local WHERE project_id = 18;
-- 0
Data is arriving correctly into EAP storage instead:


SELECT count(), countDistinct(trace_id) FROM eap_items_1_local
WHERE project_id = 18 AND timestamp > now() - INTERVAL 24 HOUR;
-- 402 spans, 388 traces
Kafka topic offsets:


ingest-transactions:            37,294,373 messages  (ingest-consumer lag = 5)
transactions (snuba topic):     30,577,971 messages  (transactions_group lag = 0)
taskworker:                    280,149,858 messages  (taskworker lag = 5)
taskworker-ingest-transactions:          0 messages
Celery queue (Redis):                    0 items
The Celery worker shows no save_event_transaction activity in logs, and the
dedicated taskworker-ingest-transactions Kafka topic is empty. The generic
taskworker topic has 280M messages consumed — so if save_event_transaction
is routing there, it is being processed, but we cannot confirm it is calling
set_project_flag_and_signal("has_transactions") correctly.

We traced the code path that should set has_transactions:

ingest-consumer processes transaction from ingest-transactions
-> dispatches save_event_transaction task
-> calls EventManager.save_transaction_events()
-> calls set_project_flag_and_signal("has_transactions")
-> sets project.flags.has_transactions = True
We can see this code exists in event_manager.py and tasks/store.py. However,
save_event_transaction in 25.9.0 carries a
TaskworkerConfig(namespace=ingest_transactions_tasks). The Celery worker shows
no save_event_transaction activity in logs, the dedicated
taskworker-ingest-transactions Kafka topic has 0 messages, and the Celery queue
in Redis is empty. The generic taskworker topic has 280M messages consumed but
we cannot confirm save_event_transaction is among them or that
set_project_flag_and_signal is being reached.

The freeze date of transactions_local matches our 25.9.0 upgrade exactly.
Before the upgrade, data was flowing into transactions_local normally. After the
upgrade, all transaction data routes to eap_items_1_local instead, and
has_transactions stopped being set for any new project.

Is save_event_transaction still being dispatched and executed in the default
self-hosted 25.9.0 setup, and is it still responsible for setting
has_transactions? Or was this path changed as part of the EAP migration in 25.9.0?

[lcian]

Moving this to self-hosted because this is not a problem specific to the Rust SDK -- most SDKs still send transactions and not standalone spans.
@aldy505 please disregard the misleading comments about the SDK sending transactions instead of standlone spans, as that's the case for most SDKs currently. The problem seems to be that traces are not displaying in the UI for this user.