Skip to content

Commit fc73fc1

Browse files
markushiclaude
markushi
and
claude
committed
docs: Add develop-docs with Perfetto profiling pipeline documentation
Introduce a develop-docs folder for internal developer documentation, with a README defining the docs conventions (flat structure, prefixes, naming, images, Mermaid) and a first feature doc covering the Android Perfetto profiling pipeline from the SDK through Relay to the monolith. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 913e479 commit fc73fc1

2 files changed

Lines changed: 310 additions & 0 deletions

File tree

‎develop-docs/README.md‎

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
# Develop Docs
2+
3+
This folder holds internal developer documentation for the Sentry Java/Android SDK:
4+
architecture notes, feature deep-dives, design decisions, and cross-module concepts
5+
that don't belong in the public [Sentry docs](https://docs.sentry.io) or in inline
6+
code comments.
7+
8+
If you are documenting **how** or **why** something works for the people who maintain
9+
this SDK, it goes here. If you are documenting **how to use** the SDK for end users,
10+
it belongs in the public docs instead.
11+
12+
## Rules
13+
14+
These rules keep the docs consistent, easy to navigate, and easy to grep.
15+
16+
### Directory structure
17+
18+
- Use a **flat directory structure**. All documents live directly in `develop-docs/`.
19+
Do not create nested category folders.
20+
- Group and namespace documents with **filename prefixes** instead of folders. Common
21+
prefixes:
22+
- `general-` — cross-cutting topics (e.g. `general-development.md`, `general-architecture.md`)
23+
- `feature-` — a specific SDK feature (e.g. `feature-errors.md`, `feature-profiling.md`)
24+
- `integration-` — a specific integration or module (e.g. `integration-opentelemetry.md`, `integration-spring.md`)
25+
- `platform-` — platform-specific concerns (e.g. `platform-android.md`, `platform-jvm.md`)
26+
- `process-` — team processes and workflows (e.g. `process-release.md`)
27+
28+
Add a new prefix only when an existing one clearly does not fit, and keep the list
29+
above up to date.
30+
31+
### File naming
32+
33+
- File names are **lowercase**.
34+
- Use **dashes** (`-`) as separators, never underscores or spaces. For example, use
35+
`feature-profiling.md`, not `feature_profiling.md` or `Feature Profiling.md`.
36+
- Use the `.md` extension for all documents.
37+
- Choose short, descriptive names. The prefix conveys the category, so the rest of the
38+
name only needs to describe the topic (`feature-session-replay.md`, not
39+
`feature-session-replay-how-it-works.md`).
40+
41+
### Images and other assets
42+
43+
- When a document embeds images (or other binary assets), store them in a **folder with
44+
the same base name** as the document. For example, `feature-profiling.md` keeps its
45+
assets in `feature-profiling/`:
46+
47+
```text
48+
develop-docs/
49+
feature-profiling.md
50+
feature-profiling/
51+
pipeline.png
52+
overview.svg
53+
```
54+
55+
- Reference assets with **relative paths**: `![Profiling pipeline](feature-profiling/pipeline.png)`.
56+
- Asset file names follow the same rules as documents: lowercase, dashes, descriptive.
57+
- Prefer **vector formats** (SVG) for diagrams and screenshots where practical
58+
- Prefer **Mermaid** over a static image whenever a diagram can be expressed as one
59+
(see below) — it lives in the document, is versioned as text, and is easy to update.
60+
61+
### Writing style
62+
63+
- Write in the **present tense** and the **active voice**. Describe how the system
64+
behaves now ("The transport retries failed envelopes"), not how it will or did behave.
65+
- Keep one **top-level `# ` heading** per document (the title), and nest sections with
66+
`##`, `###`, etc. Do not skip heading levels.
67+
- Keep documents focused on a **single topic**. Split large topics into multiple
68+
prefixed documents and link between them rather than growing one giant file.
69+
- Use fenced **code blocks with a language identifier** (```kotlin `,
70+
` ```bash `) so syntax highlighting works.
71+
- Prefer Kotlin snippets over Java.
72+
- When referencing code, link to the file with a **relative path** (e.g.
73+
`../sentry/src/main/java/io/sentry/Sentry.java`) rather than pasting large excerpts
74+
that fall out of date.
75+
- Avoid pinning content to a specific SDK version or date unless it is genuinely
76+
version-specific; keep docs evergreen.
77+
- Cross-link related documents with relative links (e.g. `[profiling](feature-profiling.md)`).
78+
79+
### Diagrams with Mermaid
80+
81+
- Prefer [Mermaid](https://mermaid.js.org/) for diagrams. It renders directly on GitHub
82+
and lives in the document as text, so it versions and reviews like code.
83+
- Embed a Mermaid diagram in a fenced block tagged `mermaid`:
84+
85+
````markdown
86+
```mermaid
87+
flowchart LR
88+
Event[SentryEvent] --> Processor[EventProcessors]
89+
Processor --> Transport
90+
Transport --> Sentry[(Sentry)]
91+
```
92+
````
93+
94+
- For complex diagrams, include a link to the [Mermaid Live Editor](https://mermaid.live/)
95+
so reviewers can iterate quickly.
96+
- Fall back to a static image (stored per the asset rules above) only when a diagram
97+
cannot reasonably be expressed in Mermaid.
98+
99+
## Adding a new document
100+
101+
1. Pick the right prefix (or introduce a new one and document it above).
102+
2. Create `develop-docs/<prefix>-<topic>.md` with a single `# ` title.
103+
3. If it embeds assets, create the matching `develop-docs/<prefix>-<topic>/` folder.
104+
4. Link to it from related documents so it is discoverable.

‎develop-docs/feature-perfetto-profiling.md‎

Lines changed: 206 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,206 @@
1+
# Perfetto profiling on Android
2+
3+
This document describes how continuous profiling works on Android when the SDK
4+
captures traces through the OS-level [`android.os.ProfilingManager`](https://developer.android.com/reference/android/os/ProfilingManager)
5+
API (available on API 35+), and how a captured **profile chunk** flows all the way
6+
from the device to a downloadable profile in Sentry.
7+
8+
## What Perfetto is
9+
10+
[Perfetto](https://perfetto.dev/) is Android's system-wide tracing stack. Its
11+
[stack-sampling profiler](https://perfetto.dev/docs/data-sources/native-heap-profiler)
12+
records periodic samples of native and Java call stacks and writes them to a binary
13+
`.pftrace` trace file (a serialized [Perfetto protobuf](https://perfetto.dev/docs/reference/trace-packet-proto)).
14+
Starting with Android 15, apps can request such traces at
15+
runtime via `ProfilingManager` without root or `adb`, which is what makes on-device
16+
continuous profiling possible.
17+
18+
Useful Perfetto references:
19+
20+
- Perfetto docs: https://perfetto.dev/docs/
21+
- Trace format (`TracePacket` proto): https://perfetto.dev/docs/reference/trace-packet-proto
22+
- Perfetto UI (to open a downloaded `.pftrace`): https://ui.perfetto.dev/
23+
24+
## Pipeline overview
25+
26+
A profile chunk is captured on the device, embedded into a Sentry envelope, expanded and
27+
routed by Relay, and finally symbolicated and stored by the monolith so it can be served
28+
as a flamegraph and downloaded as a raw Perfetto trace.
29+
30+
```mermaid
31+
flowchart TD
32+
subgraph device["Android device — sentry-java"]
33+
PM[android.os.ProfilingManager]
34+
PP[PerfettoProfiler]
35+
PCP[PerfettoContinuousProfiler]
36+
PC[ProfileChunk]
37+
ENV["Envelope item<br/>[JSON metadata][raw .pftrace]<br/>header: meta_length"]
38+
PM --> PP --> PCP --> PC --> ENV
39+
end
40+
41+
subgraph relay["Relay (processing mode)"]
42+
SPLIT[Split payload at meta_length]
43+
CONV[Convert Perfetto → Sample v2]
44+
OS1[Upload raw .pftrace to object store]
45+
KAFKA[["Kafka topic: profiles<br/>ProfileChunkKafkaMessage<br/>(Sample v2 + attachment stored_id)"]]
46+
SPLIT --> CONV --> KAFKA
47+
SPLIT --> OS1
48+
end
49+
50+
subgraph monolith["Monolith — getsentry/sentry"]
51+
TASK[process_profile_task]
52+
SYM[Symbolicate / deobfuscate]
53+
VR[vroomrs: parse + normalize]
54+
OS2[(Object store)]
55+
SNUBA[(Snuba: function metrics)]
56+
DB[(ProfileChunkAttachment row)]
57+
TASK --> SYM --> VR
58+
VR --> OS2
59+
VR --> SNUBA
60+
TASK --> DB
61+
end
62+
63+
ENV -->|envelope| relay
64+
KAFKA --> TASK
65+
OS1 -.stored_id.-> DB
66+
VROOM[getsentry/vroom<br/>serve + merge flamegraphs]
67+
OS2 --> VROOM
68+
SNUBA --> VROOM
69+
```
70+
71+
## SDK (getsentry/sentry-java)
72+
73+
On API 35+, [`AndroidOptionsInitializer`](../sentry-android-core/src/main/java/io/sentry/android/core/AndroidOptionsInitializer.java)
74+
wires up `PerfettoContinuousProfiler` automatically. On older devices the SDK falls back
75+
to the legacy `Debug`-based [`AndroidContinuousProfiler`](../sentry-android-core/src/main/java/io/sentry/android/core/AndroidContinuousProfiler.java),
76+
gated by the `enableLegacyProfiling` option (manifest key
77+
`io.sentry.profiling.enable-legacy-profiling`, defaults to `true`). Only **continuous
78+
profiling** is supported on the Perfetto path — transaction-based and app-start profiling
79+
are not.
80+
81+
### Capturing chunks
82+
83+
Continuous profiling emits a stream of independent [`ProfileChunk`](../sentry/src/main/java/io/sentry/ProfileChunk.java)s
84+
rather than one profile per transaction. `PerfettoContinuousProfiler` drives a chained
85+
loop: each chunk runs for `MAX_CHUNK_DURATION_MILLIS` (60s) via `PerfettoProfiler`, which
86+
calls `ProfilingManager.requestProfiling(PROFILING_TYPE_STACK_SAMPLING, …)` at
87+
`PROFILING_FREQUENCY_HZ` (101 Hz). When a chunk's trace file is ready, a new chunk starts,
88+
so profiling runs continuously.
89+
90+
A chunk keeps a stable `profilerId` across the session and a per-chunk `chunkId`. When the
91+
OS produces the trace file, the profiler builds a `ProfileChunk` tagged with the Perfetto
92+
content type:
93+
94+
```kotlin
95+
ProfileChunk.Builder(profilerId, chunkId, measurements, traceFile, timestamp, ProfileChunk.PLATFORM_ANDROID)
96+
.setContentType(ProfileChunk.CONTENT_TYPE_PERFETTO) // "application/x-perfetto-trace"
97+
.build()
98+
```
99+
100+
The chunk is captured via `scopes.captureProfileChunk(...)` and sent as its own envelope
101+
with item type [`SentryItemType.ProfileChunk`](../sentry/src/main/java/io/sentry/SentryItemType.java)
102+
(wire name `profile_chunk`).
103+
104+
### Envelope format and the `meta_length` header
105+
106+
A legacy chunk base64-encodes its trace into the `ProfileChunk` JSON. A Perfetto chunk is
107+
much larger, so [`SentryClient`](../sentry/src/main/java/io/sentry/SentryClient.java) instead
108+
routes it through the new `SentryEnvelopeItem.fromPerfettoProfileChunk(...)` factory, which
109+
avoids base64 by sending the raw binary alongside the JSON.
110+
111+
The trick is a single envelope **item** whose payload concatenates the JSON metadata and
112+
the raw `.pftrace` bytes with **no delimiter**:
113+
114+
```text
115+
[ProfileChunk JSON bytes][raw .pftrace binary bytes]
116+
```
117+
118+
A new `meta_length` property on the [envelope item header](../sentry/src/main/java/io/sentry/SentryEnvelopeItemHeader.java)
119+
tells the server where the JSON ends and the binary begins. The standard envelope item
120+
structure (header line + newline + payload) is unchanged; `meta_length` simply subdivides
121+
the payload:
122+
123+
```text
124+
{"type":"profile_chunk","content_type":"application/x-perfetto-trace","filename":"…","length":<total>,"meta_length":<json bytes>}
125+
<ProfileChunk JSON><raw perfetto binary>
126+
```
127+
128+
- `length` — total payload size (JSON + binary), as for any envelope item.
129+
- `meta_length` — byte length of the JSON prefix. It is only known after the payload is
130+
serialized, so the header computes it lazily (via a `Callable<Integer>`) and omits the
131+
field entirely for non-Perfetto items, keeping the change backward compatible.
132+
133+
## Relay (getsentry/relay)
134+
135+
Relay processing is gated behind the `organizations:continuous-profiling-perfetto` feature
136+
flag; without it the chunk is dropped. In processing mode Relay:
137+
138+
1. **Splits** the compound item payload at `meta_length` into `(metadata JSON, raw profile)`
139+
and reads `content_type: "perfetto"` from the metadata.
140+
2. **Converts** the binary Perfetto trace into the existing **Sample v2** profile JSON
141+
format (`relay_profiling::expand_perfetto(...)`, backed by a checked-in subset of the
142+
Perfetto protobuf definitions).
143+
3. **Uploads** the raw `.pftrace` blob to object store (usecase `profiles`, keyed per
144+
org/project, with an attachment-retention TTL).
145+
4. **Produces** a `ProfileChunkKafkaMessage` to the `profiles` Kafka topic. The message
146+
carries the expanded Sample v2 JSON as `payload` plus an `attachments` array, where each
147+
attachment records:
148+
- `name` (e.g. `profile.perfetto`),
149+
- `content_type` (e.g. `application/x-perfetto-trace`),
150+
- `stored_id` — the object store key of the uploaded raw blob.
151+
152+
The monolith later uses `stored_id` to fetch the raw trace back.
153+
154+
## Monolith (getsentry/sentry)
155+
156+
A profiling task consumes the `profiles` topic. `process_profile_task` (in
157+
`src/sentry/profiles/task.py`):
158+
159+
1. Unpacks the message and parses the Sample v2 `payload`.
160+
2. Runs **deobfuscation** (Android only for now).
161+
3. Calls `vroomrs` to parse and normalize the chunk
162+
(`vroomrs.profile_chunk_from_json_str(...)`), compresses it, and saves it to the profile
163+
**object store** bucket.
164+
4. Extracts function metrics and emits them to **Snuba** (for querying and flamegraph
165+
metadata).
166+
5. Persists a lightweight **`ProfileChunkAttachment`** DB row for each attachment on the
167+
message — storing `project_id`, `profiler_id`, `chunk_id`, `name`, `content_type`, and
168+
the `stored_id` object store key — so the raw Perfetto trace can be downloaded by ID
169+
rather than exposing the `stored_id` directly.
170+
171+
`getsentry/vroom` is an extra service on top, which reads the stored chunk and Snuba-indexed metadata to serve and merge multiple profile chunks into a single flamegraph. The API endpoint is served by the monolith, but it just passes the request through to vroom.
172+
173+
### Perfetto format dispatch (vroom / vroomrs)
174+
175+
Older Android SDKs emit the legacy Android trace format tagged as a "faulty" `version=2`,
176+
and the pipeline historically keyed off the platform rather than the version. To
177+
distinguish legacy from Sample v2 chunks, `ProfileChunk` carries a dedicated `version`
178+
field, and both `vroom` and `vroomrs` now dispatch on it instead of the platform:
179+
180+
- Version `""` or `2.android-trace` → legacy Android trace format.
181+
- Any other version → Sample v2.
182+
183+
## Downloading a Perfetto profile
184+
185+
The monolith exposes two feature-gated endpoints (both require the
186+
`organizations:continuous-profiling-perfetto` flag):
187+
188+
- **List attachments** — `GET /organizations/{org}/profiling/chunk-attachments/`
189+
(`sentry-api-0-organization-profiling-chunk-attachments`). Requires a `project` and
190+
`profiler_id`; resolves the visible `chunk_id`s (same logic as the flamegraph) and returns
191+
the matching `ProfileChunkAttachment` metadata.
192+
- **Download** — `GET /projects/{org}/{project}/profiling/chunks/{profiler_id}/{chunk_id}/attachments/{attachment_id}/?download`
193+
(`sentry-api-0-project-profiling-chunk-attachment`). The `?download` param is required; it
194+
streams the raw blob back from object store via the stored `stored_id`. Access requires
195+
the org's configured attachments role, analogous to generic event attachments.
196+
197+
In the flamegraph UI, a toolbar button (added for continuous profiles when the flag is on
198+
and at least one attachment exists) lists and downloads these traces.
199+
200+
## References
201+
202+
- SDK: [sentry-java#5251](https://github.com/getsentry/sentry-java/pull/5251) — Android `ProfilingManager` (Perfetto) support
203+
- Relay: [#5659](https://github.com/getsentry/relay/pull/5659), [#5932](https://github.com/getsentry/relay/pull/5932), [#6099](https://github.com/getsentry/relay/pull/6099), [#6102](https://github.com/getsentry/relay/pull/6102) — Perfetto parsing, pipeline, and object-store routing
204+
- vroom: [#672](https://github.com/getsentry/vroom/pull/672) — version dispatch for Android trace profiles
205+
- vroomrs: [#93](https://github.com/getsentry/vroomrs/pull/93) — accept Android profiles in Sample v2 format
206+
- Monolith: [sentry#118029](https://github.com/getsentry/sentry/pull/118029) (chunk attachments + endpoints), [sentry#118071](https://github.com/getsentry/sentry/pull/118071) (flamegraph download button)

0 commit comments

Comments
 (0)