Repository navigation
Conversation
|
bugbot run |
e5b773e to
0cc15fb
Compare
|
bugbot run |
0cc15fb to
d53ea8b
Compare
|
bugbot run |
There was a problem hiding this comment.
Code Review
This pull request introduces the capability to read training samples from MCAP files via read_mcap(samples=...), adding new modules for sample row processing, task sizing, and topic specifications, alongside corresponding unit and integration tests. The code review identified several critical compatibility and runtime issues: using the key argument in bisect.bisect_right will break on Python 3.9; calling files.value_lengths() and anchor.fill_null(0) will raise AttributeErrors in PyArrow; and accessing self._samples.nbytes is incompatible with PyArrow versions older than 14.0.0. Code suggestions were provided to resolve these issues using backward-compatible PyArrow and standard library alternatives.
d53ea8b to
a334086
Compare
|
bugbot run |
a334086 to
dc72fcb
Compare
|
bugbot run |
11609d7 to
36c32a1
Compare
aa1215b to
921ed60
Compare
921ed60 to
58c4eed
Compare
58c4eed to
da721f4
Compare
da721f4 to
c38003f
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit c38003f. Configure here.
| ) | ||
| if not_specs: | ||
| raise ValueError(f"topics must map each topic to a TopicSpec: {not_specs}") | ||
| probe = _first_samples(samples, PROBE_SAMPLES) |
There was a problem hiding this comment.
When the samples dataset ends in a sort, as the docstring suggests ("sorted by file and anchor"), we sort twice: once at planning to probe the first 256 samples, and again in the read. A sort has to see every sample before it outputs any, so this is expensive.
There was a problem hiding this comment.
Fixed in dfd9423. The docstring now says to sort the samples, then call .materialize(). Planning reads the first samples to size the read tasks, so an unmaterialized sort would run twice.
There was a problem hiding this comment.
OK for this PR. Since #67991 reworks the sizing anyway, could it also stop running the samples at planning? With only the docstring, anyone who skips .materialize() still runs the samples twice.
c38003f to
1b8bbb2
Compare
1b8bbb2 to
dfd9423
Compare
dfd9423 to
82d9c2f
Compare
82d9c2f to
df4578a
Compare
7ce43bd to
470185c
Compare
Problem:
- A sample row needs each topic's messages around its anchor, read
from the recording's chunks. Reading each sample on its own reads
the same chunks many times when samples lie close together.
- The chunks a task holds at once must stay within its budget.
Change:
- mcap_sample_rows: a read task takes its samples file by file, in
anchor order, and groups samples whose windows meet into runs. A
run ends before its chunks would hold more than a read task's
budget (RAY_DATA_MCAP_BIN_PACKING_BYTES). Each run reads the chunks
its topics' steps reach once (ChunkTimeline), and forgets chunks
the next run does not need.
- A file without a chunk index is scanned once (WholeFileTimeline).
The scan keeps only what steps can take: the runs' spans, the
nearest messages around each span, and the first and last messages
inside each sample's bounds.
- A corrupt chunk fails the read task. Ray's max_errored_blocks then
decides whether the task's block is dropped or the read fails.
- SampleRows builds one row per sample: the sample's columns, then each
topic's data:, times: and is_pad: lists. A row over
RAY_DATA_MCAP_MAX_ROW_BYTES fails the read, naming the sample.
- mcap_summary.topic_chunks: the chunks that may hold a topic, one rule
for the reader and, later, for sizing tasks.
- MCAPDatasourceV2(topics={topic: TopicSpec}, samples_schema=...) sets
up a sample read: its schema is the sample row's, and its reader
builds sample rows from manifest rows that name each sample's file.
read_mcap wires it up in the next PR.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- Robot learning reads examples, not messages: a clip of a recording,
or a moment inside one, with each topic read over its own window.
- The stack below adds the spec, step resolution, sample checks and
the sample reader; nothing reaches them from read_mcap yet.
Change:
- read_mcap(samples=ds, topics={topic: TopicSpec(...)}) reads one row
per sample. The samples feed ReadFiles as its manifest; planning
lists no files. Options that only a read of paths takes are refused.
- Planning runs the samples once: their schema comes from their plan
when it is known, else from the samples probed for task size.
- A task takes as many samples as read about
RAY_DATA_MCAP_BIN_PACKING_BYTES of chunks, estimated from the first
256 samples (mcap_sample_tasks).
- A checkpoint keyed on a sample column skips finished samples before
their files are opened: ReadFiles(checkpoint_on_input=True), a
checkpoint filter only at the read nearest the sink, and ReadFiles
skipping empty manifest blocks. A checkpoint keyed on a column the
samples lack, such as row_id, is refused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>

How to review
Start with
_read_mcap_samplesinread_api.py: it is the whole path from a samples dataset toReadFiles. The planner and checkpoint changes are small and only matter for resume.read_api.pyread_mcap(samples=...): checks, task sizing, the manifest map.mcap_sample_tasks.pyplanner.py,plan_read_files_op.py,checkpoint/plan_read_files_op.py,read_operator.pytest_mcap.py,test_mcap_resume.py,test_mcap_sample_tasks.py,BUILD.bazelProblem.
Change. A table of samples goes in; one row per sample comes out.
TopicSpec,stepsandmatchcome from the PRs below; this PR connects them toread_mcap.ReadFilesas its manifest; planning lists no files. A task takes as many samples as read aboutRAY_DATA_MCAP_BIN_PACKING_BYTESof chunks, estimated from the first 256. The estimate uses the reader's reach rule (TopicReach, [Data] read_mcap: read sample rows from a file's chunks #67927). Samples whose windows meet or touch share one coalesced read (see the reader PR below).local://files read on the driver's node, and fail atread_mcapunder Ray Client, as a read oflocal://paths does.preserve_order. The docstring says so, and says to match rows to samples by joining on a sample column such assample_id.ReadFiles(checkpoint_on_input=True)asks for this. The planner puts the filter only on the read nearest the sink, ahead of theStreamingRepartitionthat cuts the read's input into tasks.ReadFilesskips empty manifest blocks. A resumed run plans:read_mcaprefuses a checkpoint whoseid_columnthe samples lack, such asrow_id. The error suggests a column that names each sample.Measured on S3: 1,024 one-minute recordings (115 GB), 2 s back-to-back samples against the same windows read by the earlier
WindowSpecdesign, 128 CPUs, raw payloads:The remaining gap is planning, mostly the probe that sizes the tasks. Planning runs the samples once: their schema comes from their plan when it is known, else from the samples probed for task size.
Decisions to confirm:
checkpoint_on_inputas an explicitReadFilesflag (core Ray Data change; reviewers for the checkpoint planner please weigh in).Related issues
read_mcap.Additional information
483 passed, 23 skipped; test_checkpoint.py: 97 passed, 17 skipped. pre-commit and pyrefly clean.
AI assistance (Claude Code) was used to draft this change; I reviewed every line and ran the tests above locally.
🤖 Generated with Claude Code