Skip to content

Experimental alpha-7 CUDA and PhotonVision migration - #2

Closed
Project516 wants to merge 2 commits into
Spectrum3847:mainfrom
Project516:project516/wpilib-2027-alpha7-cuda
Closed

Project516 wants to merge 2 commits into
Spectrum3847:mainfrom
Project516:project516/wpilib-2027-alpha7-cuda

Conversation

@Project516

@Project516 Project516 commented Sep 25, 2026 •

Copy link
Copy Markdown
Member

Summary

  • Moves the Jetson scripts and native bridge to WPILib/AllWPILib 2027.0.0-alpha-7.
  • Carries SpectrumJetson's whole PhotonVision patch series, photonvision-00 through photonvision-73, onto PhotonVision source 1f419c9d (alpha-7) as one patch, patches/photonvision-2027-alpha7-migration.patch.
  • Ports the JNI bridge to alpha-7's org.wpilib AprilTag class and C-compatible WPI_RawFrame headers, without the C++23-only wpi/util/jni_util.hpp. The bridge links no WPILib library, so 04 only fetches the pinned alpha-7 headers.
  • Documents the CUDA 13, Java 25, and hardware validation requirements in docs/CUDA13-MIGRATION.md.

This is PR 2 of 2 for alpha-7. #1 holds the robot-facing version policy.

Rebased onto main at ab6e086 on 2026-10-12, as one commit. Main's new work since 8765dea is ported too.

What changed in this update (2026-10-12)

Rebase conflicts. Main reworked eight files this PR also changes. Each keeps main's new structure with the alpha-7 edits on top:

  • 03-build-photonvision-fork.sh: main now pins Node, Temurin and pnpm with SHA-256s. The alpha-7 build pins Node v24.21.0, Temurin 25.0.4.1+1 and pnpm 11.28.2 the same way, instead of taking the latest Node 24, JDK 25 and pnpm 11.
  • tests/robot-clock, tests/jetson-telemetry: main's unattended versions (bench helpers, deadlines, PASS/FAIL checks) on Java 25.
  • tests/detector-frame-sizes: main's 2560x1600 case and FAR_SEARCH=off switch, with natives from PhotonVision's jar.
  • tests/fake-robot/FakeRobot.java: main's clock-offset options, org.wpilib imports.
  • README.md, 05-build-gpudetector.sh: the alpha-7 text and the refusal, as before.

Patches 62 to 73 ported, cherry-picked in order onto the alpha-7 tree:

  • 62: alpha-7 publishes the control word only as the ControlWord struct, under /DriverStation (not /FMSInfo), and publishes no FMSControlData (DriverStationBackend.MatchDataSender at v2027.0.0-alpha-7). The struct layout is the one 62 decodes. Without this, idle and quiet mode never saw the robot's state on alpha-7. The test publishes to /DriverStation too.
  • 63: the vision loop's catch-all wraps alpha-7's loop, which upstream restructured (cropping, configureFrameProviderForPipeline). VisionModule.stop() stays private as alpha-7 has it, and the pipeline copy uses Avaje instead of JacksonUtils. Snapshot restore keeps the Avaje meta read. The catch-all also exits when the error was caused by an interrupt: alpha-7's FileFrameProvider wraps the stop interrupt in a RuntimeException, and looping on it hung VisionModuleManagerTest forever.
  • 64 to 73 applied without conflicts. None adds JSON types, and the new spectrumGpu field is on the already @Json UIGeneralSettings.

Earlier port notes

Patches 35 to 61 ported. Each was cherry-picked in order onto the alpha-7 tree and resolved against the new base. The main differences:

  • The new base has no Jackson. Patches 40, 45, 46 and 50 to 56, and 60 used ObjectMapper, so they now use Avaje Jsonb (@Json records, Jsonb.instance(), ctx.json). Patch 45 applied cleanly but referenced classes that no longer exist.
  • AprilTagFieldLayout is gone from PhotonVision's config. TagQualityTest (61) uses org.wpilib.fields.Field, AprilTagImageGenerator and Rotation3d.inverse().
  • NT options are constants in alpha-7 (SEND_ALL, KEEP_DUPLICATES, TOPICS_ONLY), and startClient4 is startClient.
  • 42: the base already contains upstream #2617, so its side was kept, along with the patch's ?.resolution guards.

Times are nanoseconds in alpha-7. This fixes Copilot's clock-sync finding and the same bug in two more files. WPI_Now, NetworkTablesJNI.now(), NT value timestamps and cscore grabFrame are all steady-clock nanoseconds (timestamp.cpp, UsbCameraImpl.cpp:583).

  • RobotClockSync: the staleness and rate limits are 5 s and 30 s in ns, and the age converts with / 1_000_000.
  • RewindManager and RewindRecorder convert to microseconds at the source. frames.csv and session.json stay in microseconds, so detector/rewind_reader.h, tools/fieldcal and rewind-export.py are unchanged.
  • An audit of every other WPILib or cscore time in the patch found no other mismatch. toMidExposureNs was already right.

Jar. Shadow 9.0.0 kept dependencies' duplicate LICENSE and META-INF/NOTICE* entries, so unzip -t called the jar a zip bomb, and make-prebuilt-bundle.sh would have refused it. photon-server now excludes those duplicates. Service files still merge.

Main's new scripts and tests on alpha-7:

  • tests/regression, tests/tag-quality, tests/fake-robot and tests/jetson-telemetry use org.wpilib packages, the alpha-7 NT API and Java 25.
  • make-prebuilt-bundle.sh no longer packs libwpiutil, libwpimath and libapriltag, since the alpha-7 detector doesn't link them.
  • install.sh --prebuilt refuses a bundle whose MANIFEST lacks the migration patch, so it won't install a WPILib 2026 release onto this checkout. It installs openjdk-25-jdk.
  • release/NOTICES.md describes the alpha-7 jar and the allwpilib headers.
  • upstream_watch.py watches PhotonVision main against UPSTREAM_SHA and tries the migration patch, instead of the 4143 fork. allwpilib tags compare as alpha, beta, rc and release, so it would have exited on TAG=v2027.0.0-alpha-7.

Earlier Copilot findings, all fixed in this commit:

  • 13 passes -DALLWPILIB_DIR.
  • 04 creates $HOME/build first.
  • 02 rewrites its managed env block, so an existing Java 17 JAVA_HOME is replaced.
  • 05 refuses and points to 07 and 08 bos, since the 4143 detector can't load in the alpha-7 jar.

Validation

GitHub Actions on a temporary fork branch (workflow not in this PR). Run: https://github.com/Project516/SpectrumJetson/actions/runs/38484200349

  • The real host build, scripts/host/03-build-photonvision-fork.sh, applies the patch to 1f419c9d with the pinned tools and produces the linuxarm64 jar. unzip -t passes.
  • ./gradlew test: 332 tests, 0 failures, 2 skipped (photon-core, photon-targeting, photon-lib).
  • 9 of the 10 tests/**/*.java programs compile against the jar. tests/robot-vision-live/LiveCheck.java doesn't: it is compiled against the robot-side jars, not PhotonVision's (see Noticed).
  • Locally: git apply --check of the patch on a clean 1f419c9d, and scripts/ci/lint.sh clean.

Hardware gate

This is still an experimental port. The hosted Java build is not proof for Jetson, CUDA, NVJPEG, TensorRT, JVM, or the robot network. The native detector has not been compiled: alpha-7 needs G++ 14 and C++23, and JetPack 6 ships GCC 11.

Before this is treated as working, test it on the target Jetson with real cameras, native libraries, the robot process, and a joint robot-network run, following docs/CUDA13-MIGRATION.md. Keep this PR as a draft until those results are recorded.

Noticed, not fixed

  • Every change to a numbered photonvision-NN patch on main must also be ported into the migration patch. The doc says so, but nothing checks it. A CI job that applies each stack to its base would catch drift.
  • /api/spectrum/uiState pipeline entries now come from SerializationUtils.objectToHashMap, so enum fields may serialize differently than under Jackson. The UI tests compare entries with each other and find pipelines by nickname, so they don't depend on it, but anything reading enum names there would.
  • robot_us in Rewind's frames.csv stays in microseconds, while alpha-7 PhotonLib result timestamps are nanoseconds. docs/REWIND.md now says to divide by 1000. Whether the robot's AdvantageKit log on alpha-7 is in µs or ns hasn't been checked.
  • The PhotonVision release report on this branch will skip the April Fools v2026.7 once Upstream watch: skip prereleases and record reviewed tags #12 merges and this is rebased.
  • robot-vision builds against both WPILib 2026 and 2027 on its own, and tests/robot-vision-live still runs the robot-side jars on Java 17. Neither is part of this port.

@Project516
Project516 force-pushed the project516/wpilib-2027-alpha7-cuda branch 3 times, most recently from f935fb7 to 0ee2903 Compare September 26, 2026 23:42
@Project516

Copy link
Copy Markdown
Member Author

@allengregoryiv @SpectrumFRC3847 ready for a review. This one is still a draft, because the hardware gate in the body is not met.

Rebased onto main at e9b82ed. Two conflicts, both from main's training site work, and both are described in the updated body:

  • scripts/host/03-build-photonvision-fork.sh: this branch predates the training site bundling and would have silently dropped it. I kept the training/ copy into photon-client/public/docs and took the branch's git fetch --tags "$UPSTREAM_URL".
  • README.md build table row 4: took the alpha-7 jar description and kept main's note about the bundled training site.

patches/photonvision-2027-alpha7-migration.patch is byte-identical to the previous branch tip, so the Actions validation linked in the body still holds. I re-ran bash -n on all 13 changed shell scripts and git diff --check outside patches/, both clean.

What I would like checked:

  • The ported patch series, especially patches 30 to 34 (field calibration 3D view and tuning locks, USB bandwidth reporting and allocation, quieter frame errors, focus score) rewritten to alpha-7 conventions. Those are the least mechanical part of the diff.
  • detector/GpuDetectorJNI.cc against the alpha-7 headers. It merged cleanly with main's CUDA wait block and thread-count switch, so a clean merge is not evidence the port is right.

The known gap, now written into docs/CUDA13-MIGRATION.md and docs/UPSTREAM-PORT.md: the migration patch covers photonvision-00 through photonvision-34. photonvision-35 (purple theme and dino mascot) and photonvision-36 (positive-right yaw sign) landed on main after the patch was cut and are not ported, so this branch builds without them. Porting them is a follow-up, and nothing in CI catches that drift.

Still unproven, and the reason this stays a draft: the native detector has never been compiled. Alpha-7 needs G++ 14 and C++23, JetPack 6 ships GCC 11, and the hosted Java build says nothing about CUDA, NVJPEG, TensorRT, the JVM, or the robot network.

@SpectrumFRC3847

Copy link
Copy Markdown
Contributor

@Project516 merge conflicts

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Unresolved critical build and clock-synchronization defects, plus moderate Jetson setup and compatibility issues, remain.

Review effort: Lite
Findings: 4 High severity · 2 Medium severity

Open (6)
What changed in this PR

Ports the Jetson/PhotonVision CUDA stack to WPILib 2027 alpha-7 and documents the experimental migration requirements.

Changes:

  • Updates Java, NetworkTables, JNI, and native build tooling for alpha-7.
  • Adds the PhotonVision migration patch and CUDA 13 validation guidance.
  • Updates Jetson scripts, tests, documentation, and detector integration.
File Summary
tests/​robot-clock/​run.sh Configurable Java 25 runtime.
tests/​robot-clock/​FakeRobotClock.java Alpha-7 NetworkTables migration.
tests/​jvm-check.sh Configurable Java diagnostics.
tests/​jetson-telemetry/​run.sh Java 25 runtime setup.
tests/​jetson-telemetry/​NtTelemetryDump.java Alpha-7 NetworkTables APIs.
tests/​gpudetector-handles/​run.sh Alpha-7 detector and AprilTag testing.
tests/​detector-frame-sizes/​run.sh Alpha-7 detector runtime setup.
tests/​detector-frame-sizes/​FrameSizeTest.java PhotonVision native loading test.
tests/​cpu-profile.sh Configurable Java tooling.
scripts/​jetson/​13-build-fieldcal-detect.sh Field-calibration detector build.
scripts/​jetson/​07-build-bos-detector.sh Alpha-7 CUDA detector build.
scripts/​jetson/​06-install-fork-jar.sh Alpha-7 jar installation.
scripts/​jetson/​05-build-gpudetector.sh Legacy detector compatibility path.
scripts/​jetson/​04-build-allwpilib.sh Pinned alpha-7 headers.
scripts/​jetson/​02-jetpack.sh Jetson Java environment updates.
scripts/​host/​03-build-photonvision-fork.sh Alpha-7 PhotonVision build.
README.md Experimental migration workflow.
docs/​UPSTREAM-PORT.md Migration history and upstream notes.
docs/​TECHNICAL.md Alpha-7 technical baseline.
docs/​CUDA13-MIGRATION.md Migration contract and validation checklist.
detector/​GpuDetectorJNI.cc Alpha-7 JNI and raw-frame handling.
detector/​CMakeLists.txt Alpha-7 header validation and linkage.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread detector/CMakeLists.txt
Comment thread patches/photonvision-2027-alpha7-migration.patch Outdated
Comment thread scripts/jetson/04-build-allwpilib.sh
Comment thread scripts/jetson/13-build-fieldcal-detect.sh
Comment thread scripts/jetson/02-jetpack.sh Outdated
Comment thread scripts/jetson/05-build-gpudetector.sh Outdated
@Project516
Project516 force-pushed the project516/wpilib-2027-alpha7-cuda branch 2 times, most recently from c1fc7d3 to d6801a8 Compare October 1, 2026 20:09
@Project516
Project516 force-pushed the project516/wpilib-2027-alpha7-cuda branch from d6801a8 to 5385532 Compare October 13, 2026 01:54
FileFrameProvider wraps the stop interrupt in a RuntimeException, which
the photonvision-63 catch-all logged and looped on, so stopProcess
joined forever and VisionModuleManagerTest hung.
@Project516

Copy link
Copy Markdown
Member Author

@allengregoryiv rebased on main and ready for a review. The alpha-7 port now covers photonvision-62 to 73, and the jar builds and PhotonVision's unit tests pass on Actions. It stays a draft until it has run on a Jetson (see Hardware gate). Details are in the description.

@SpectrumFRC3847

Copy link
Copy Markdown
Contributor

@Project516 merge conflicts

@SpectrumFRC3847

Copy link
Copy Markdown
Contributor

we dont plan to update to alpha 7

@Project516
Project516 deleted the project516/wpilib-2027-alpha7-cuda branch October 16, 2026 02:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants