GRU pose encoder + avg-pool ablation arm - #3
Open
bkabbarah wants to merge 4 commits into
Open
Conversation
(cherry picked from commit 73dc799)
…one repo The avg-pool baseline was previously only runnable from the upstream `opentouch` repo while the GRU arm ran from `opentouch-gru`, which made the published 13.43-vs-45.46 comparison a cross-codebase one -- and the repo's own matched re-run of that baseline reads 16.76, not 13.43. "mean" pools over time and zero-pads 128->240 so both modes share one projection of identical shape; the GRU is not constructed in mean mode, so a state_dict from either arm fails loudly rather than half-loading into the other. Adds OpenTouch-DINOv3-B16-Retrieval-AvgPool.json to select it. Tests pin that gru remains the default, that mean is order-invariant while gru is not, and that the zero-pad region is exactly zero. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit f147054)
The mean arm zero-padded its pooled 128-dim vector to 240 so both modes could share a projection shape. Functionally equivalent, but it carried 7,168 dead input columns and, more importantly, made every historical avg-pool checkpoint fail to load with a size mismatch on pose.projection.weight -- which is how it was found, when evaluating the upstream avg-pool run through this codebase. Mean mode now projects from 128 directly, matching upstream, so those checkpoints load unchanged and the same-codebase comparison can be made without retraining. A test builds an upstream-shaped state_dict and asserts a strict load; the obsolete padding test is removed. Note the overnight p2t_scene_avgpool checkpoint was trained under the padded variant and needs commit 618eba4 or earlier to load. Its eval results are already saved to results_retrieval_scene_p2t_scene_avgpool_{val,test}.json. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit f71a25e)
Upstream avg-pool projects the pooled vector directly; the ReLU arrived with the GRU in 73dc799 and was never part of that architecture. Mean mode had inherited it, which both diverged from the published model and handicapped the baseline by zeroing every negative pooled feature. Found by evaluating the upstream avg-pool checkpoint through this codebase: it scored 10.25 mAP against the 16.76 that run originally reported. Two things had to match to reproduce it, the projection width and the absence of this ReLU. This matters beyond checkpoint loading: the overnight scene-disjoint avg-pool arm was trained WITH the ReLU, so that baseline was handicapped and the 5.36x participant-disjoint ratio is measured against too weak an opponent. That arm needs retraining before the ratio is quoted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit 1ab43d4)
bkabbarah
pushed a commit
to bkabbarah/opentouch
that referenced
this pull request
Aug 5, 2026
…os final All three clip-split avg-pool runs (seeds 42/0/1, matched to the GRU arm's seeds, full provenance) trained to epoch 300 exit 0 and bootstrap-evaluated: T->P 16.46 +/- 0.36, P->T 16.07 +/- 0.29. Consistent with the unprovenanced single-run 16.76 it replaces. Final matched 3v3: 2.80x (T->P), 2.84x (P->T). PR OpenTouch-MIT#3 body updated to the seeded table; 2.20's D6 caveat marked resolved. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces average pooling in
PoseEncoderwith a 2-layer bidirectional GRU (readout = concat of final forward and backward hidden states, 240-d).Motivation — a structural asymmetry, not a capacity shortfall. The tactile encoder already used a GRU; the pose encoder averaged all T=20 frames, discarding temporal order. InfoNCE was being asked to align a temporally-aware embedding against a temporally-flat one. This gives both encoders matching temporal structure.
Retrieval results
TEST split, matched 3-seed comparison (seeds 42/0/1 on both arms, identical hyperparameters,
--split-group-by clipstated explicitly):Clip CIs (1000 clip-clustered bootstrap draws, fixed gallery): GRU 45.50 [42.09, 48.65] vs avg-pool per-seed lows ≥ 14.2 — no overlap in any direction on either split. Scene-disjoint holds participants out entirely, so the gain is not clip memorisation; it is larger there. (Scene rows are single-run per arm; clip rows are the seeded headline.)
Two caveats
--split-group-byis not recorded in checkpoints and defaults toclip. A scene-trained model evaluated without it explicitly set scores 72.09 mAP instead of 28.31. Always pass it explicitly when reproducing.What's in the diff
temporal_mode={gru,mean}keeps both arms in one class so the ablation runs from this repo.meanis architecturally faithful to the original — 128-wide projection, no ReLU — so historical avg-pool checkpoints load unchanged and reproduce their numbers. Getting either detail wrong scores an upstream checkpoint at ~10 mAP instead of 16.8.15 tests, including guards that the two modes' state dicts are not interchangeable, that mean mode stays order-invariant while GRU mode does not, and that an upstream-shaped avg-pool
state_dictloads strictly.3 files, +237/−5.