Is the inference-time pooling position correct in the GRU4Rec / BERT4Rec / NextItNet templates? (and could you share one or two Claude Code trajectories?)

[fudan-chen]

Hi, and thank you for open-sourcing RecHarness.

While independently replicating Table 3 (same four Amazon datasets, same
leave-one-out HR@10 protocol), we may have found an inconsistency in three of
the five cold-start templates, and I'd like to ask whether it is intended.

The observation

In all three of gru4rec_perdataset, bert4rec_perdataset,
nextitnet_perdataset, user histories are right-aligned (zero-padded at
the front, latest item at the end) on both the training side and the
inference side. For example, in the predict script generated by
write_predict_script (gru4rec template, train.py around line 116):

seq_arr = [0] * maxlen
for i, item in enumerate(reversed(history[-maxlen:])):
    seq_arr[maxlen - 1 - i] = item      # latest item ends up at the tail

But the user representation is gathered as if sequences were left-aligned:

# gru4rec_perdataset/train.py:194-195
lengths = (seq != 0).sum(dim=1).clamp(min=1) - 1
return out[torch.arange(seq.size(0), device=seq.device), lengths]

# bert4rec_perdataset/train.py:221-222  (nextitnet: 248-249, same pattern)
user_emb = out[0, lengths[0]]

For a history of length L with maxlen M, the real tokens occupy indices
[M-L, M-1], while the code reads index L-1. Whenever L < M/2 the gathered
position falls inside the padding region, so the model is scored from a
padding position. Concretely, with M=200 and L=20, data sits at [180, 199]
but the code reads index 19.

SASRec (log_feats[:, -1, :]) and HSTU (out[0, -1]) take the last position
directly and are consistent with right-aligned sequences, which is why this
looks like an unintended asymmetry rather than a convention.

Also worth noting: encode() is only called from score_candidates(), so
training itself is unaffected — only validation and the final evaluation
read from the wrong position. This makes the bug easy to miss, because
training loss decreases normally.

Measured impact (our replication)

Fixing only the gather position (one line) and re-training/re-scoring the
unmodified template, same evaluator, test AvgHR@10:

backbone	template as shipped	pooling fixed	delta
GRU4Rec	0.2870	0.4771	+0.1901
NextItNet	0.3469	0.4836	+0.1367
BERT4Rec	0.4007	0.4157	+0.0150 (within our run-to-run noise)
SASRec	0.4633	0.4728	+0.0095 (empty control)
HSTU	0.4814	0.4773	−0.0041 (empty control)

The two unaffected backbones move by ±0.01, so we believe the +0.19 / +0.14
on GRU4Rec and NextItNet are attributable to this line. This also implies the
Base numbers reported in the paper for the three affected backbones may be
systematically low, since the paper's Base matches ours closely.

(Caveats: single run each, RTX 3090s rather than A800s, and our replication
replaces the implementer with a different code-editing agent under the same
12h budget — happy to share scripts and configs if useful.)

The request

In our replication, the code-editing agent independently noticed and fixed
this line early in the search (one candidate switched to
pack_padded_sequence with true lengths, another simply took out[:, -1]
with the rationale "histories are left-padded, so the latest item is at the
tail"). A large share of its end-to-end gain on GRU4Rec and NextItNet comes
from that single repair.

Since the paper's Ours arm uses Claude Code as the implementer, it would be
very informative to see whether the same thing happened there: could you
share one or two example trajectories for GRU4Rec or NextItNet — i.e. the
sequence of hypotheses/code edits the agent produced during a run? Even a
condensed log (arm chosen, hypothesis text, resulting val score per round)
would be enough to tell how much of the reported Base→Ours gain is this
repair versus other changes.

Thanks again for releasing the harness — the bandit routing design is the
part we found most interesting to study.