Skip to content

A diff says what still differs once registers, stack slots and order are set aside - #182

Open
silvervest wants to merge 3 commits into
n-delic:mainfrom
silvervest:feat/diff-residue
Open

silvervest wants to merge 3 commits into
n-delic:mainfrom
silvervest:feat/diff-residue

Conversation

@silvervest

@silvervest silvervest commented Nov 2, 2026 •

Copy link
Copy Markdown

Merge order: any time: independent of the other PRs of this set (#178-#185).

Improvement.

Every diff an agent reads (compile_and_diff, last_diff; every console and mode) gets a second line: the target's instructions the candidate lacks and the other way round, among the differing rows, with register names, symbol names, stack slot offsets, branch targets and instruction order set aside; or that nothing is left once they are, which wants another approach (statement order, temporaries, types) than code that's missing.

Of 489 near misses on an x86 game with real differences left, 273 had some past the 60 rows a diff shows, behind rows that differed only by register, and 6 showed none at all.

Test: test_a_diff_says_what_differs_once_registers_and_order_are_set_aside (tests/test_diff_output.py). ruff is clean.

🤖 Generated with Claude Code

@github-actions github-actions Bot added the enhancement New feature or request label Nov 2, 2026

@n-delic n-delic left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. A diff that says what's left once registers, stack slots and order are set aside is a good aid, and the score and match verdict are untouched. But the residue can say there's no difference when there is one, which is the opposite of its purpose:

1. x86 prefixes hide the real opcode (scoring.py:215-216, 218-229). shape() takes only the first word as the mnemonic, so in lock add the opcode add is treated as an operand name and becomes S. Real capstone output: lock add dword ptr [eax], ecx vs lock sub dword ptr [eax], ecx gives residue == ([], []). render_rows then says "Registers, stack slots and order aside, your instructions are the target's", while the summary above says 1 different. Parse lock/rep/repe/repne/xacquire/xrelease/bnd together with the next word, and test a real prefixed pair.

2. Stack-slot normalisation also hides index scales and shift amounts (:204, :224). _NUMBER runs over the whole bracketed operand, so [esp + ecx*4 + 0x18] and [esp + ecx*8 + 0x18] both become [R+R*S+S], with an empty residue. 4 vs 8 is an element-size difference, not a frame offset. ARM's [sp, r2, lsl #2] vs lsl #3 behaves the same way. Replace only the displacement, and test both.

Please also add tests on real-shaped diffs from each arch the residue claims to read (MIPS, PowerPC, ARM, x86).

silvervest pushed a commit to silvervest/decompdecomp that referenced this pull request Nov 3, 2026
… scale and shift

Review of n-delic#182: `lock add` vs `lock sub` came out as no difference (the opcode was read as an operand's name), and
so did `[esp + ecx*4 + 0x18]` vs `ecx*8` and ARM's `[sp, r2, lsl n-delic#2]` vs `lsl n-delic#3` (every number in a stack slot
was its offset). A prefix is read with the word after it, and in a stack slot only the displacement is the
frame's. Tested on capstone's own text for x86, ARM, MIPS and PowerPC.
@silvervest

Copy link
Copy Markdown
Author

Thanks, both fixed in the new last commit: a prefix (lock, rep*, xacquire/xrelease, bnd, notrack) is read with the word after it, and in a stack slot only the displacement is normalised (an index's *4/*8 and ARM's lsl #n stay). The tests run capstone's own text for x86, ARM, MIPS and PowerPC, each with a real difference and a same-but-another-slot pair.

@n-delic n-delic left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. All three points are fixed: the prefixes, _SLOT_NUMBER (scales and shift amounts), and the capstone tests for the four architectures. One small gap before merging:

ARM's shift operators aren't in _KEYWORDS (scoring.py:200-201), so the residue treats them as noise. ldr r0, [sp, r2, lsl #2] vs ldr r0, [sp, r2, asr #2] gives an empty residue, though the instructions differ. Add lsl/lsr/asr/ror/rrx (and the extend ops uxtb/uxth/sxtb/sxth/uxtw/sxtw), with a test on 02019de7 vs 42019de7.

Nick Silvestro added 3 commits November 4, 2026 18:28
…are set aside

Every diff an agent reads (compile_and_diff, last_diff, every console and mode) now has a second line: the target's
instructions the candidate lacks and the other way round, among the differing rows, with register names, symbol
names, stack slot offsets, branch targets and instruction order set aside; or that nothing is left once they are,
which wants another approach (statement and declaration order, temporaries, types) than code that's missing.

Why: of 489 near misses on an Xbox game with real differences left, 273 had some of them past the 60 rows a diff
shows, behind rows that only differed by register, and 6 showed none of them at all; and a quarter of the near
misses on two Xbox games differed by registers and order alone, which the counts line didn't say.
… scale and shift

Review of n-delic#182: `lock add` vs `lock sub` came out as no difference (the opcode was read as an operand's name), and
so did `[esp + ecx*4 + 0x18]` vs `ecx*8` and ARM's `[sp, r2, lsl n-delic#2]` vs `lsl n-delic#3` (every number in a stack slot
was its offset). A prefix is read with the word after it, and in a stack slot only the displacement is the
frame's. Tested on capstone's own text for x86, ARM, MIPS and PowerPC.
`ldr r0, [sp, r2, lsl n-delic#2]` against `asr n-delic#2` came out as no difference: `lsl` and `asr` were names, set aside like
registers. ARM's shifts (lsl, lsr, asr, ror, rrx) and extends (uxtb ... sxtx) are now keywords, and an extend's
amount in a stack slot stays like a shift's. Tested on capstone's text for ARM (lsl vs asr) and ARM64 (uxtw vs sxtw).
@silvervest

Copy link
Copy Markdown
Author

Thanks, fixed in the new last commit: ARM's shifts (lsl/lsr/asr/ror/rrx) and extends (uxtb…sxtx) are keywords, and an extend's amount in a stack slot stays like a shift's. Tests: 02019de7 vs 42019de7, plus ARM64's uxtw #2 vs sxtw #2. Both fail without the change. Rebased onto the current main.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants