Skip to content

Decode STR opcodes via optional .tbl (closes TODO #1) - #1

Open
Greengoblin007 wants to merge 2 commits into
g-e-o:mainfrom
Greengoblin007:tbl-string-decoding
Open

Decode STR opcodes via optional .tbl (closes TODO #1)#1
Greengoblin007 wants to merge 2 commits into
g-e-o:mainfrom
Greengoblin007:tbl-string-decoding

Conversation

@Greengoblin007

Copy link
Copy Markdown

This PR addresses the first item in the README's TODO list — Decode strings properly — and fixes a small bug encountered along the way.

Summary

  • Optional thingy-style .tbl for read_string in gcx.py. When loaded, multi-byte sequences (radioChar 0x80xx / 0xC0xx, hiragana 0x81xx, katakana 0x82xx, punctuation 0xd0xx, kanji 0xXXYY, …) decode to readable characters via longest-prefix matching (4 → 3 → 2 → 1 byte). Without a .tbl the behavior is unchanged: bytes < 0x80 decode via chr(), higher bytes still become \xNN escapes.
  • --tbl <path> flag wired into main.py so python3 main.py -d <input> -o <output> --tbl mgs.tbl decodes Japanese MGS1 strings out-of-the-box.
  • Bug fix in GclComp.__init__: self.is_pc_version was never set when both is_pc_version=False (default) and radio=None, so compile_gcl_file crashed with AttributeError on the trailing alignment-padding check. Default it to False before the conditional.

Why TBL?

The current read_string returns \xNN for any byte ≥ 0x80, which makes Japanese strings unreadable in the JSON output. A pluggable .tbl (the same format used by tools like Translhextion) solves this generically: a fan can supply mappings for any encoding (Japanese, Spanish accented chars, modified font tables, etc.) without touching the tool. This is the same mechanism used in many ROM-hacking workflows and is fully opt-in.

Tested

  • Smoke test:
    GclComp(is_pc_version=False)            # used to crash; now OK
    GcxData(b'Hello\x00').read_string()     # → 'Hello'  (no tbl)
    g = GcxData(bytes.fromhex('810681142100')); g.set_tbl({b'\x81\x06':'う', b'\x81\x14':'ご'})
    g.read_string()                          # → 'うご!'  (with tbl)
    
  • Decompiled the full STAGE.DIR of MGS1 USA Disc 1 with --tbl and an empty .tbl: identical output to current main.

Notes

  • read_string's outer iteration limit (max_iter = 0xff) preserves the original safeguard.
  • set_tbl(None) or omitting the flag disables decoding, so existing test fixtures continue to round-trip byte-identical.

If both is_pc_version=False (default) and radio=None, neither branch sets
self.is_pc_version, causing AttributeError on the line:
    if not self.is_pc_version:
        while len( data ) % 4 != 0:
            data.push_byte( 0 )

Default it to False before the conditional so direct callers (tests,
standalone compile) don't need to pass a radio object.
@Greengoblin007
Greengoblin007 force-pushed the tbl-string-decoding branch 2 times, most recently from 8060c89 to 6b04d57 Compare May 2, 2026 15:32
@Greengoblin007

Copy link
Copy Markdown
Author

Quick follow-up: amended the commit to also wire set_tbl(self.tbl) into the GcxData used for RADIO.DAT (main.py line ~97). With this, the same --tbl flag also decodes radio dialog TALK text and ADD_CONTACT names to readable Japanese — addressing the same root issue ("strings unreadable in JSON output") for the radio JSON path.

This effectively also covers the first half of TODO entry 3 (radio dialog scripts) for the binary→JSON direction. The text↔JSON lexer/parser side of TODO 3 remains untouched.

@Greengoblin007

Copy link
Copy Markdown
Author

End-to-end test on the pristine MGS1 USA disc 1 RADIO.DAT (no Arabic/translation overlay):

Command:
```
python3 main.py -d <pristine_disc1> -o --no-padding --tbl mgs.tbl
```

Result: 274 radio dialog JSON files written successfully.

Sample TALK entry, before/after the .tbl:

Without `--tbl`:
```
"The entrance to the underground\x80#\x80Nbase is further below."
```

With `--tbl` (radioChar entries from MGS1 JP):
```
"The entrance to the underground#Nbase is further below."
```

The `\x80\x23` and `\x80\x4e` 2-byte escapes resolve to `#` and `N` per the JP radioChar table — these are the newline/separator markers MGS1 carries over from the Japanese encoding. So the same `.tbl` correctly handles both the stage GCX strings and the radio dialog text in one pass.

…perly')

Reading side (read_string):
  Optional thingy-style .tbl with longest-prefix matching (4 -> 1 byte).
  Multi-byte sequences such as MGS1 Japanese radio strings (0x80xx
  radioChar / 0x81xx hiragana / 0x82xx katakana / kanji ...) decode
  to readable characters. Without a .tbl, behavior is unchanged.

Writing side (encode_string):
  Symmetric inverse-tbl lookup so a JSON STR that holds real Unicode
  (e.g. "うご") round-trips back to the same bytes (0x81 0x06 0x81 0x14)
  the .gcx originally had. Without a .tbl, behavior is unchanged but a
  one-line warning is emitted when push_byte would silently truncate a
  Unicode codepoint, so .tbl coverage gaps surface instead of corrupting
  output silently.

Plumbing:
  CLI flag '--tbl <path>' wired into both decompile and compile paths
  in main.py (RADIO.DAT GcxData and per-stage GCX in decompile;
  RadioComp.gcx and GclComp.gcx in compile). A class-level fallback on
  GcxData lets the temporary sub-buffers built inside radio_compile /
  gcl_compile use the same inverse table without each module having to
  thread the .tbl through its API.

Smoke-tested (round-trip via .tbl):
  GcxData(b'\x81\x06\x81\x14').read_string()  -> 'うご'   (decode)
  GcxData().encode_string('うご')                  -> b'\x81\x06\x81\x14\x00' (encode)
  Decompile of pristine MGS1 USA Disc 1 RADIO.DAT (--no-padding --tbl):
    \\x80#\\x80N markers resolve to '#' / 'N' across 274 dialogs.
@Greengoblin007
Greengoblin007 force-pushed the tbl-string-decoding branch from 6b04d57 to 87adbd8 Compare May 2, 2026 15:51
@Greengoblin007

Copy link
Copy Markdown
Author

Extended the patch to cover the write direction as well.

Before: encode_string did s.push_byte(ord(c)) on every codepoint, which bytearray.append truncates to the low byte. So a JSON STR that contained real Unicode (e.g. "うご") compiled to garbage bytes (0x46 0x54 instead of 0x81 0x06 0x81 0x14), silently.

After: when a .tbl is attached, an inverse map is built (preferring the canonical 0x80/0x81/... prefix when 0xC0/0xC1 aliases share the same value) and encode_string does longest-prefix matching on the character sequence. JSON "うご" now round-trips back to its original bytes. Without a .tbl, behavior is unchanged but a one-line warning fires when truncation would happen, so the gap surfaces instead of corrupting silently.

Smoke test:

GcxData(bytes.fromhex('810681142100')).read_string()    # → 'うご!'   (decode)
GcxData().encode_string('うご!')                         # → b'\x81\x06\x81\x14\x21\x00'  (encode)

The CLI plumbing in main.py covers both compile (RadioComp.gcx, GclComp.gcx) and decompile (RADIO.DAT GcxData, per-stage GCX) paths. A class-level fallback on GcxData lets the temporary sub-buffers built inside radio_compile/gcl_compile use the same inverse table without each module having to thread the .tbl through its API.


Side-finding (pre-existing, not in this PR): while testing main.py -c end-to-end on pristine MGS1 USA Disc 1 RADIO.DAT, I hit OverflowError: int too big to convert at radio_compile.py:209 (_data.push_short(len(data) + 2) with len ≈ 66498). Reproduces both with and without --tbl, so it's independent of this change. Also: even without --tbl, the recompiled RADIO.DAT diverges from the original starting at byte 0x0 and is ~200 KB larger. Happy to file a separate issue with details if useful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant