Add initial PS4 ELF project support with a pinned public compiler

CONTRIBUTING.md

@@ -92,6 +92,7 @@ through one file: the connector, `consoles/__init__.py`.
 | `consoles/ps1/` | PlayStation | ROM | `exe.py` the game's program (PS-X EXE) and the disc's BOOT line; its discs and overlays through the PS2's readers · `guides/` |
 | `consoles/ps2/` | PlayStation 2 | ROM | `elf.py` the game's program · `disc.py`, `chd.py` disc images · `overlays.py` code in the disc's other files · `syscalls.py` kernel calls · `symtab.py`, `mdebug.py`, `stabs.py`, `cheader.py`, `debuginfo.py` the game's own debug symbols · `guides/` |
 | `consoles/gc_wii/` | GameCube, Wii | PROJECT | `scratch.py` a dtk-template project for the game · `guides/` |
+| `consoles/ps4/` | PlayStation 4 | PROJECT | `scratch.py` reads decrypted Orbis ELF segments and function maps; `project_template/` builds strict objdiff targets with public clang; `guides/` |
 | `consoles/pc/` | Windows PC | PROJECT | `scratch.py` Ghidra finds the functions, decompdecomp cuts them out of the .exe · `project_template/` · `guides/` |
 | `consoles/nds/` | Nintendo DS | PROJECT | `scratch.py` a ds-decomp project for the ROM, a CodeWarrior version per source file · `project_template/` · `guides/` |
 

README.md

@@ -74,6 +74,9 @@ brings back a window closed while helpers work. Free tools use Windows builds wh
 > GameCube, Wii, DS and Windows PC games, and N64 games built with IDO, don't need WSL: their compilers run on
 > Windows itself.
 
+PS4 projects currently need the Linux app on x86-64 (including inside WSL 2), a decrypted `eboot.elf`,
+and the pinned public Clang 5.0.2 available in Compilers & tools. No Sony SDK or game files are included.
+
 ### macOS (Intel and Apple Silicon)
 
 Install the current source branch, or clone the repository and install the checkout:
@@ -215,6 +218,7 @@ game's code is its rights holder's: keep it to yourself.
 | Nintendo 64 | Recommended |
 | PlayStation (PS1) | Beta |
 | PlayStation 2 | Beta |
+| PlayStation 4 (decrypted ELF, Linux x86-64) | Beta |
 | PSP | Beta |
 | GameCube, Wii | Beta |
 | Windows PC games | Beta |

ROADMAP.md

@@ -15,43 +15,44 @@ Each feature and the platforms it covers today. A — is a gap: pick one up (the
 start). Keep this table true when a feature ships or reaches another platform. Each column's code is its console's
 folder, `src/decompnet/consoles/<id>/`; a new console adds a column (CONTRIBUTING.md, "Consoles").
 
-| Feature | N64 | PS1 | PS2 | PSP | GameCube · Wii | Windows PC | DS | Public decomps |
-| :-- | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: |
-| Status | recommended | beta | beta | beta | beta | beta | beta | the project's own |
-| Start from your own copy | ✓ | ✓ | ✓ | ✓ unencrypted or decrypted program and modules (ISO, CSO, CHD) | ✓ | ✓ | ✓ | fork by address |
-| Judge | link · compare | link · compare | link · compare | link · compare | objdiff | objdiff | objdiff | objdiff |
-| Machine drafts (m2c, Ghidra for x86) | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ Ghidra | — | by architecture |
-| Autopilot: free matches from drafts | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ exact drafts of tiny shapes; Ghidra's made compilable | — | — |
-| Drafts follow the header, made in the background; `redraft` | ✓ | ✓ | ✓ | ✓ | — | ✓ Ghidra's | — | — |
-| C++ methods drafted as `Class::method` | ✓ | ✓ | ✓ | ✓ | — | ✓ (RTTI, and classes from the vtables constructors store) | — | — |
-| Twins: free matches from identical code | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
-| Permuter on near misses | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
-| Hand-written assembly set aside | ✓ | ✓ | ✓ | ✓ | ✓ | Visual C++'s exception-handling stubs | BIOS call stubs, pieces of functions | MIPS, PowerPC, ARM's BIOS stubs and pieces of functions |
-| Pieces of functions joined to their function | — | — | — | — | — | ✓ (`scratch.merge_pieces`) | ✓ (`recover.merge_pieces`) | — |
-| Agents' verdict "no C makes this" (3 agents of 2 models: set aside, red in the map) | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
-| Check again (and before helpers start, once after each update) | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
-| Kernel stubs set aside and named | — | set aside (BIOS calls) | ✓ | set aside, named by library and NID | — | ✓ import thunks | — | — |
-| Function-end reports (`report_boundary`) | ✓ | ✓ | ✓ | ✓ | — | ✓ (agents agree, the rebuild checks) | — | — |
-| Compiler ranges and suggestions | ✓ | ✓ | ✓ | ✓ | from scratch: per source file (`try_compiler`, Settings); else the project's build | per source file (`try_compiler`, Settings) | per source file (`try_compiler`, Settings) | the project's build |
-| Flag search (`search_flags`) | ✓ | ✓ | ✓ | ✓ | the project's build | per source file (/Oy-, /O1, /Ob1) | the project's build | the project's build |
-| Compiler map and the judge's survey | ✓ | ✓ | ✓ | ✓ | — | ✓ per source file, Rich header | — | — |
-| Range checks: a range applies after its neighbours confirm it | ✓ | ✓ | ✓ | ✓ | — | ✓ over source files | — | — |
-| C++ classes from the code (`get_class`) | ✓ | ✓ | ✓ | ✓ | — | ✓ (named by RTTI where the program has it) | — | — |
-| Struct layouts from loads and stores (`guess_struct`) | ✓ | ✓ | ✓ | ✓ | — | ✓ | — | — |
-| The game's own debug symbols: names, types, prototypes | — | — | ✓ symbol table, STABS, DWARF 1; the compiler of each object file | ✓ symbol table, DWARF 1 | — | ✓ PDB (Visual C++ 6.0's PDB 2.0, MSF 7.00), linker map, COFF symbols | — | — |
-| Twins across your games | ✓ | ✓ | ✓ | ✓ | — | ✓ (between PC projects) | — | — |
-| Shared knowledge (two models agree) | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
-| Tidy each source file once it's polished (shared types in one header, names, dead declarations) | — | — | — | — | ✓ | ✓ | ✓ | ✓ |
-| Polish matched code (still matching) | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
-| Real names for the game's symbols (`rename_symbol`) | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | dtk and ds-decomp projects |
-| Save and restore (`progress/`, Open a saved project) | ✓ | ✓ | ✓ | ✓ | the project's git; best attempts saved | the project's git; best attempts saved | the project's git; best attempts saved | the project's git; best attempts saved |
-| Saved README preserves notes, with completion bar and function map | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
-| Reimport: a new split, matched work kept (Settings) | ✓ | ✓ | ✓ | ✓ | — | ✓ (Ghidra again, the newest rules) | — | — |
-| Rebuildable export | ✓ | ✓ | ✓ | ✓ | the project | the project (`ninja rebuild`) | the project | the project |
-| Progress bars for foreground jobs | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
-| Rooms: decompile together | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
+| Feature | N64 | PS1 | PS2 | PSP | PS4 | GameCube · Wii | Windows PC | DS | Public decomps |
+| :-- | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: |
+| Status | recommended | beta | beta | beta | beta | beta | beta | beta | the project's own |
+| Start from your own copy | ✓ | ✓ | ✓ | ✓ unencrypted or decrypted program and modules (ISO, CSO, CHD) | ✓ decrypted ELF | ✓ | ✓ | ✓ | fork by address |
+| Judge | link · compare | link · compare | link · compare | link · compare | objdiff | objdiff | objdiff | objdiff | objdiff |
+| Machine drafts (m2c, Ghidra for x86) | ✓ | ✓ | ✓ | ✓ | ✓ Ghidra | ✓ | ✓ Ghidra | — | by architecture |
+| Autopilot: free matches from drafts | ✓ | ✓ | ✓ | ✓ | — | ✓ | ✓ exact drafts of tiny shapes; Ghidra's made compilable | — | — |
+| Drafts follow the header, made in the background; `redraft` | ✓ | ✓ | ✓ | ✓ | — | — | ✓ Ghidra's | — | — |
+| C++ methods drafted as `Class::method` | ✓ | ✓ | ✓ | ✓ | — | — | ✓ (RTTI, and classes from the vtables constructors store) | — | — |
+| Twins: free matches from identical code | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
+| Permuter on near misses | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
+| Hand-written assembly set aside | ✓ | ✓ | ✓ | ✓ | — | ✓ | Visual C++'s exception-handling stubs | BIOS call stubs, pieces of functions | MIPS, PowerPC, ARM's BIOS stubs and pieces of functions |
+| Pieces of functions joined to their function | — | — | — | — | — | — | ✓ (`scratch.merge_pieces`) | ✓ (`recover.merge_pieces`) | — |
+| Agents' verdict "no C makes this" (3 agents of 2 models: set aside, red in the map) | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
+| Check again (and before helpers start, once after each update) | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
+| Kernel stubs set aside and named | — | set aside (BIOS calls) | ✓ | set aside, named by library and NID | — | — | ✓ import thunks | — | — |
+| Function-end reports (`report_boundary`) | ✓ | ✓ | ✓ | ✓ | — | — | ✓ (agents agree, the rebuild checks) | — | — |
+| Compiler ranges and suggestions | ✓ | ✓ | ✓ | ✓ | the project's build | from scratch: per source file (`try_compiler`, Settings); else the project's build | per source file (`try_compiler`, Settings) | per source file (`try_compiler`, Settings) | the project's build |
+| Flag search (`search_flags`) | ✓ | ✓ | ✓ | ✓ | the project's build | the project's build | per source file (/Oy-, /O1, /Ob1) | the project's build | the project's build |
+| Compiler map and the judge's survey | ✓ | ✓ | ✓ | ✓ | — | — | ✓ per source file, Rich header | — | — |
+| Range checks: a range applies after its neighbours confirm it | ✓ | ✓ | ✓ | ✓ | — | — | ✓ over source files | — | — |
+| C++ classes from the code (`get_class`) | ✓ | ✓ | ✓ | ✓ | — | — | ✓ (named by RTTI where the program has it) | — | — |
+| Struct layouts from loads and stores (`guess_struct`) | ✓ | ✓ | ✓ | ✓ | — | — | ✓ | — | — |
+| The game's own debug symbols: names, types, prototypes | — | — | ✓ symbol table, STABS, DWARF 1; the compiler of each object file | ✓ symbol table, DWARF 1 | — | — | ✓ PDB (Visual C++ 6.0's PDB 2.0, MSF 7.00), linker map, COFF symbols | — | — |
+| Twins across your games | ✓ | ✓ | ✓ | ✓ | — | — | ✓ (between PC projects) | — | — |
+| Shared knowledge (two models agree) | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
+| Tidy each source file once it's polished (shared types in one header, names, dead declarations) | — | — | — | — | — | ✓ | ✓ | ✓ | ✓ |
+| Polish matched code (still matching) | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
+| Real names for the game's symbols (`rename_symbol`) | ✓ | ✓ | ✓ | ✓ | ✓ functions | ✓ | ✓ | ✓ | dtk and ds-decomp projects |
+| Save and restore (`progress/`, Open a saved project) | ✓ | ✓ | ✓ | ✓ | the project's git; best attempts saved | the project's git; best attempts saved | the project's git; best attempts saved | the project's git; best attempts saved | the project's git; best attempts saved |
+| Saved README preserves notes, with completion bar and function map | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
+| Reimport: a new split, matched work kept (Settings) | ✓ | ✓ | ✓ | ✓ | — | — | ✓ (Ghidra again, the newest rules) | — | — |
+| Rebuildable export | ✓ | ✓ | ✓ | ✓ | — | the project | the project (`ninja rebuild`) | the project | the project |
+| Progress bars for foreground jobs | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
+| Rooms: decompile together | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
 
 Where to start on a gap:
+- **PS4**: initial Linux x86-64 support imports decrypted ELF programs and judges functions with strict objdiff. Encrypted SELF/PKG, PRX modules, whole-game relinking/export, compiler switching, source splitting and function-piece joining remain open. The public compiler needs no Sony SDK; fixed imported-global bindings must be configured per game.
 - **PowerPC (GameCube, Wii) and x86 (PC)**: class recovery and the compiler map work on a project's objects in fork
   mode, not on a ROM; the readers are the part to write (CodeWarrior's vtable pointer is often not at offset 0). x86
   has its reader (`arch/x86_fields.py`): PC classes and struct layouts come from it (`Console.classes`,

docs/GUIDE.md

@@ -184,6 +184,7 @@ the game.
 | **Nintendo 64** | ★ **Recommended** | ✓ ROM | KMC GCC 2.7.2 · IDO 5.3 / 7.1 | ✓ export | ✓ |
 | **PlayStation** | Beta | ✓ disc (BIN, ISO, CHD) or PS-X EXE | GCC 2.6.3 · 2.7.2 · 2.8.1 (PSY-Q), with maspsx | ✓ export | ✓ |
 | **PlayStation 2** | Beta | ✓ disc or ELF | EE-GCC 2.9 · 2.95.2/2.95.3 (SN) · 2.96 · 3.2 · CodeWarrior 2.3.3 / 3.0.1 | ✓ export | ✓ |
+| **PlayStation 4** | Beta | ✓ decrypted x86-64 Orbis ELF (Linux x86-64, including WSL 2) | Public Clang 5.0.2 (verify per game) | — | ✓ |
 | **PSP** | Beta | ✓ UMD image (ISO, CSO, CHD) with its other modules, or ELF/PRX, unencrypted or decrypted | CodeWarrior for PSP 3.0.1 · PSP SDK GCC 3.3.3 · SN pspsnc | ✓ export | ✓ |
 | **GameCube · Wii** | Beta | ✓ disc (ISO, RVZ, WBFS…) or DOL | CodeWarrior GC 1.0 – Wii 1.7 | ✓ dtk-template | ✓ |
 | **Windows PC** | Beta | ✓ program (32-bit x86 .exe) | Visual C++ 6.0 · 7.0 · 7.1 · 2005 · 2008 · 2010 | — | ✓ |
@@ -230,6 +231,7 @@ of its own project.
 | :-- | :-- |
 | N64, PS1, PS2, PSP | splat and spimdisasm. They also find code that the game loads later, and overlays. |
 | GameCube, Wii | decomp-toolkit, in a [dtk-template](https://github.com/encounter/dtk-template) project (`consoles/gc_wii/scratch.py`) |
+| PlayStation 4 | Ghidra, or an explicit function map selected with the decrypted executable. Strict objdiff projects; encrypted SELF/PKG, PRX modules and full-game relinking remain planned. |
 | Windows PC | [Ghidra](https://github.com/NationalSecurityAgency/ghidra). decompdecomp cuts each function out of the .exe and the game's own DLLs into a project for Visual C++ and objdiff (`consoles/pc/scratch.py`). |
 
 **C and C++.** GameCube, Wii, PS2 and Windows PC games are often C++. On these platforms, agents can write

projects/demo-ps4/README.md

@@ -0,0 +1,15 @@
+# PS4 importer demo
+
+This builds the repository's CC0 Demo Quest source with the pinned public clang
+5.0.2 and lld. It needs no Sony SDK or game files. The output is a synthetic Orbis
+ELF for importer and matching tests, not a bootable PS4 application.
+
+Install Clang 5.0.2 in Compilers & tools, then run:
+
+```sh
+uv run python projects/demo-ps4/build.py
+```
+
+Open `build/demoquest.elf` in decompdecomp. Keep its adjacent
+`demoquest.functions.jsonl` file: it supplies exact function extents without
+requiring Ghidra. Generated files stay in the ignored `build/` directory.

projects/demo-ps4/build.py

@@ -0,0 +1,57 @@
+"""Build a shareable PS4-shaped ELF from Demo Quest's CC0 source.
+
+The pinned public clang and lld compile/link the demo. Its section table is removed
+and its e_type changed to Orbis EXEC so the importer exercises the loader-header
+path, not a game's symbols. The adjacent map supplies the known function extents.
+This is an importer fixture, not a bootable PS4 application; no SDK is included.
+"""
+from __future__ import annotations
+
+import json
+import struct
+import subprocess
+from pathlib import Path
+
+from elftools.elf.elffile import ELFFile
+
+from decompnet import toolchains
+from decompnet.consoles.ps4.project_template.tools.compiler_flags import CL_FLAGS
+from decompnet.limits import capped
+
+HERE = Path(__file__).resolve().parent
+BUILD = HERE / "build"
+
+
+def main() -> None:
+    """Write build/demo.json and the ELF/map used by console conformance tests."""
+    compiler = toolchains.binary("clang-5.0.2")
+    linker = compiler.with_name("ld.lld")
+    BUILD.mkdir(exist_ok=True)
+    obj, game = BUILD / "game.o", BUILD / "demoquest.elf"
+    source = HERE.parent / "demo-n64" / "original" / "game.c"
+    for cmd in ([str(compiler), *CL_FLAGS, "-c", str(source), "-o", str(obj)],
+                [str(compiler), *CL_FLAGS, "-c", str(HERE / "extras.c"), "-o", str(BUILD / "extras.o")],
+                [str(linker), "-e", "abs_s32", "-Ttext=0x1000", str(obj), str(BUILD / "extras.o"), "-o", str(game)]):
+        subprocess.run(capped(cmd), check=True, timeout=120)
+    with game.open("rb") as file:
+        elf = ELFFile(file)
+        code = next(s for s in elf.iter_segments() if s["p_type"] == "PT_LOAD" and s["p_flags"] & 1)
+        symbols = {s.name: (s["st_value"], s["st_size"])
+                   for s in elf.get_section_by_name(".symtab").iter_symbols()
+                   if s["st_info"]["type"] == "STT_FUNC" and s["st_size"]}
+        rows = [{"a": f"{addr-code['p_vaddr']:x}", "end": f"{addr+size-code['p_vaddr']:x}", "n": name}
+                for name, (addr, size) in sorted(symbols.items(), key=lambda item: item[1][0])]
+    data = bytearray(game.read_bytes())
+    struct.pack_into("<H", data, 16, 0xFE10)
+    struct.pack_into("<Q", data, 40, 0)  # e_shoff
+    struct.pack_into("<HHH", data, 58, 0, 0, 0)  # e_shentsize/e_shnum/e_shstrndx
+    game.write_bytes(data)
+    game.with_suffix(".functions.jsonl").write_text("".join(json.dumps(row)+"\n" for row in rows), encoding="utf-8")
+    (BUILD / "demo.json").write_text(json.dumps({"game": game.name,
+                                               "symbols": {name: addr for name, (addr, _) in symbols.items()}}, indent=2),
+                                     encoding="utf-8")
+    print(f"built {game.name}: {len(rows)} functions; no game data")
+
+
+if __name__ == "__main__":
+    main()

projects/demo-ps4/extras.c

@@ -0,0 +1,4 @@
+/* SPDX-License-Identifier: CC0-1.0
+ * Small independent importer fixtures; no game-derived code. */
+int demo_choose(int a, int b, int choose_a) { return choose_a ? a : b; }
+unsigned demo_sum(unsigned a, unsigned b) { return a + b; }

src/decompnet/app/forkengine.py

@@ -64,8 +64,16 @@ def _path_glob(glob: str) -> str:
 _REGVAR = re.compile(r"\b(?:asm|__asm__|__asm)\s*\(\s*\"\$?[a-z]{1,2}\d{0,2}\"\s*\)", re.I)
 
 
+# A typed extern with an assembler *name* changes linkage only; it emits no assembly.
+# Keep this grammar narrow so instruction strings/extended asm still fail the check.
+_LINKAGE_DECL = re.compile(
+    r'\bextern\s+__typeof__\s*\(\s*[A-Za-z_]\w*\s*\)\s+[A-Za-z_]\w*\s+'
+    r'__asm__\s*\(\s*"[A-Za-z_]\w*"\s*\)\s*'
+    r'__attribute__\s*\(\(\s*visibility\s*\(\s*"hidden"\s*\)\s*\)\)\s*;')
+
+
 def _strip_comments(code: str) -> str:
-    return _REGVAR.sub(" ", re.sub(r"//[^\n]*|/\*.*?\*/", " ", code, flags=re.S))
+    return _REGVAR.sub(" ", _LINKAGE_DECL.sub(" ", re.sub(r"//[^\n]*|/\*.*?\*/", " ", code, flags=re.S)))
 
 
 def _rewrite(path: Path, text: str) -> None:
@@ -2301,7 +2309,7 @@ def _draft(self, proj, fp: ForkProject, cfg: dict, fn, unit) -> str | None:
                 found = ghidra.decompile_object(unit.target, fp.root / STATE / "ghidra")
                 names = [meta.get("symbol") or fn["name"], *self._names(meta, fn)]
                 body = next((found[n] for n in names if n in found), None)
-                text = body and "/* Ghidra */\n" + body.strip()
+                text = body and "/* Ghidra */\n" + consoles.get(cfg.get("platform") or "").adapt_draft(body.strip())
         except OSError:
             text = None
         with self.net.store.tx() as db: