Dreamcast platform notes — a checklist, now with a second disc

A running checklist, to be carried from one Dreamcast documentation pipeline to the next and added to by each.

It covers two discs. Sonic Adventure (USA), MK-51000, 1999, opened in full at dc-sonicadventure-doc, and Dino Crisis (Italy), T7019D 13, Capcom, 2000, at dc-dinocrisis-doc. This document was written before the first pipeline rather than out of it, which is the opposite of how the CD-i and Amiga CD checklists were built.

What the second disc did to it. Promotion is the boring outcome and this document says so in its own procedure section. Counted from the diff:

eight [1 of 1] marks promoted to [2 of 2], and sixteen [2 of 2] marks in total once the claims the first disc had not marked are counted;
one claim falsified and rewritten — section 2 point 3, hash the bootstrap region, which was the highest-value instruction in the file and is now [2 of 2] saying the opposite: diff it, do not hash it;
one [0 of 2] — section 9's "MPEG audio on stream 0xC0", which is wrong about both discs and was inherited from a tool's own census line;
two corrections in place, with the wrong version quoted above each;
two columns added to section 10 that the first row asked for and could not fill, plus a second row that makes four of the columns discriminate;
six of the six original open questions marked, and six new ones left.

Every [2 of 2] below is a hypothesis with a sample of two, not a law. The third disc's job is to break them, and the most useful thing the second one did was break something rather than agree with it.

This document opened with twenty-one [unverified] marks: three in the procedure sections, which are about the mark itself, and eighteen on claims. Of those eighteen the first disc

exercised and converted sixteen to [1 of 1];
corrected five of them in place, with the wrong version quoted above the correction so the history stays attached — sections 3, 4, 5, 6 and 7;
left two alone, because it did not test them: the SH-4 load address and vector unit in section 5, and the tile-based deferred renderer in section 7. Neither can be tested without disassembling 6,751,168 bytes of SH-4, which this pipeline did not do.

A mark still reading [unverified] means what it always meant: this came from public documentation of the platform, not from a disc that was opened. Two of the eight remaining occurrences are inside the quoted wrong versions of corrections, where they belong.

The corrections are the most useful lines in this file, and the first of them is the sentence section 5 used to call the single most important line in the document. The first retail disc that ever tested it falsified it.

What the first pipeline does to this document

The first disc converts marks. For each claim it exercises:

confirm it and re-mark it [1 of 1], with the measurement in the text;
correct it and say so in place, keeping the wrong version visible — a corrected claim with its history attached is worth more than a claim that was always right;
delete it, if the disc shows it was never true of this platform.

Claims the first disc does not exercise stay [unverified]. Do not promote a mark you did not test. The CD-i document's most useful sections are the ones where a later disc falsified an earlier one, and none of that works if [unverified] quietly becomes [all] because nothing contradicted it.

This platform needs the warning more than most. It is the best-documented machine in this collection by a distance — there are homebrew SDKs, emulator sources and twenty-five years of scene notes — and the temptation is to inherit all of it as fact. Inherited facts are exactly what the other two checklists exist to falsify: on Amiga CD, four widely repeated claims about the .TM block turned out to be wrong, and each of them was wrong because everyone had copied it from everyone rather than opening a disc.

When the platform has enough real discs behind it that the apology no longer means anything, delete this section and the preamble above. It has one, and one is not yet enough: almost everything interesting in the list below is a question that needs a second disc to answer, and the first one's job was to leave the measurements where the second can falsify them.

1. The dump is not the disc — read the track layout first

[1 of 1] This is the section with the most ways to go wrong, and it comes first for that reason. Sonic Adventure arrived as a three-track BIN/CUE: MODE1/2352 14,564 sectors, AUDIO 1,063 sectors, MODE1/2352 504,150 sectors, 1,222,515,861 bytes over four files, every track dividing by 2,352 with remainder zero. That form keeps the sync pattern, the header address, the EDC and both ECC parities, and all of them were checked: 518,714 data sectors of 518,714 correct on every one of the five tests. It is the first object in this collection whose dump made that measurement possible at all.

A Dreamcast title ships on GD-ROM, a Sega/Yamaha format holding roughly 1.2 GB by packing the pits more tightly than a CD. The disc has two areas:

a low-density area, readable by an ordinary CD drive, normally carrying a small data track and an audio track — often the spoken and printed warning about playing the disc in a CD player;
a high-density area, where the game actually lives, and which an ordinary drive cannot read at all.

[2 of 2] The high-density area is addressed from LBA 45000, and on the first disc to test it the base is asserted three times independently and the three agree:

every extent in every directory record resolves to a sensible file offset if and only if 45,000 is subtracted — 2,826 files, all inside the track;
the volume descriptor declares 504,150 sectors, which is the whole track, remainder zero;
the CD sector headers say so themselves. The MSF address at offset 12..14 of track 3's first sector reads 10 02 00 = 45,150 frames = LBA 45,000, and header LBA − file index is 45,000 on 504,150 of 504,150 sectors (and 0 on 14,564 of 14,564 on track 1).

The third is the one worth carrying: it is written by the CD frame layer rather than by the filesystem, so it is an independent witness, and it costs one pass over the image.

And a trap that only shows up past 99 minutes. The MSF minutes field is packed BCD, two digits, so it runs out at 99:59:74 = 449,999 frames — which on a GD-ROM is only four fifths of the way through the high-density area. Past that the tens nibble keeps counting into a, b, c. On Sonic Adventure that is 99,300 sectors of 504,150, and a validator that requires BCD reports them all as corrupt. Read the minute as (byte >> 4) * 10 + (byte & 15) with the high nibble allowed above 9 and all 504,150 agree.

Dumps arrive in several shapes, and each loses something different:

.gdi — a track list naming one file per track, with the LBA and mode of each. The most faithful of the common forms, and the one that makes the layout explicit.
.cdi — a single-file container. Convenient, and it hides the track structure behind a parser.
.chd — compressed, and increasingly the archival form. Track metadata is preserved; read it out rather than assuming.
a bare .iso of the data track — the high-density file system and nothing else. Everything in sections 1 and 2 of this document is missing from a dump of this kind, and a surprising amount of published analysis is built on one without saying so.

Record what you were given before you record anything about the game. Track count, mode and LBA of each; which area each track belongs to; total bytes; and what the dump form cannot represent.

2. `IP.BIN` — the header, and the first thing to read

[2 of 2] The first sixteen sectors — 0x8000 bytes — of the high-density data track are the initial program, universally called IP.BIN. Confirmed on Sonic Adventure: 32,768 bytes at absolute LBA 45,000, sha1 c8bbf86ce125dfc716137a77988500fe56e8bb3c, with the field layout below exact to the byte and 42.82 % of the bytes after the 256-byte header non-zero. Confirmed again on Dino Crisis: 32,768 bytes at the same place, sha1 c27cdf06e2aca267806b3420fa88ff9e030c9811, same layout, same field offsets. It is not part of the ISO 9660 file system; it sits in front of it, which makes it this platform's equivalent of the CD-i pre-file-system region and the Amiga CD .TM block. On both of those platforms, the bytes in front of the file system turned out to be the highest-yield first move on the disc.

Expect a fixed-layout header at the start of it:

Hardware ID          "SEGA SEGAKATANA"
Maker ID             "SEGA ENTERPRISES"
Device information   disc number of total, and a checksum of the product fields
Area symbols         which of J / U / E the disc admits to
Peripherals          the controllers and accessories the title declares
Product number       e.g. HDR-nnnn / T-nnnnn
Product version      e.g. V1.000
Release date         YYYYMMDD
Boot filename        normally 1ST_READ.BIN
Company name         the publisher, as the disc spells it
Software name        the title, as the disc spells it

followed by the licence-screen bootstrap code and logo data that fill the rest of the 0x8000 bytes.

[1 of 1] on points 1 and 2; points 3 and 4 not tested. Four things to do with it immediately:

Compare every field against reality. The release date against the file timestamps; the area symbols against what the disc actually contains; the declared peripherals against what the code reads. A header that disagrees with its own disc is a finding.
Verify the device-information checksum. It is computed over the product number and version. If it fails, say so — you are probably looking at a modified image. [2 of 2] It is a CRC-16/CCITT-FALSE — polynomial 0x1021, initial value 0xFFFF, no final xor — over the sixteen bytes at +64, product number and version concatenated. On Sonic Adventure, CRC16(b'MK-51000 V1.004') = 0x2BB6; on Dino Crisis, CRC16(b'T7019D 13V1.000') = 0xB987. Both are exactly the four hex characters at the head of the device-information field. An algorithm derived from one disc reproduced on a second, from a different publisher, a year later, without adjustment. Three single-character mutations give C095, 3B97 and 4278, so the check has teeth. Point 1 also fired: the area field says U and the disc carries mission text and speech in five languages including Japanese, and the release date 19990812 is ten days after the latest build stamp inside the executable. A header that disagrees with its own disc is a finding, and this one does.
CORRECTED [2 of 2], and the correction is that hashing is the wrong instruction. The version this document opened with, and which the first disc converted unchanged, read:

Hash the bootstrap and logo region across discs. This is a shared, pressed-verbatim asset handed to every developer, which is precisely the shape of the CD-i bumper result. Expect a small number of generations; the interesting outcome is a disc that carries an odd one.

The prediction is right and the method is wrong. The two hashes differ — ecc407ca... against abb6b44c... — and a pipeline that stops there concludes the region is not shared. It is shared: 32,411 of 32,512 bytes agree, 99.6893 %, in four patches totalling 101 bytes:
- an eight-slot region table at image offset 0x3700, each slot 32 bytes of bra .+32 ; nop followed by 28 bytes of text, of which exactly one slot is filled — slot 1 on the U disc and slot 2 on the E disc, matching the position of the non-blank character in the area-symbols field;
- one library version stamp, AIP Ver 1.07 Build:Apr 27 1999 14:00:30 against AIP Ver 1.03 Build:Jan 21 1999 17:12:59 — and 38 of the 39 printable runs in the two images are byte-identical, including syTmr Version: 0.51, its build stamp, SCC80831 and INI90405;
- one byte at image offset 0x60C0, 0 on the European disc and 1 on the American one, not derived;
- one pointer whose low byte moved four bytes because the AIP string moved four bytes.
So: diff the region, do not hash it. A hash can prove sameness and cannot measure difference, and the difference is where the platform is. The instruction is now: cook the first sixteen sectors of both discs and diff them byte by byte, reporting agreement, first difference and a map of the differing runs. dc-dinocrisis-doc/tools/ipdiff.py does it in one command.

What a third disc settles in ten seconds: the region table has six slots never seen filled. A Japanese disc should fill slot 0 and confirm the indexing, or fill something else and break it.
Read the peripheral bits properly. They are a declaration, not a measurement — a title that declares the VMU and never touches it is worth a sentence, and only a call-site count can tell you which it is.

3. ISO 9660 on the high-density area

CORRECTED [1 of 1]. The version this document opened with read:

[unverified] Unlike the 3DO, this platform uses ISO 9660, so ordinary tools work — which is a convenience and a trap in equal measure. The convenience is that isoinfo and friends will list the volume. The trap is that they will list it from the start of the track, and every LBA they report needs the high-density base added back before it means anything on the disc.

Ordinary tools do not work, and the way they fail is worse than not working. The volume descriptors are at file sector 16 and read correctly, so a tool reports the volume identifier, the publisher, the data preparer and the dates without complaint. But every extent in every directory record is an absolute LBA based at 45,000, and a tool that resolves an extent as a file offset lands 92,200,960 bytes into the image, finds a zero, reads it as a record length of 0, terminates the walk, and prints

directories : 0
files       : 0

with exit status 0. That is what this collection's iso9660.py did to a 1,032,499,200-byte image containing 2,826 files. Padding the image with 45,000 zero sectors does not help: the descriptor scan then starts at file sector 16 and the PVD is at 45,016, so the tool reports no volume descriptor of type 1 — a different lie.

The fix is one subtraction in one place: keep the descriptor scan where it is, subtract the base only when an extent becomes an offset, and default the base to 0 because on the low-density area the extents are file offsets. And the check that matters more than the fix: a root directory record that declares a non-zero length and yields no entries must raise, not return an empty list.

The convenience half of the original claim survives: it is ISO 9660, and the volume descriptor fields do read straight off. Decide the convention once, write it at the top of the report, and use it everywhere. On the Amiga CD family the equivalent confusion — hunk offsets against file offsets — was serious enough that "fix the offset convention before quoting an address" became step one of the order of work.

Read the volume descriptor's own fields and note them: volume identifier, publisher, data preparer, application identifier, and the creation and modification timestamps. On Amiga CD, the data-preparer field once named a member of the development team, and on CD-i the descriptor's timestamp against the files it indexes gave up the last overnight code change before mastering.

[2 of 2], on two volumes of each of two discs. No Joliet and no Rock Ridge, on the high-density area and on the low-density one: one namespace each, primary only.

And a thing this section did not think to ask for. On Sonic Adventure the low-density volume declares 15,627 sectors against a track of 14,564 — and 14,564 + 1,063 = 15,627 exactly, where 1,063 is the audio track. The low-density volume covers the whole session, audio included. One disc cannot tell an authoring-tool property from an arithmetic coincidence on one specimen. The second disc tells in ten seconds: read its low-density PVD and see whether the identity holds with a different pair of track lengths.

4. Census the disc before reading any of it

The three cheap passes, unchanged, because they pay for themselves on every optical format:

All-zero and near-zero files. A byte histogram per file. Ten seconds, and it is often the best leftover-hunting move on a format with space to spare — a 1.2 GB disc has a great deal of space to spare.
Timestamps. Bucket the directory dates by day. Note anything stamped after the volume descriptor; note anything stamped implausibly early. On Amiga CD, a whole set of 1992 timestamps on a 1996 disc turned out to be the real build date of an earlier release, and a different set on a different disc turned out to be a wrong clock. The outliers are the finding.
Sector map. Attribute every sector to a file, to the file system, to IP.BIN, or to nothing — and hexdump whatever belongs to nothing. On a disc this large, expect a great deal of padding, and measure it rather than estimating it.

CORRECTED [0 of 1]. The version this document opened with read:

[unverified] Add the platform-specific pass: check for a dummy file. It was common practice to pad a GD-ROM out with one enormous junk file so that the real data sits at the outer edge of the disc, where the drive reads fastest. If one is present, it may be several hundred megabytes, and reporting the disc's "capacity used" without separating it out makes the number meaningless.

The pass is worth running and the first disc answers zero. Sonic Adventure has no dummy file — 0 of 2,826 filenames contain DUMMY in any casing — and pads instead with unclaimed sectors: 3,436 of them, 7,036,928 bytes, 0.5756 % of the raw dump, in two runs, both entirely zero. 3,286 sectors sit between the end of the subdirectory's records and the first file; 150 are the tail of the track.

The consequence is the opposite of what the paragraph expected: the capacity figure on this disc is honest without adjustment. 500,714 of 504,150 sectors claimed = 99.3185 %, remainder zero on the full accounting 18 + 2 + 71 + 500,623 + 3,436 = 504,150.

One disc cannot say what was common practice on a platform of several hundred titles, so the practice claim is not deleted. What is corrected is the instruction: measure the unclaimed sectors, not just the dummy file — on this disc the padding is not a file at all, and a pass that only looks for a filename reports nothing and misses seven megabytes.

[2 of 2], and the second disc makes fifty times the same point. Dino Crisis has no dummy file either — 0 of 294 filenames — and 171,491 unclaimed sectors, 351,213,568 bytes, 34.0159 % of the high-density track, against the first disc's 0.5756 % of the raw dump. Fifty times emptier and still not a file, so the correction holds at both extremes.

And the second disc shows the reason the original paragraph gave, which the first could not. The hole is in front: sectors 27 to 171,367, so the first file begins at absolute LBA 216,368 and the 294 files run from there to the end of the track with zero inter-file gap over 332,632 sectors. That is not slack; it is one block of nothing followed by one block of everything, and it buys precisely the outer-edge seek benefit the paragraph described — paid for with a hole rather than a junk file.

So the pass has three parts, not one: look for a dummy file; measure the unclaimed sectors; and check where they are. A hole at the front and a hole at the back mean different things, and only the second disc could show it.

A third figure worth recording. The word this pass looks for is on the Dino Crisis disc 32,361 times — as the sixteen ASCII bytes dummy header padding the header block of 265 .DAT files. Zero filenames, half a megabyte of the string. A filename search would have reported nothing about a disc that says dummy thirty-two thousand times.

5. `1ST_READ.BIN`, and the scramble that will waste your afternoon

[unverified], and deliberately still so. The main executable is normally 1ST_READ.BIN, named by IP.BIN, loaded at 0x8C010000, and running on the SH-4 — 32-bit, little-endian, with a vector floating-point unit the platform's matrix code leans on heavily. Note the endianness against the 3DO's, which is the opposite; the two checklists in this collection now bracket both.

The name is confirmed — IP.BIN on the first disc says 1ST_READ.BIN and that file is there. The load address and the vector unit are not, and cannot be without disassembling the binary, which the first pipeline did not do. The mark stays.

And a warning about the endianness claim, which is true and answers a different question than the one you will ask. The CPU is little-endian. The files on the first disc are mixed, and the split runs along the age of the format rather than along the machine:

	byte order	measured on
`.ADX` — CRI audio, older than this console	big	4,197 of 4,197 files
`.AFS` — CRI archive, written for this console	little	2 of 2
`PVMH` / `PVRT` — Sega textures	little	17,083 textures
`SMLT` — sample bank	little	123 of 123
ISO 9660	both, by specification	useless as evidence

So "the SH-4 is little-endian" does not tell you how to read a field on this platform. Read the field.

CORRECTED, and this was the line this document used to call its single most important. The version it opened with read:

And then the trap, which is the single most important line in this document: on a retail disc, 1ST_READ.BIN is normally scrambled. The bootstrap in IP.BIN descrambles it as it loads. A copy pulled straight off the file system and thrown at a disassembler yields nothing, and — worse — yields nothing that looks like nothing, because the entropy is high enough that a scan for instructions will happily report a few thousand plausible opcodes.

So: descramble first, then check that you did.

The first retail disc that ever tested this line falsified it. Sonic Adventure's 1ST_READ.BIN is not scrambled, and three independent measures say so before a disassembler is opened:

measure	Sonic Adventure	a scrambled file
entropy	5.5078 bits/byte	above 7.9
zero bytes	38.46 % (2,596,213 of 6,751,168)	about 0.39 %
printable runs of 6+	9,355, covering 1.18 %	a handful, by chance
bytes at +0x30	`Lib Handle Start`	noise

[2 of 2] not scrambled, and no single threshold survives. Dino Crisis's 1ST_READ.BIN is 1,745,124 bytes at entropy 6.9511 with 10.95 % zero bytes, against the first disc's 5.5078 and 38.46 %. Both are far from the 7.9+ and 0.39 % of an encrypted file and far from each other, so measure the entropy first survives and any numeric threshold taken from one disc would not have.

And a result neither disc asked for: the two executables share their first 44 bytes exactly, sha1 a968fd11f2b7182a0991ff693b0b642d45e8b0e2, with Lib Handle Start at +0x30 on both, diverging at byte 44 on a little-endian SH-4 address whose top byte is 0x8C — 0x8C0100A0 there, 0x8C15A520 here. A loader stub and then a pointer that depends on the size of the program. Read and hash the first 64 bytes of 1ST_READ.BIN on every disc; it costs one command and it is this platform's cheapest cross-disc test.

The word normally may still be true of the platform and one disc cannot show otherwise — but it was doing work this document did not admit, and the practical instruction is now inverted:

Measure the entropy first. Descramble only if the number says to. One command, and it costs nothing on a disc that does need descrambling.

Record which it was either way: the presence or absence of the scramble is a fact about the build. And when publishing a hash, say which bytes it covers — on this disc there is only one set, so 721651c549ad00a04e6a8c299a780cb7e72669f9 needs no qualifier beyond as it lies on the disc.

[1 of 1], and two of the three came back negative. Then the standard three passes:

Toolchain fingerprints. Compiler banners, @(#) strings, source paths, library version strings, assertion text with file and line. Sega's own libraries, the SDK version and the compiler are all normally identifiable. On Sonic Adventure the libraries are and the compiler is not. Seventy-five distinct library build stamps — Ver 2.00 Build:Aug 02 1999 21:46:09, Ver.5.51 Build:Mar 12 1999 15:55:56 in two files at the same second, and so on — across five binaries, running 1998-11-10 to 1999-08-02. But no compiler banner at all: gcc counts 66 and GCC 222 on a track where a three-byte string has an expectation of 61.54, which is noise, and there is no version line anywhere at 100 % coverage. And no source paths: six drive-lettered hits in 1,451,990,369 bytes, five distinct strings, every one of them five to eight characters long inside compressed texture data. Zero real paths. The library stamps are the fingerprint on this platform, not the compiler.
Filenames, both ways. Every path-like string in the binary against the directory listing, and every file on the disc against every binary's strings. Names with no file are cut content; files no binary names are generated names or dead weight.
Which runtime is this? Some titles boot Sega's own libraries directly; others ship a Windows CE runtime with Direct3D on top. These are very different objects and the difference is visible in the first few kilobytes of strings. Establish which before writing a word about the engine. [1 of 1] Sonic Adventure is Sega's own: Microsoft 0, Windows CE 0, PowerVR 0, Hitachi 0, at 100 % coverage over 1,451,990,369 bytes; against Ninja 5, KAMUI 3, Shinobi 4 — all five characters or longer, so real at any count above zero. PowerVR counting zero is worth a sentence on its own: the platform's most famous component leaves no string in its most famous game.

6. Middleware fingerprints — expect to be reading someone else's format

[1 of 1], and more strongly than this paragraph expected. On Sonic Adventure, 74.02 % of the file bytes are CRI's formats and the mastering tool named in both volume descriptors is CRI's too — CRI CD CRAFT VER.2.30. This is the first object in the whole collection where three quarters of the content belongs to a third party whose name is written into the disc's own volume descriptor.

The names to grep for, and what each implies:

CRI — ADX audio, Sofdec video, and AFS archives. If the disc has AFS files, most of its content is inside them and the archive index is the first thing to parse.
Sega's Ninja library — model and texture-list files, commonly .NJ and .NJTL.
Sega's Kamui — the PowerVR rendering layer.
Sega's Shinobi — the higher-level library set.
PRS — CORRECTED. This line used to read "Sega's LZ-plus-Huffman compression, used widely enough that a first-pass compression scan should look for it specifically." There is no Huffman stage. .PRS is byte-aligned LZ77 with an 8-bit control word read least-significant-bit-first, one bit per token, 1 = copy one literal byte through, 0 then a second bit selecting a two-byte long match (13-bit negative offset, 3-bit size, size 0 meaning a following length byte) or a two-bit-plus-one-byte short match. 00 00 after a long-match prefix is end of stream. A Huffman-coded stream cannot begin with a literal byte at a byte boundary chosen by a flag byte that is itself a literal 0xff. Derived and closed on 843 of 843 files of Sonic Adventure, 130,472,746 bytes in, 266,057,533 out, 2.0392×, input consumed exactly on every one, and a single-bit mutation of the first control byte fires on 8 of 8. The rest of the line stands and understates itself: .PRS is 12.7659 % of the high-density files as stored, hides 16,641 textures in 582 of them, and hides the entire staff roll in plain ASCII in another.
Windows CE / Direct3D — a different world again, with its own file conventions.

[1 of 1] State the middleware inventory as counts, in a table, before describing any single format. For Sonic Adventure, against 1,051,494,320 bytes of declared files:

format	vendor	files	bytes	%
`.ADX` loose	CRI	109	455,967,110	43.3637 %
`.SFD`	CRI (Sofdec)	10	183,687,168	17.4692 %
`.AFS`	CRI	2 (4,088 members)	138,684,416	13.1893 %
`.PRS`	Sega	843	130,472,746	12.4084 %
`.MLT`	Sega/AICA	123	52,254,080	4.9695 %
`.PVR`	Sega/PowerVR	442	30,541,812	2.9046 %
`.DRV`	Sega (ARM7)	2	66,816	0.0064 %
became a six-entry table across eighteen discs and turned out to be one of the
most useful things in the whole document.

7. Graphics — PowerVR, tiles, and proving a geometry

[unverified] The GPU is a tile-based deferred renderer, which matters for reading the code rather than the assets: there is no depth buffer in the usual sense and no front-to-back sorting in the program, so the rendering loop will not look like the rendering loop of a contemporary machine. Do not read its absence as an absence of a renderer.

The texture format to expect is PVR, normally with a PVRT chunk, sometimes preceded by a GBIX global-index chunk. Expect twiddled (Morton-ordered) texel layouts, several colour formats including 4-4-4-4, 1-5-5-5 and 5-6-5, palettised 4- and 8-bit variants, and VQ compression — a codebook plus indices — which is common enough on this platform to be the default assumption for a large texture rather than the exception.

[1 of 1], with one correction. On Sonic Adventure, 17,083 textures: 442 loose .PVR with GBIX + PVRT, and 16,641 inside 582 PVMH archives that come out of .PRS files, where the global index lives in the archive's directory entry instead of in a chunk. PVRT layout, derived:

+0  4 'PVRT'   +4 4 u32 LE length   +8 1 pixel fmt   +9 1 data fmt
+10 2 padding, zero everywhere      +12 2 u16 LE w   +14 2 u16 LE h   +16 .. texels

The two zero bytes at +10 are the trap. Read width at +10 and the first texture comes out 0 × 64 instead of 64 × 64, and the geometry closes on nothing at all. Twiddled, VQ and mipmapped are all present as described.

The correction: palettised variants are not the default anywhere. On this disc there is not one palettised texture out of 17,083 — every pixel format is 16-bit (ARGB1555, RGB565, ARGB4444), and the platform's 4- and 8-bit palettised formats are declined everywhere. One disc, so the line is corrected to "expect them and be prepared for a disc that uses none" rather than deleted.

[1 of 1] The rule from both other checklists applies unchanged: prove a geometry before you believe it. Dimensions times depth, plus the header, must equal the file size, or must equal it under a stated rule. With twiddling and VQ in play, a wrong assumption still produces an image — just a scrambled one — so the arithmetic matters more here, not less.

Rules that close on Sonic Adventure: w × h × 2 for twiddled and rectangle, 2048 + w × h / 4 for VQ (a 256-entry codebook of 2×2 blocks, one index byte per block, the indices themselves twiddled). 414 of 414 loose textures where a rule was stated, zero misses. The mipmap padding rule and the small-VQ codebook length were not derived and those 28 are reported as rule not stated rather than averaged into the success rate.

And the step the arithmetic cannot replace: 200 of 200 sampled textures were decoded and looked at, and what they showed was mission text in five languages on a disc whose area field says U .

8. Audio

[1 of 1], and sharpened. The sound hardware is the AICA, an ARM7 core with its own memory running a driver the SH-4 uploads at start-up, so a Dreamcast disc carries two programs for two architectures and the second one is easy to miss entirely.

On the first disc there are two of them. /MANATEE.DRV and /SONICADV/MASARU.DRV, tag SDRV, 33,408 bytes each, different bytes, different directories. They share 31.42 % of their bytes at identical offsets across 3,550 differing runs — the signature of a recompilation, not of a configuration change — and twelve of their thirteen printable runs of 8+ are byte-identical. The thirteenth is the banner, and it differs in one field:

MANATEE.DRV   1998,(C)SEGA ENTERPRISES   1998.12.08:DIGITALMEDIA :Y.Kashima / K.Suyama
MASARU.DRV    1998,(C)SEGA ENTERPRISES   1998.11.27:DIGITALMEDIA :Y.Kashima / K.Suyama

So the instruction is not "find the ARM7 driver" but "find out how many there are, and read its copyright banner first." On this disc that banner cost one strings pass and produced the only human names in the entire readable portion of a 1.2 GB object — everything else was behind a compressor. The two hashes are published so a second disc can test whether either binary is shared: MANATEE.DRV 4f4e69812921c98875eeda08020ef5006d78705e, MASARU.DRV a4af14e535d1c101212bad6c583b5b9fa4ceb8b2.

[2 of 2]. The binary is not shared; the product is. Dino Crisis carries one MANATEE.DRV of 36,160 bytes, sha1 041bd143c4af93ad107a13ae1d1bcf0e9e9cbfda, and the diff against the first disc's two is decisive: common prefix 4 bytes — the SDRV tag — and 10.75 % agreeing bytes against MANATEE.DRV, 10.62 % against MASARU.DRV. Even the two 1998 drivers, built eleven days apart, agree on only 31.42 %.

The authorship is shared and there are now three builds, banner at +0x50 on 3 of 3:

1998.11.27  DIGITALMEDIA  Y.Kashima / K.Suyama   Sonic Adventure  MASARU.DRV   33,408
1998.12.08  DIGITALMEDIA  Y.Kashima / K.Suyama   Sonic Adventure  MANATEE.DRV  33,408
1999.06.16  DigitalMedia  Y.Kashima / K.Suyama   Dino Crisis      MANATEE.DRV  36,160

Same two people, same division, three builds, and the division's name changes case between 1998 and 1999. A Capcom title released in Europe in October 2000 ships a Sega driver built in June 1999, which makes this a versioned product distributed to third-party developers and not a per-title binary. It is the first cross-disc series in this family. Record the build date and the version pair on every disc; a fourth adds a fourth row.

Then the content, in the usual three places:

Streamed music and voice, normally ADX — CRI's ADPCM. Do the duration arithmetic: sample rate, channels, frame count, minutes and seconds. [1 of 1] The header is big-endian — u16 BE magic 0x8000, u16 BE copyright offset with (c)CRI at that value minus 2 and the data at plus 4, then encoding type, block size, bits per sample, channel count, u32 BE rate, u32 BE total samples, u16 BE high-pass cutoff — and it is followed after the samples by a self-describing footer: 80 01, a u16 BE padding count, that many bytes. The whole file closes as data_start + ceil(samples/32) × block × channels + 4 + pad. 4,196 of 4,197 on Sonic Adventure — 5 h 46 m 07.72 s of audio — with the one failure published by name (SSRACING.ADX, 83.67 % of its declared samples present, no footer, size an exact multiple of 2,048). Do not assume a fixed footer length from a sample: four files by hand all read pad = 14, and the field runs to 2,024 across the disc.
.AFS, CRI's archive, is AFS\0 + a u32 little-endian entry count
- that many u32 LE (offset, size) pairs. This is the endianness finding: ADX is big-endian and AFS is little-endian, on the same disc, from the same vendor. The split runs along the age of the format, not along the machine.
Sample banks for the AICA's own channels, commonly .MLT — 64 channels of PCM and ADPCM, with the bank uploaded to sound RAM. [1 of 1] as an identification, and no further. 123 files on this disc, tag SMLT on 123 of 123, 52,254,080 bytes. The record stride was not derived — a flat 32-byte walk matches the header's count on 1 of 123. What did work is crossing the banks against the drivers by structure, since the names never cross: SFOB, SFPB, SMSB and SMPB are named by both drivers and carried by 122 of 123 banks; SOSB is named by both drivers and carried by 0 banks; SFPW is carried by 122 banks and named by neither driver. Both of those asymmetries are findings and neither was expected.

[2 of 2], and both asymmetries are resolved. On Dino Crisis the four tags are not chunks inside a bank at all: SOSB is a section type inside the .DAT room containers, 247 of 253 sections, 32,346,912 bytes, and SFPW is named in DINOSND.MLT, a 320-byte table that points at the others, beside DINOSND.FOB (64 bytes) and DINOSND.FPB (3,168) as whole files. All four carry the version pair 02 00 after the tag where the first disc's carried 01 01. Do not conclude a tag is unused because the disc in front of you keeps it in a different container.
The low-density audio track, which on most discs is the warning message and is content in only the narrowest sense — but measure it, and check whether it is byte-identical across discs. It is a strong candidate for a shared, pressed-verbatim asset, which is exactly the CD-i bumper shape. [2 of 2], and the answer is better than byte-identity. Sonic Adventure: 1,063 sectors, 2,500,176 bytes = 14.1733 s, of which a 150-sector all-zero pregap and a 12.1733 s body, 83.50 % non-zero, sha1 0fbfde20b242982862a011ba8ed8031e2c6f24a4. Dino Crisis: 6,100 sectors, 14,347,200 bytes, the same 150-sector all-zero pregap and a 79.3333 s body, 97.41 % non-zero, sha1 a34bdc73ac1e360b13d0592b501a388e0225e994.

Not a shared recording — and the length is a measurement, not a detail. Decoded, segmented at its five quiet stretches of 0.6 s or more, and pitch-tracked per second, the European track is five blocks by five voices, median fundamentals 190, 158, 236, 141 and 249 Hz. A human listened: it is the platform's spoken warning in English, French, German, Spanish and Italian, and block 1 is 12.50 s — the length of the American disc's entire track.

So the red-book track's length is a count of the region's languages. Record the sector count, the block count and the languages. A Japanese disc should give one block of about twelve seconds, and that is the falsification.

And listen to it. Neither disc's track had been played by anyone in this collection until the second session; the first disc's still has not been.

9. Video

[2 of 2] Expect Sofdec — CRI's MPEG-1-based full-motion video, normally .SFD — and expect it to account for a large share of any disc that has cut scenes at all. On Sonic Adventure: 10 files, 183,687,168 bytes, 17.4692 % of the file bytes. On Dino Crisis: 7 files, 428,367,872 bytes, 62.8827 % — the largest single category on that disc by a factor of three. Both are ordinary MPEG-1 program streams, 00 00 01 BA on 17 of 17, video on stream 0xE0. avi.py and every RIFF walker are inapplicable and that is worth one line after looking, not instead.

CORRECTED [0 of 2] — this section was wrong about both discs. The version above used to continue:

video on stream 0xE0, MPEG audio on stream 0xC0, no CRI audio smuggled into a private stream.

The payload of stream 0xC0 is CRI ADX, on 7 of 7 films here and on 10 of 10 there. Both open 80 00 01 1c 03 12 04 02 with (c)CRI at copyright_offset − 2 = 284, encoding type 3, 18-byte blocks, 4 bits per sample, 44,100 Hz stereo — the same four invariants adx.py checks on archive members. A stream id is a slot number, not a codec, and the first pipeline read the id and stopped. Extract the 0xC0 payload and test its first two bytes before naming the codec. The consequence is not cosmetic: on Dino Crisis the films add 30,754,320 bytes of ADX, taking the disc's ADX total from 21.57 % to 26.0831 % of its file bytes.

And there is a metadata stream the first disc could not see. Every Dino Crisis film carries exactly one packet of stream 0xBF, MPEG private stream 2, 2,030 bytes, and Sonic Adventure's ten carry none — and no printable string at all in 183,687,168 bytes. Its layout, derived:

+0x0E   SofdecStream                      +0x0186  480, constant on 7/7
+0x2E   <name>.sfd  YYYYMMDDHHMM          +0x0188  bit rate / 400  (5,000,000 bps)
+0x4E   SFDMUX Ver.1.07 19990520 CRI      +0x01CA  44100, the sample rate
+0x16E  <name>.m1v  YYYYMMDDHHMM          +0x00A6  u32 audio duration, ms
+0x18E  DCMVCRE 1.00APWI                  +0x00AA  u32 video duration, ms
+0x1AE  <name>.sfa  YYYYMMDDHHMM          +0x00AE  u16 FRAME COUNT

The frame count and the video duration agree with an independent walk of the elementary stream on 7 of 7. A quantity encoded twice in two different ways that agrees on N of N is a structure demonstrated. On a disc whose filesystem stamps 297 files with one instant, seven files carry their own authoring minute — here 2000-06-08 15:23 to 2000-06-26 19:30. Check stream 0xBF before believing a disc has only one clock.

Two things this section did not warn about:

The bit rate is in the filename. SA1_600 through SA8_600 all declare 4,800 kbit/s in their sequence headers = 600 kilobytes per second, and SA7_300 declares 2,400 = 300. A quantity encoded twice, agreeing on 8 of 8.
[2 of 2] the geometry is anamorphic and a person has to look. Dino Crisis adds two more shapes — six films at 320×320 and one at 160×192, neither of which is 4:3 at square pixels — so four geometries are now known and none of them is the display shape. On the 320×320 films a point-light bloom is a vertical ellipse at 1:1 and a circle stretched 4:3, which settles it the same way the first disc's logo did. Ten sessions in this branch have settled a geometry by decoding a frame and looking at it, ten times out of ten.
Two of the ten are 320×448 and they are anamorphic, not interlaced. The obvious reading of 448 = 2 × 224 is two stacked fields, and it is false: interleaving the halves makes the picture 43 times rougher by vertical gradient (34.36 against 0.79). They are half-horizontal-resolution frames for the VGA output, displayed at 640×448, while the eight 320×240 files double both ways for a television. The sequence header's aspect-ratio code is no help — it reads 12, 1 and 2 across the ten and none of the three yields a sensible display shape. A person looking at the studio logo settled it: stretched 2:1 the letters are correctly proportioned and at 1:1 they are not.

The container question from CD-i §9b transfers: validate a container by chaining it. Walk from the first chunk to the last and require that the final one end exactly at end-of-file. Nearly chaining is not understanding. [1 of 1], three times over on one disc: the ten .SFD walk pack by pack to a 00 00 01 B9 end code with no unparsable byte, 10 of 10; the 582 PVMH archives chain PVRT chunks to the exact last byte and match their header's declared entry count, 582 of 582 on both; the two .AFS account for every byte including inter-member gaps, remainder zero on 2 of 2. And an independent decoder is worth the two minutes it costs: ffprobe -count_frames returned exactly this session's frame counts on 4 of 4 files (2,934 / 3,257 / 200 / 334) and the same width and height.

And the ratio question, which on CD-i predicted more than any other single number: what fraction of the disc is video? A title that is 70 % Sofdec is a different object from one with no video at all, and on a 1.2 GB disc the difference is hundreds of megabytes of pressing decisions.

10. Baselines, so you can tell signal from noise

Two rows. This table is the point of the whole document, and it fills up one disc at a time. The second row is the moment it starts to be worth something, because a column that gives the same answer twice is a column that does not discriminate, and four of these now do.

Disc	Year	Studio	Product no.	Areas	Dump form	HD track	Capacity used	Dummy file	Files	Dirs	Middleware	Scrambled	Video %	Padding %	Languages	Build stamp span
Sonic Adventure (USA)	1999	Sonic Team	`MK-51000`	U	BIN/CUE, 3 tracks, 2352	504,150 sectors	99.3185 %	none	2,826 + 16	1 + 1	CRI ADX / Sofdec / AFS, 74.02 %	no	17.4692 %	0.5756 %	5	75 stamps, 1998-11-10 → 1999-08-02
Dino Crisis (Italy)	2000	Capcom	`T7019D 13`	E	BIN/CUE, 3 tracks, 2352	504,150 sectors	65.9841 %	none	294 + 3	0 + 0	CRI ADX / Sofdec / AFS, 84.7172 %	no	62.8827 %	29.2394 %	6	44 stamps, 1999-04-23 → 2000-03-07, + linker 2000-10-23 15:27:59

The high-density track is 504,150 sectors on both — 1,185,760,800 bytes to the byte, two publishers, fourteen months. If that holds on a third disc, this column has a fixed denominator and "capacity used" becomes directly comparable between titles in a way it is on no other platform in this collection. Two of two is not a law; it is a much better hypothesis than one of one.

Notes on the columns, because the first row is also the first test of whether they discriminate:

Capacity used is claimed sectors over track sectors: 500,714 / 504,150 and 332,659 / 504,150. Neither disc has a dummy file and the column still spans 33 points, so it discriminates — but quote it with the padding column beside it or it lies. On the second disc a third of the track is a hole, and the hole is deliberately in front of the data.
Middleware % is against the declared-file denominator: 1,051,494,320 bytes on the first disc, 681,217,379 on the second. Every percentage in a row of this table must name its denominator or it cannot be compared with the next row. This is the single most important convention in the table, and the second disc proves it: the same object weighs 1,201,166,751, 1,047,768,000 or 681,217,379 bytes depending on what you call it, a spread of 76.33 % where the first disc's was 16.26 %, and the middleware figure is 84.7172 % or 48.0456 % accordingly — thirty-six and a half points and ten places in a league table, decided by one sentence.
Padding % is unclaimed sectors as a fraction of the raw dump.
Studio is Sonic Team and the first disc says so once, in capitals, in a compressed file. On the second disc the field is a trap of the opposite shape: three strings name three different companies. IP.BIN's company field says SEGA LC-T-70, both volume descriptors say CAPCOM CO.,LTD., and 1ST_READ.BIN says CAPCOM twice. The cell says Capcom on the volume descriptors and the copyright file, and SEGA LC-T-70 is recorded here as what it is: a licensee code, not a publisher name. Record which string a row's Studio cell came from.
Languages and Build stamp span were proposed by the first row and are added here, because the second disc gave them values that differ from the first's and a column that never differs is not a column. Languages counts what is present, not what is declared — both discs declare one area and carry five and six. Build stamp span counts distinct Build: stamps inside the binaries and their range; on the second disc the linker's own stamp is quoted separately because it is four hours and thirty-six minutes before the mastering instant, which is the tightest link between two clocks either disc has produced.

The columns are a proposal, not a commitment. The CD-i table settled at thirteen columns and the Amiga CD one at nineteen, in both cases after the third disc showed which questions actually discriminated. Add a column when a disc makes you want one; delete any that never separates two titles.

11. Order of work — proposed, and untested

[1 of 1] as reordered below. This was the CD-i and Amiga CD order adapted to what this platform was documented to be. The first pipeline ran it and it survived with three changes, marked in place:

Read the dump form before the image. Track list, LBAs, modes, which area each track is in, and what this dump form cannot represent. Write down the LBA convention you are going to use and then use it everywhere.
IP.BIN before the file system. Every header field, the checksum, and a hash of the bootstrap and logo region.
ISO 9660: volume descriptor fields, then the listing.
CHANGED — census: all-zero files, timestamp histogram, sector map, unclaimed-sector measurement, and the dummy-file check before quoting any capacity figure. The step used to say "padding measurement" and assume the padding was a file. On the first disc it was 3,436 unclaimed sectors and no file at all, and a timestamp histogram with exactly one bucket for 2,842 files.
Attribute every sector. Anything owned by nothing gets a hexdump.
CHANGED — measure the entropy of 1ST_READ.BIN before descrambling it. The step used to read "descramble, and prove that you did." One command — entropy and a zero-byte fraction — tells you whether there is anything to descramble, and on the first disc there was not. Record which it was.
Middleware inventory as counts, before describing any single format.
Strings, both ways, and read anything in the root that nothing references.
CHANGED — find all the ARM7 sound drivers, and read the copyright banner first. The step used to read "find the ARM7 sound driver." The first disc had two, and their banners carried the only human names in the readable portion of the whole object.
Prove one texture geometry, end to end, before opening any other.
Chain one container, end to end, before believing any other.
Publish notes/sha1-all.txt (section 12).
Fill in one row of section 10, and re-mark in this document every claim the disc exercised.

12. The hash lists

[1 of 1] Every disc repository publishes notes/sha1-all.txt: one record per file, with path, size and SHA-1. It costs nothing, and it is the only way the next pipeline can find something this one could not see. Sonic Adventure publishes it plus three companion lists: 2,826 high-density files, 16 low-density files, and 4,088 .AFS members.

The CD-i family learned two things about these lists that transfer directly:

A file-level list is not enough. When the unit of sharing is smaller than a file — a track inside an archive, a texture inside a bank — file hashes cannot see it. That family ended up publishing a second, finer list beside the first, and it found fourteen crossings where the file-level list had found two.
Hash the right bytes. Comparing a descrambled 1ST_READ.BIN against the scrambled bytes on the disc will report a difference for ever. State which of the two any published hash refers to.

[2 of 2] For this platform the finer list is per member inside the AFS and other archives, and the first disc shows why: two .AFS files in the directory are 4,088 members, and 582 .PRS files are 16,641 textures. A file-level list of Sonic Adventure has 2,842 rows and the disc has at least 22,987 addressable things in it. IP.BIN's bootstrap region is a third list of its own — ecc407cacf78c18d9bc85da896c172edfb1a0d58, 32,512 bytes — and it is the one most likely to cross between unrelated discs.

[2 of 2], and the second disc adds a fourth kind of list and one absence. Dino Crisis publishes sha1-all.txt plus sha1-highdensity.txt (294), sha1-lowdensity.txt (3), sha1-afs-members.txt (822) and sha1-vendor-assets.txt, which is the list of sub-file regions that might be pressed verbatim — IP.BIN whole, its header, its bootstrap, the audio track whole and its body, the ARM7 driver, the executable, and the 44-byte loader prologue. Those are the bytes a file-level crossing cannot see, and on this platform they are where the sharing is: crossall.py found 0 crossings of 291 hashes against 73 repositories and 52,991 hash tokens, while the bootstrap region is 99.69 % shared and the prologue is identical.

And record absences. There is no 0GDTEX.PVR on the second disc and no .PVR of any kind, 0 of 294, which removes it from the first disc's list of three candidate vendor assets. An absence written down is a measurement; an absence unmentioned is a hole in the next pipeline's day.

Open questions

Written before the first disc, and now marked by two. Answered means two discs agree and a third can still break it.

Where the six below stand after two discs

ANSWERED, and the method was wrong. One block, 99.6893 % shared, patched in four places. Diff it, do not hash it. See section 2 point 3.
ANSWERED: not one recording, and the length is the finding. 12.17 s in one language on the U disc, 79.33 s in five on the E disc. See section 8.
ANSWERED for two discs and open for the platform. Neither has a dummy file. Padding is 0.5756 % on one and 29.2394 % on the other, and on the second it sits in front of the data. See section 4.
ANSWERED: the binary is not shared, the product is. Three builds, two names, one division, 1998-11-27 to 1999-06-16. See section 8.
UNTOUCHED. Neither disc is a Windows CE title. Windows CE and Microsoft both count 0 on both.
ANSWERED: yes, on 2 of 2. The first disc's area field says U and it carries five languages including Japanese; the second says E and carries six including Japanese dialogue bitmaps it cannot reach. The area field is a pressing declaration, not an inventory.

And the ones the second disc leaves for the third

Does the eight-slot region table in IP.BIN index by area-symbol position? Two discs fill slots 1 and 2 for U and E. A Japanese disc should fill slot 0. Six slots have never been seen filled.
What is the byte at IP.BIN image offset 0x60C0? 0 on E, 1 on U. A third value or a repeat decides between a video-standard flag and a region index.
Is the high-density area always 504,150 sectors? Two of two, exactly. ls -l settles it.
Is the 44-byte loader prologue universal? sha1 a968fd11f2b7182a0991ff693b0b642d45e8b0e2, two of two. Read 44 bytes.
Does SFDMUX always write a 0xBF metadata packet? 7 of 7 with Ver.1.07 19990520; 0 of 10 on a 1999 disc that names no multiplexer at all.
Is dummy header a Capcom convention or a middleware one? 32,361 occurrences on the second disc and it is what made its container legible. On a non-Capcom disc it would be a tool; on Capcom's alone, a house style.

The original list, as written before there was anything to be open about

How many generations of the IP.BIN bootstrap and logo are there? This is the CD-i bumper question transplanted, and it is the most likely place on this platform for a cross-disc byte-identity result.
What is in the low-density area, disc to disc? Is the warning track one recording pressed everywhere, a handful per region, or per publisher?
How much of a typical disc is dummy padding, and did the practice change over the platform's short life?
Is the ARM7 sound driver shared? It is uploaded rather than executed in place, it comes from a small number of sources, and nobody looks at it. If one binary turns up on unrelated titles, that is a result of the same shape as the Amiga CD .TM block.
What did the Windows CE titles actually ship? They are a different platform wearing the same case, and the fraction of the library that used them is small enough to be enumerable.
Does the header ever lie? Area symbols, peripheral declarations and release dates are all assertions by the build, and all three are checkable against the disc that carries them.

FilesExpand file tree

dc-platform-notes.md

Latest commit

History