Catalogue: libmagic flag and parser divergences with no shipped-definition exposure

[ESultanik]

Summary

A full audit of every string, search, regex and pstring modifier against libmagic 5.48 ran on
2026-09-11. It found ten divergences and four incidental ones. Three were significant enough to get
their own issues — #3562 (W), #3568 (the sort-key bits), #3575 (the b pass gate).

This issue catalogues the remainder: every divergence with no shipped-definition exposure. None
affects a file PolyFile ships definitions for today. They are recorded so they are not rediscovered
one at a time, and so that anyone writing their own .magic file has a list of what differs.

Audited at fde96d4 against the reference binary at file/src/file (libmagic 5.48). Verification
was behavioral, not just by reading: hand-written definitions run through both tools, plus a
1,800-case randomized differential (900 string, 900 search) under LC_ALL=C that produced 14
divergences, all of which are accounted for here or in the three issues above.

1. regex/C is not implemented, and the declaration fails the whole file

REGEX_ICASE (file/src/file.h:432) fires on either STRING_IGNORE_LOWERCASE or
STRING_IGNORE_UPPERCASE, so regex/C and regex/cC are case-insensitive regexes. PolyFile's
REGEX_TYPE_FORMAT (polyfile/magic.py:3048) accepts only [bcslTt].

$ printf '0\tregex/C\taB\tHIT\n' > c.magic ; printf 'AB\n' > in.bin
$ TZ=UTC ./file/src/file -b -m c.magic in.bin
HIT, ASCII text

PolyFile: ValueError: ... line 1: Invalid regex type declaration: 'regex/C'

The failure mode is the concern rather than the flag: MagicMatcher.parse raises and the entire
user definition file is lost, where file loads it and carries on. The same applies to items 2
and 3 below.

2. Parse strictness: repeated / and joined count-plus-flags

apprentice.c:2021-2023 allows repeated / between modifiers "for readability", and the flag loop
reads digits and letters in any order.

$ printf '0\tstring/W/c\ta\\ b\tHIT\n'    # file: HIT       polyfile: ValueError
$ printf '0\tsearch/40/c/f\tab\tHIT\n'    # file: HIT...    polyfile: ValueError
$ printf '0\tsearch/40c\tab\tHIT\n'       # file: HIT...    polyfile: ValueError

string/16/c, regex/5/c, search/c/40 and search/c40 all work today.

3. pstring ignores t and rejects b

set_test_type honors STRING_TEXTTEST for FILE_PSTRING (apprentice.c:1253-1266), but
PascalStringType defines no declared_test_types, so the base class returns BINARY
unconditionally. PSTRING_TYPE_FORMAT (magic.py:2699) omits b, although CHAR_BINTEST is
type-agnostic in libmagic.

$ printf '0\tpstring/Bt\tabcdefghi\tHIT\n' > p.magic ; printf '\tabcdefghi' > in.bin
$ TZ=UTC ./file/src/file -b -m p.magic in.bin
HIT, ASCII text, with no line terminators

PolyFile: ['HIT'] — right verdict, missing the text-encoding description, because it ran in the
binary pass. pstring/Bb raises.

4. regex/l with no count applies a line limit where libmagic applies none

str_range == 0 makes linecnt = 0 and bytecnt = nbytes, so libmagic imposes no line limit and
only the 8 KiB regex_max cap (softmagic.c:1411-1443). RegexType.__init__ substitutes
8*1024//80 = 102 lines. On a 200-line file with the target on the last line, file matches and
PolyFile does not. All 40 shipped regex/l declarations carry an explicit count.

5. pstring with a 2- or 4-byte prefix and the = relation

getstr folds the prefix size into m->vallen (apprentice.c:3181-3187), but mconvert
left-shifts the string and NUL-terminates only at len (softmagic.c:1241-1266), so the extra bytes
file_strncmp compares are stale copies of the value's own tail. 0 pstring/H ab never matched
b"\x00\x02ab" in any shape tried; PolyFile matches all of them. The inflated vallen also makes
libmagic fail the offset_oob pre-check on short buffers where PolyFile succeeds.

All 35 shipped multi-byte-prefix pstrings use the x or > relation, and x-relation value
extraction agrees exactly for B, H, h, L, l and J.

Open question, stated as such: whether pstring/H|h|L|l with = can ever match in libmagic.
The reading above says no, but that was established by exhaustion over the shapes tried rather than
derived from the code.

6. c and C are locale-dependent in libmagic

file.c:211 calls setlocale(LC_CTYPE, ""), and file_strncmp is not wrapped in the C-locale guard
that file_regcomp and file_regexec use (funcs.c:755-800). Under LANG=en_US.UTF-8,
0 string/c \xe9x matches \xc9x; under LC_ALL=C it does not. PolyFile is always ASCII-only, so it
agrees with the C locale.

No shipped c/C value contains a byte at or above 0x80. PolyFile's behavior is arguably the
better one — a file's type should not depend on the caller's locale — so this may deserve a comment
in the code rather than a fix. The same locale effect gives file a seventh whitespace byte
(\xa0) for W and f on macOS, which PR #3573 pinned out deliberately.

7. Truncated UTF-8 in a test value classifies the test differently

file_looks_utf8 does goto done when a multi-byte sequence runs off the end of the value and
returns 1, so a search or regex value ending in a bare lead byte is text to libmagic
(encoding.c:405-427, read at apprentice.c:1277-1283). PolyFile's _looks_like_utf8
(magic.py:4193) requires a successful decode, so it is binary, and the test lands in the other
pass.

Every level 0 search/regex in magic_defs/ without b or t was checked against a
reimplementation of file_looks_utf8: 0 classify differently.

Incidental, not flag-related

A one-byte file. funcs.c:363-365 short-circuits any one-byte file to very short file (no magic) before soft magic runs.

$ printf 'A' > one.bin
$ TZ=UTC ./file/src/file -b -m file/magic/magic.mgc one.bin
very short file (no magic)
$ polyfile one.bin
data

This is the same shape as #3566, which #3569 fixed for zero-byte files, and it is the only incidental
here that is visible on an ordinary file with the shipped definitions.

search with the x relation. libmagic prints from the match offset to the end of the search
buffer; PolyFile stops at the first CR, LF or NUL. 0 search/40 x V[%s] over b"abc\ndef\n" gives
V[abc\012def\012] in file and V[abc] in PolyFile. Masked when T is set.

A regex value beginning with ^. libmagic's value parser reads the leading ^ as a relational
operator; PolyFile reads it as a regex anchor. Adjacent to #3565.

Not audited

The indirect-offset arithmetic behind INDIRECT_RELATIVE (only letter acceptance was checked),
bestring16/lestring16 matching beyond modifier rejection, and file_magic_strength itself, which
was treated as an input to the sort key rather than audited.

Flags confirmed correct

f, s, t, T, w and the digit/count forms agree with libmagic, verified behaviorally. c and
C agree on string, search and pstring outside the locale case above.