Summary
When a w or W value ends with a blank and the same definition sets f, PolyFile's compiled
pattern gives a whitespace byte back so the full-word terminator can match. libmagic consumes the
whitespace run once and never revisits it, so it reports no match.
Reproducer
$ printf '0\tstring/Wf\tA\\ B\\ \tfound\n' > wf.magic
$ printf 'A B x' > in.bin
$ LC_ALL=C TZ=UTC ./file/src/file -b -m wf.magic in.bin
ASCII text, with no line terminators
PolyFile on master at 54696a1:
from pathlib import Path
from polyfile.magic import MagicMatcher
print({str(m) for m in MagicMatcher.parse(Path("wf.magic")).match(b"A B x")})
{'found, ASCII text, with no line terminators'}
0 string/wf A\ B\ behaves the same way, and so does the W spelling before and after the fix
for #3562: the flag that renders the blank does not matter, only that a quantifier sits at the end
of the pattern.
Cause
StringMatch.pattern_string renders the value's trailing blank as a greedy quantifier and appends
FULL_WORD_TERMINATOR, which is a lookahead:
A[ \t\n\v\f\r]+B[ \t\n\v\f\r]+(?=[\0\s]|\Z)
Python's engine backtracks. The trailing + first takes both spaces of A B x, the lookahead
sees x and fails, the + gives one space back, and the lookahead then sees the second space and
succeeds.
libmagic runs the two steps in order and only once. The whitespace skip advances b past the whole
run (file/src/softmagic.c:2107-2109), and the full-word check then reads whatever byte b landed
on (file/src/softmagic.c:2127-2130):
if (!isspace(*a))
while (b < eb && isspace(*b))
b++;
...
if (len == 0 && v == 0 && (flags & STRING_FULL_WORD)) {
if (*b && !isspace(*b))
v = 1;
}
With b on the x, v becomes 1 and the test fails.
Impact
Latent in the bundled definitions. 60 definitions carry f, all of them magic_defs/commands
interpreter lines that also carry w, and none of them ends its value with a blank, so the
quantifier is never the last element of the pattern. A definition that did would match files
file rejects.
Found while fixing #3562, which does not introduce it: master diverges on the same input before
and after that change.
Summary
When a
worWvalue ends with a blank and the same definition setsf, PolyFile's compiledpattern gives a whitespace byte back so the full-word terminator can match. libmagic consumes the
whitespace run once and never revisits it, so it reports no match.
Reproducer
PolyFile on
masterat54696a1:0 string/wf A\ B\behaves the same way, and so does theWspelling before and after the fixfor #3562: the flag that renders the blank does not matter, only that a quantifier sits at the end
of the pattern.
Cause
StringMatch.pattern_stringrenders the value's trailing blank as a greedy quantifier and appendsFULL_WORD_TERMINATOR, which is a lookahead:Python's engine backtracks. The trailing
+first takes both spaces ofA B x, the lookaheadsees
xand fails, the+gives one space back, and the lookahead then sees the second space andsucceeds.
libmagic runs the two steps in order and only once. The whitespace skip advances
bpast the wholerun (
file/src/softmagic.c:2107-2109), and the full-word check then reads whatever byteblandedon (
file/src/softmagic.c:2127-2130):With
bon thex,vbecomes 1 and the test fails.Impact
Latent in the bundled definitions. 60 definitions carry
f, all of themmagic_defs/commandsinterpreter lines that also carry
w, and none of them ends its value with a blank, so thequantifier is never the last element of the pattern. A definition that did would match files
filerejects.Found while fixing #3562, which does not introduce it:
masterdiverges on the same input beforeand after that change.