Skip to content

A trailing w or W blank backtracks so the f flag's terminator can match #3570

Description

@ESultanik

Summary

When a w or W value ends with a blank and the same definition sets f, PolyFile's compiled
pattern gives a whitespace byte back so the full-word terminator can match. libmagic consumes the
whitespace run once and never revisits it, so it reports no match.

Reproducer

$ printf '0\tstring/Wf\tA\\ B\\ \tfound\n' > wf.magic
$ printf 'A B  x' > in.bin
$ LC_ALL=C TZ=UTC ./file/src/file -b -m wf.magic in.bin
ASCII text, with no line terminators

PolyFile on master at 54696a1:

from pathlib import Path
from polyfile.magic import MagicMatcher

print({str(m) for m in MagicMatcher.parse(Path("wf.magic")).match(b"A B  x")})
{'found, ASCII text, with no line terminators'}

0 string/wf A\ B\ behaves the same way, and so does the W spelling before and after the fix
for #3562: the flag that renders the blank does not matter, only that a quantifier sits at the end
of the pattern.

Cause

StringMatch.pattern_string renders the value's trailing blank as a greedy quantifier and appends
FULL_WORD_TERMINATOR, which is a lookahead:

A[ \t\n\v\f\r]+B[ \t\n\v\f\r]+(?=[\0\s]|\Z)

Python's engine backtracks. The trailing + first takes both spaces of A B x, the lookahead
sees x and fails, the + gives one space back, and the lookahead then sees the second space and
succeeds.

libmagic runs the two steps in order and only once. The whitespace skip advances b past the whole
run (file/src/softmagic.c:2107-2109), and the full-word check then reads whatever byte b landed
on (file/src/softmagic.c:2127-2130):

if (!isspace(*a))
	while (b < eb && isspace(*b))
		b++;
...
if (len == 0 && v == 0 && (flags & STRING_FULL_WORD)) {
	if (*b && !isspace(*b))
		v = 1;
}

With b on the x, v becomes 1 and the test fails.

Impact

Latent in the bundled definitions. 60 definitions carry f, all of them magic_defs/commands
interpreter lines that also carry w, and none of them ends its value with a blank, so the
quantifier is never the last element of the pattern. A definition that did would match files
file rejects.

Found while fixing #3562, which does not introduce it: master diverges on the same input before
and after that change.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions