Text tests run against the file's bytes for every non-UCS encoding, where libmagic uses its decoded buffer

[ESultanik]

Summary

MatchContext.text_test_context re-encodes the buffer for UCS encodings only, so PolyFile's text
pass runs against the file's bytes for every other text encoding. libmagic runs its text tests
against the UTF-8 rendering of its decoded ubuf regardless of which encoding it detected.

#3489 introduced text_test_context and fixed the UCS case, which was the one the corpus exercised.
This issue records the remainder.

The assumption that no longer holds

polyfile/magic.py:841-859:

def text_test_context(self, encoding: Optional[str]) -> "MatchContext":
    """The buffer libmagic runs its text tests against.

    libmagic decodes its input into a UCS-4 buffer, drops the byte order mark, re-encodes that
    buffer as UTF-8, and runs its text tests against the result rather than against the file's
    bytes (``file_ascmagic_with_encoding`` in ``file/src/ascmagic.c``). That only changes the
    bytes for a UCS encoding, so every other input keeps this context and the offsets its tests
    report stay offsets into the file.
    """
    if encoding is None or encoding not in _UCS_BOM_LENGTHS:
        return self

The sentence "that only changes the bytes for a UCS encoding" is the part that is wrong. It holds
for ASCII and UTF-8, where the decoded buffer is the file's bytes. It does not hold for:

• The eight-bit families. A byte at or above 0x80 in ISO-8859 text is one byte in the file and
  two bytes in UTF-8, so both the bytes a text test sees and the offsets it reports differ.
• EBCDIC, once detect_text_encoding has no EBCDIC branch, so EBCDIC text is reported as binary #3507 lands. There the whole buffer is translated through a 256-byte table, so
  essentially every byte differs.

Impact

A text definition that matches high-byte Latin-1 or EBCDIC content can fail to match, and a test
that does match reports an offset into a buffer that is not the file.

No corpus stem covers it. file/tests/ has no high-byte Latin-1 or EBCDIC stem whose expected
result depends on a text test, which is why #3489 could fix UCS alone and stay green.

The tradeoff, which is why the current behavior is defensible

Returning self keeps reported offsets meaningful as offsets into the file. Re-encoding makes the
bytes faithful to libmagic but makes every offset an offset into a derived buffer, which is a
different thing from what PolyFile reports everywhere else and what its Match objects mean.

libmagic does not face this, because it does not report structure — it prints a description and
stops. PolyFile maps files, so it has a constraint libmagic does not. Resolving this means deciding
whether text-test offsets are file offsets or decoded-buffer offsets, and possibly mapping between
them, rather than just widening the condition on line 857.

Suggested next step

Establish first whether any bundled text definition actually matches content above 0x7F. If none
does, this is latent and the fix is a documentation correction plus a test that pins the intent. If
some do, the offset question above has to be answered before the buffer is switched.

Found while implementing #3507. Related: #3489, which fixed the UCS half, and #3488.

[ESultanik]

One more encoding belongs on the list, found while fixing #3537: UTF-8 with a byte order mark.

This issue's summary says the decoded buffer is the file's bytes "for ASCII and UTF-8". That holds
for UTF-8 without a mark. With one, looks_utf8_with_BOM classifies buf + 3 and fills ubuf from
there (file/src/encoding.c:437-444), so libmagic's text tests read the file's bytes shifted left
by the three bytes of the mark.

After #3537 lands, detect_text_encoding names that buffer utf-8-sig, which
text_test_context does not decode, so the divergence is observable:

$ printf '\xef\xbb\xbf<?xml version="1.0"?>\n<svg xmlns="http://www.w3.org/2000/svg"></svg>\n' > bom.svg
$ TZ=UTC ./file/src/file -b -m file/magic/magic.mgc bom.svg
SVG Scalable Vector Graphics image, Unicode text, UTF-8 (with BOM) text
$ polyfile bom.svg
Unicode text, UTF-8 (with BOM) text

polyfile/magic_defs/sgml:6 anchors \<?xml\ version= at offset 0, which the file's own bytes
cannot satisfy while the mark is in front of them. Without the mark the same file matches SVG, XML
1.0 and XML.

This case is narrower than the eight-bit and EBCDIC ones this issue is mainly about: the mark is a
pure prefix, so the buffer is a suffix of the file and every offset shifts by a constant 3. It does
not raise the "offsets into a derived buffer" question the way transcoding does, and it is the same
shape as the UCS case #3489 already resolved, so it may be worth splitting out and widening line
857 for it alone.

I left it out of the #3537 PR because text_test_context is this issue's territory.