Summary
file_is_csv prints CSV <encoding> text and parses the file's raw bytes, so file names the
text encoding in every CSV verdict and accepts CSV in any encoding whose commas and newlines
survive as bytes. PolyFile's CSVTest and polyfile/magic_defs/csv do neither: the message names
the Python dialect instead of the encoding, and the test decodes the buffer as UTF-8 first, so
every CSV that is not ASCII or UTF-8 is missed.
Found while fixing #3537, which asked whether CSV carries the byte-order-mark divergence #3500
recorded for JSON. It does not: a BOM-prefixed UTF-8 CSV is CSV to file and to PolyFile alike,
because csv_parse walks bytes and the mark only joins the first field. These two divergences are
what is actually there.
Reproducer
$ printf 'a,b,c\n1,2,3\n4,5,6\n' > plain.csv
$ printf '\xef\xbb\xbfa,b,c\n1,2,3\n4,5,6\n' > bom.csv
$ printf 'a,b,caf\xe9\n1,2,3\n4,5,6\n' > latin1.csv
$ python3 -c 'open("utf16.csv","wb").write("a,b,c\n1,2,3\n4,5,6\n".encode("utf-16-le"))'
| file |
file -b |
PolyFile |
plain.csv |
CSV ASCII text |
CSV text (excel dialect) |
bom.csv |
CSV Unicode text, UTF-8 (with BOM) text |
CSV text (excel dialect) |
latin1.csv |
CSV ISO-8859 text |
ISO-8859 text |
utf16.csv |
CSV Unicode text, UTF-16, little-endian text |
Unicode text, UTF-16, little-endian text |
Defect 1: the message names the dialect, not the encoding
file/src/is_csv.c:157-158:
if (file_printf(ms, "CSV %s%stext", code ? code : "",
code ? " " : "") == -1)
code is the file_encoding verdict that file_buffer computed at file/src/funcs.c:369, so the
encoding name belongs to the CSV message itself rather than to a description appended after it.
polyfile/magic_defs/csv:6 prints CSV text (%s dialect) instead, where %s is whichever name
from csv.list_dialects() accepted the buffer. That name has no counterpart in file, and the
encoding it replaced has no other way into the verdict, because CSVTest.precedes_soft_magic ends
the run before the text description is appended.
After #3537 the string this needs for a marked file is available:
LIBMAGIC_ENCODING_NAMES["utf-8-sig"].
Defect 2: only ASCII and UTF-8 CSV is recognized
CSVTest.test (polyfile/magic.py:3865) starts with
text = data[absolute_offset:].decode("utf-8")
and returns a FailedTest on UnicodeDecodeError, so Latin-1, UTF-16 and UTF-32 CSV never reach
the dialect loop. csv_parse counts ", , and \n bytes and ignores everything else
(file/src/is_csv.c:129-190), so the NUL-interleaved commas and newlines of UTF-16 parse as
ordinary field and row separators and a high byte in Latin-1 is just another field character.
Scope
Pre-existing, and not a regression from anything in v0.6.0. No corpus stem in file/tests/ is a
CSV, so test_file_corpus does not cover either half.
Defect 1 changes the message of every CSV match, so it is worth deciding deliberately whether
PolyFile keeps the dialect name it reports today. Defect 2 only adds matches.
Summary
file_is_csvprintsCSV <encoding> textand parses the file's raw bytes, sofilenames thetext encoding in every CSV verdict and accepts CSV in any encoding whose commas and newlines
survive as bytes. PolyFile's
CSVTestandpolyfile/magic_defs/csvdo neither: the message namesthe Python dialect instead of the encoding, and the test decodes the buffer as UTF-8 first, so
every CSV that is not ASCII or UTF-8 is missed.
Found while fixing #3537, which asked whether CSV carries the byte-order-mark divergence #3500
recorded for JSON. It does not: a BOM-prefixed UTF-8 CSV is CSV to
fileand to PolyFile alike,because
csv_parsewalks bytes and the mark only joins the first field. These two divergences arewhat is actually there.
Reproducer
file -bplain.csvCSV ASCII textCSV text (excel dialect)bom.csvCSV Unicode text, UTF-8 (with BOM) textCSV text (excel dialect)latin1.csvCSV ISO-8859 textISO-8859 textutf16.csvCSV Unicode text, UTF-16, little-endian textUnicode text, UTF-16, little-endian textDefect 1: the message names the dialect, not the encoding
file/src/is_csv.c:157-158:codeis thefile_encodingverdict thatfile_buffercomputed atfile/src/funcs.c:369, so theencoding name belongs to the CSV message itself rather than to a description appended after it.
polyfile/magic_defs/csv:6printsCSV text (%s dialect)instead, where%sis whichever namefrom
csv.list_dialects()accepted the buffer. That name has no counterpart infile, and theencoding it replaced has no other way into the verdict, because
CSVTest.precedes_soft_magicendsthe run before the text description is appended.
After #3537 the string this needs for a marked file is available:
LIBMAGIC_ENCODING_NAMES["utf-8-sig"].Defect 2: only ASCII and UTF-8 CSV is recognized
CSVTest.test(polyfile/magic.py:3865) starts withand returns a
FailedTestonUnicodeDecodeError, so Latin-1, UTF-16 and UTF-32 CSV never reachthe dialect loop.
csv_parsecounts",,and\nbytes and ignores everything else(
file/src/is_csv.c:129-190), so the NUL-interleaved commas and newlines of UTF-16 parse asordinary field and row separators and a high byte in Latin-1 is just another field character.
Scope
Pre-existing, and not a regression from anything in v0.6.0. No corpus stem in
file/tests/is aCSV, so
test_file_corpusdoes not cover either half.Defect 1 changes the message of every CSV match, so it is worth deciding deliberately whether
PolyFile keeps the dialect name it reports today. Defect 2 only adds matches.