Skip to content

CSV verdicts name the dialect instead of the encoding, and only ASCII and UTF-8 CSV is recognized #3554

Description

@ESultanik

Summary

file_is_csv prints CSV <encoding> text and parses the file's raw bytes, so file names the
text encoding in every CSV verdict and accepts CSV in any encoding whose commas and newlines
survive as bytes. PolyFile's CSVTest and polyfile/magic_defs/csv do neither: the message names
the Python dialect instead of the encoding, and the test decodes the buffer as UTF-8 first, so
every CSV that is not ASCII or UTF-8 is missed.

Found while fixing #3537, which asked whether CSV carries the byte-order-mark divergence #3500
recorded for JSON. It does not: a BOM-prefixed UTF-8 CSV is CSV to file and to PolyFile alike,
because csv_parse walks bytes and the mark only joins the first field. These two divergences are
what is actually there.

Reproducer

$ printf 'a,b,c\n1,2,3\n4,5,6\n' > plain.csv
$ printf '\xef\xbb\xbfa,b,c\n1,2,3\n4,5,6\n' > bom.csv
$ printf 'a,b,caf\xe9\n1,2,3\n4,5,6\n' > latin1.csv
$ python3 -c 'open("utf16.csv","wb").write("a,b,c\n1,2,3\n4,5,6\n".encode("utf-16-le"))'
file file -b PolyFile
plain.csv CSV ASCII text CSV text (excel dialect)
bom.csv CSV Unicode text, UTF-8 (with BOM) text CSV text (excel dialect)
latin1.csv CSV ISO-8859 text ISO-8859 text
utf16.csv CSV Unicode text, UTF-16, little-endian text Unicode text, UTF-16, little-endian text

Defect 1: the message names the dialect, not the encoding

file/src/is_csv.c:157-158:

	if (file_printf(ms, "CSV %s%stext", code ? code : "",
	    code ? " " : "") == -1)

code is the file_encoding verdict that file_buffer computed at file/src/funcs.c:369, so the
encoding name belongs to the CSV message itself rather than to a description appended after it.
polyfile/magic_defs/csv:6 prints CSV text (%s dialect) instead, where %s is whichever name
from csv.list_dialects() accepted the buffer. That name has no counterpart in file, and the
encoding it replaced has no other way into the verdict, because CSVTest.precedes_soft_magic ends
the run before the text description is appended.

After #3537 the string this needs for a marked file is available:
LIBMAGIC_ENCODING_NAMES["utf-8-sig"].

Defect 2: only ASCII and UTF-8 CSV is recognized

CSVTest.test (polyfile/magic.py:3865) starts with

text = data[absolute_offset:].decode("utf-8")

and returns a FailedTest on UnicodeDecodeError, so Latin-1, UTF-16 and UTF-32 CSV never reach
the dialect loop. csv_parse counts ", , and \n bytes and ignores everything else
(file/src/is_csv.c:129-190), so the NUL-interleaved commas and newlines of UTF-16 parse as
ordinary field and row separators and a high byte in Latin-1 is just another field character.

Scope

Pre-existing, and not a regression from anything in v0.6.0. No corpus stem in file/tests/ is a
CSV, so test_file_corpus does not cover either half.

Defect 1 changes the message of every CSV match, so it is worth deciding deliberately whether
PolyFile keeps the dialect name it reports today. Defect 2 only adds matches.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions