Summary
libmagic has a built-in compound-document reader, readcdf.c, that runs ahead of soft magic and
reports some OLE 2 payloads on its own. PolyFile has no equivalent, so for those payloads it can
only report the libmagic definition that matched, which nests the description under the OLE 2
header entry. The one corpus file that exercises this is an HWP 5.0 document: file describes it
as Hancom HWP (Hangul Word Processor) file, version 5.0, and PolyFile describes it as
OLE 2 Compound Document, v3.62, SecID 0x2, Mini FAT start sector 0x6 : Hancom HWP (Hangul Word Processor) file, version 5.0.
Reproducer
$ TZ=UTC ./file/src/file -b -m file/magic/magic.mgc file/tests/HWP2016.hwp.testfile
Hancom HWP (Hangul Word Processor) file, version 5.0
$ TZ=UTC ./file/src/file -b --mime-type -m file/magic/magic.mgc file/tests/HWP2016.hwp.testfile
application/x-hwp
$ python -c '
import polyfile.magic
from polyfile.magic import MagicMatcher
polyfile.magic.local_date = polyfile.magic.utc_date
with open("file/tests/HWP2016.hwp.testfile", "rb") as f:
for match in MagicMatcher.DEFAULT_INSTANCE.match(f.read()):
print(str(match))
'
OLE 2 Compound Document, v3.62, SecID 0x2, Mini FAT start sector 0x6 : Hancom HWP (Hangul Word Processor) file, version 5.0
$ polyfile --only-match-mime file/tests/HWP2016.hwp.testfile
application/hwp+zip
So both the description and the MIME type diverge.
Cause
This is not a definitions-handling bug. The standalone description does not come from the magic
definitions at all. file -d shows it arriving before soft magic runs:
$ TZ=UTC ./file/src/file -d -m file/magic/magic.mgc file/tests/HWP2016.hwp.testfile
[try cdf 1]
file/tests/HWP2016.hwp.testfile: Hancom HWP (Hangul Word Processor) file, version 5.0
file_buffer calls file_trycdf before the soft-magic pass, and a hit ends the run unless
MAGIC_CONTINUE is set (file/src/funcs.c:443-452). file_trycdf parses the compound document,
reads its FileHeader user stream, compares the first bytes against the HWP Document File
signature, and prints the description itself (file/src/readcdf.c:633-646):
if (cdf_read_user_stream(&info, &h, &sat, &ssat, &sst, &dir,
"FileHeader", &scn) != -1) {
#define HWP5_SIGNATURE "HWP Document File"
if (scn.sst_len * scn.sst_ss >= sizeof(HWP5_SIGNATURE) - 1
&& memcmp(scn.sst_tab, HWP5_SIGNATURE,
sizeof(HWP5_SIGNATURE) - 1) == 0) {
if (NOTMIME(ms)) {
if (file_printf(ms,
"Hancom HWP (Hangul Word Processor) file, version 5.0") == -1)
The definitions' only version 5.0 entry is polyfile/magic_defs/ole2compounddocs:269. It matches
the directory entry name FileHeader as a lestring16 relative to the OLE 2 header, so it carries
the : continuation prefix and the OLE 2 description ahead of it, and it declares
!:mime application/hwp+zip where readcdf.c emits application/x-hwp. PolyFile reports exactly
what the definitions say; what it lacks is the built-in reader that short-circuits them.
Running file with -k shows all three matches, and PolyFile reports the middle one:
$ TZ=UTC ./file/src/file -b -k -m file/magic/magic.mgc file/tests/HWP2016.hwp.testfile
Hancom HWP (Hangul Word Processor) file, version 5.0\012- OLE 2 Compound Document, v3.62, SecID 0x2, Mini FAT start sector 0x6 : Hancom HWP (Hangul Word Processor) file, version 5.0\012- data
Consequences
tests/test_magic.py:corpus_result_matches carries an escape hatch for this one stem: it accepts a
suffix match instead of an exact one, because the expected description is the tail of what PolyFile
reports. #3501 documents that branch and points here for the cause; the branch has to stay until
PolyFile can read compound documents.
HWP2016.hwp is the only compound document in the libmagic corpus, so the corpus does not measure
how much else readcdf.c affects. Its other outputs include the summary-information fields
(CDF V2 Document, ...), the CLSID line, and the Microsoft ... verdicts it derives from the
document's own streams.
Scope
libmagic's reader is file/src/cdf.c (1682 lines), file/src/cdf.h (363), and
file/src/readcdf.c (702) — a subsystem, not a definition fix, so this needs a decision about
whether PolyFile wants a compound-document reader at all. The narrow version, enough for this
divergence, is a reader that walks the directory, extracts the FileHeader stream, and matches its
signature; the full version reproduces the summary-information parsing too.
Precedent: #3488 added the equivalent of file_ascmagic for the same class of gap.
Summary
libmagic has a built-in compound-document reader,
readcdf.c, that runs ahead of soft magic andreports some OLE 2 payloads on its own. PolyFile has no equivalent, so for those payloads it can
only report the libmagic definition that matched, which nests the description under the OLE 2
header entry. The one corpus file that exercises this is an HWP 5.0 document:
filedescribes itas
Hancom HWP (Hangul Word Processor) file, version 5.0, and PolyFile describes it asOLE 2 Compound Document, v3.62, SecID 0x2, Mini FAT start sector 0x6 : Hancom HWP (Hangul Word Processor) file, version 5.0.Reproducer
So both the description and the MIME type diverge.
Cause
This is not a definitions-handling bug. The standalone description does not come from the magic
definitions at all.
file -dshows it arriving before soft magic runs:file_buffercallsfile_trycdfbefore the soft-magic pass, and a hit ends the run unlessMAGIC_CONTINUEis set (file/src/funcs.c:443-452).file_trycdfparses the compound document,reads its
FileHeaderuser stream, compares the first bytes against theHWP Document Filesignature, and prints the description itself (
file/src/readcdf.c:633-646):The definitions' only version 5.0 entry is
polyfile/magic_defs/ole2compounddocs:269. It matchesthe directory entry name
FileHeaderas alestring16relative to the OLE 2 header, so it carriesthe
:continuation prefix and the OLE 2 description ahead of it, and it declares!:mime application/hwp+zipwherereadcdf.cemitsapplication/x-hwp. PolyFile reports exactlywhat the definitions say; what it lacks is the built-in reader that short-circuits them.
Running
filewith-kshows all three matches, and PolyFile reports the middle one:Consequences
tests/test_magic.py:corpus_result_matchescarries an escape hatch for this one stem: it accepts asuffix match instead of an exact one, because the expected description is the tail of what PolyFile
reports. #3501 documents that branch and points here for the cause; the branch has to stay until
PolyFile can read compound documents.
HWP2016.hwpis the only compound document in the libmagic corpus, so the corpus does not measurehow much else
readcdf.caffects. Its other outputs include the summary-information fields(
CDF V2 Document, ...), the CLSID line, and theMicrosoft ...verdicts it derives from thedocument's own streams.
Scope
libmagic's reader is
file/src/cdf.c(1682 lines),file/src/cdf.h(363), andfile/src/readcdf.c(702) — a subsystem, not a definition fix, so this needs a decision aboutwhether PolyFile wants a compound-document reader at all. The narrow version, enough for this
divergence, is a reader that walks the directory, extracts the
FileHeaderstream, and matches itssignature; the full version reproduces the summary-information parsing too.
Precedent: #3488 added the equivalent of
file_ascmagicfor the same class of gap.