Skip to content

No equivalent of libmagic's readcdf, so an HWP 5.0 document keeps the OLE 2 prefix #3535

Description

@ESultanik

Summary

libmagic has a built-in compound-document reader, readcdf.c, that runs ahead of soft magic and
reports some OLE 2 payloads on its own. PolyFile has no equivalent, so for those payloads it can
only report the libmagic definition that matched, which nests the description under the OLE 2
header entry. The one corpus file that exercises this is an HWP 5.0 document: file describes it
as Hancom HWP (Hangul Word Processor) file, version 5.0, and PolyFile describes it as
OLE 2 Compound Document, v3.62, SecID 0x2, Mini FAT start sector 0x6 : Hancom HWP (Hangul Word Processor) file, version 5.0.

Reproducer

$ TZ=UTC ./file/src/file -b -m file/magic/magic.mgc file/tests/HWP2016.hwp.testfile
Hancom HWP (Hangul Word Processor) file, version 5.0

$ TZ=UTC ./file/src/file -b --mime-type -m file/magic/magic.mgc file/tests/HWP2016.hwp.testfile
application/x-hwp
$ python -c '
import polyfile.magic
from polyfile.magic import MagicMatcher
polyfile.magic.local_date = polyfile.magic.utc_date
with open("file/tests/HWP2016.hwp.testfile", "rb") as f:
    for match in MagicMatcher.DEFAULT_INSTANCE.match(f.read()):
        print(str(match))
'
OLE 2 Compound Document, v3.62, SecID 0x2, Mini FAT start sector 0x6 : Hancom HWP (Hangul Word Processor) file, version 5.0

$ polyfile --only-match-mime file/tests/HWP2016.hwp.testfile
application/hwp+zip

So both the description and the MIME type diverge.

Cause

This is not a definitions-handling bug. The standalone description does not come from the magic
definitions at all. file -d shows it arriving before soft magic runs:

$ TZ=UTC ./file/src/file -d -m file/magic/magic.mgc file/tests/HWP2016.hwp.testfile
[try cdf 1]
file/tests/HWP2016.hwp.testfile: Hancom HWP (Hangul Word Processor) file, version 5.0

file_buffer calls file_trycdf before the soft-magic pass, and a hit ends the run unless
MAGIC_CONTINUE is set (file/src/funcs.c:443-452). file_trycdf parses the compound document,
reads its FileHeader user stream, compares the first bytes against the HWP Document File
signature, and prints the description itself (file/src/readcdf.c:633-646):

	if (cdf_read_user_stream(&info, &h, &sat, &ssat, &sst, &dir,
	    "FileHeader", &scn) != -1) {
#define HWP5_SIGNATURE "HWP Document File"
		if (scn.sst_len * scn.sst_ss >= sizeof(HWP5_SIGNATURE) - 1
		    && memcmp(scn.sst_tab, HWP5_SIGNATURE,
		    sizeof(HWP5_SIGNATURE) - 1) == 0) {
		    if (NOTMIME(ms)) {
			if (file_printf(ms,
			    "Hancom HWP (Hangul Word Processor) file, version 5.0") == -1)

The definitions' only version 5.0 entry is polyfile/magic_defs/ole2compounddocs:269. It matches
the directory entry name FileHeader as a lestring16 relative to the OLE 2 header, so it carries
the : continuation prefix and the OLE 2 description ahead of it, and it declares
!:mime application/hwp+zip where readcdf.c emits application/x-hwp. PolyFile reports exactly
what the definitions say; what it lacks is the built-in reader that short-circuits them.

Running file with -k shows all three matches, and PolyFile reports the middle one:

$ TZ=UTC ./file/src/file -b -k -m file/magic/magic.mgc file/tests/HWP2016.hwp.testfile
Hancom HWP (Hangul Word Processor) file, version 5.0\012- OLE 2 Compound Document, v3.62, SecID 0x2, Mini FAT start sector 0x6 : Hancom HWP (Hangul Word Processor) file, version 5.0\012- data

Consequences

tests/test_magic.py:corpus_result_matches carries an escape hatch for this one stem: it accepts a
suffix match instead of an exact one, because the expected description is the tail of what PolyFile
reports. #3501 documents that branch and points here for the cause; the branch has to stay until
PolyFile can read compound documents.

HWP2016.hwp is the only compound document in the libmagic corpus, so the corpus does not measure
how much else readcdf.c affects. Its other outputs include the summary-information fields
(CDF V2 Document, ...), the CLSID line, and the Microsoft ... verdicts it derives from the
document's own streams.

Scope

libmagic's reader is file/src/cdf.c (1682 lines), file/src/cdf.h (363), and
file/src/readcdf.c (702) — a subsystem, not a definition fix, so this needs a decision about
whether PolyFile wants a compound-document reader at all. The narrow version, enough for this
divergence, is a reader that walks the directory, extracts the FileHeader stream, and matches its
signature; the full version reproduces the summary-information parsing too.

Precedent: #3488 added the equivalent of file_ascmagic for the same class of gap.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions