Read the real file_name in BAM/VCF/FASTQ/FASTA/BED classifiers (#152)

[NoopDog]

Closes #152

What changed

classify_from_header (BAM/CRAM) and classify_from_vcf_header accepted a file_name parameter but never read it — each synthesized sample{ext} from file_format alone, so every tier-2 filename rule matched the synthetic name and any signal the real name carried (hifi_reads, rnaseq, chm13) was silently dropped for the two most common file types in the corpus. Both now use the real file_name.

Name-only, no file_format signal. An earlier cut routed all classifiers through the filename_for_rules helper, which grafts file_format as a fallback extension. Per #157 (AnVIL file_format is redundant with the name), that path was removed. The graft was measured to never fire anyway: 0 of 254,478 cached records have a file_name lacking a usable extension.

Pass the known extension explicitly, don't fabricate a filename. The classifiers used to invent a placeholder name ("sample.bam") purely to carry the file-type extension into the engine, because RuleEngine.classify_extended derived the extension from the filename. That's now decoupled: the engine derives the extension from the filename when one is present, and otherwise uses the file_format the caller set. So all five header classifiers (BAM/VCF/FASTQ/FASTA/BED) build ExtendedFileInfo(filename=file_name or "", file_format="<type ext>") — the real filename drives the tier-2 rules when present, and when it is absent the extension rules run off the explicit, known extension instead of a made-up name.

Nameless files never reach a classifier in production regardless — they fail the input contract (file_name pattern ^.+$) and divert to validation_failed — so the no-name path is only exercised by header-only callers/tests.

Why

~8% of BAM files were left not_classified on three dimensions the filename would settle. This is a coverage defect, not a correctness one — nothing wrong was emitted, but the coverage was recoverable for free. Consistent with accuracy over coverage: the filename never contradicts a header-derived value, it only fills gaps.

Assumptions I made

• file_format carries no classification signal the name lacks (Classify Salmon quant.sf as transcriptomic quantification (from the name; do not route through AnVIL file_format) #157); the fallback graft never fires on the corpus, so name-only loses nothing.
• The sample.rnaseq.bam golden fixture flip is intended (issue task Determine/validate the approach for representing "NA" (not_available) vs. unspecified or not_applicable in TDR #2): it now classifies transcriptomic.bulk / alignments / RNA-seq instead of not_classified.
• The engine's extension derivation change is behavior-preserving for every caller that passes a real filename (the entire production path and all FileInfo-based callers); only the header-only path (no filename) now reads the explicit file_format.

How to verify

Definition of done from #152:

• Use file_name when present in classify_from_header and classify_from_vcf_header. See src/meta_disco/header_classifier.py.
• Regenerate the golden — python -m tests.test_output_shape; sample.rnaseq.bam now classifies transcriptomic.bulk.
• Tests — tests/test_header_classifier.py: real filename drives tier-2 rules (BAM rnaseq → transcriptomic, hifi → PACBIO; VCF chm13 → CHM13), and a header aligner signal still wins where the filename is silent.
• Re-run the corpus — over the 16,409 cached BAM headers: 1,371 files (8.4%) move off not_classified for data_modality/data_type/assay_type, 110 recover platform, 0 value regressions.

To reproduce locally:

make test          # 739 pass, incl. the new filename tests and the golden
make lint
make format-check

[Copilot]

Pull request overview

This PR fixes a coverage gap in meta-disco’s header-based classifiers by ensuring the real file_name (when provided) is used to drive tier-2 filename rules across BAM/VCF/FASTQ/FASTA/BED, with a safe fallback that grafts file_format only when the name lacks a usable extension. It also centralizes per-file-type extension tuples alongside the classifiers that rely on them to prevent drift between pipeline selection and classifier expectations.

Changes:

• Route BAM/VCF/FASTQ/FASTA/BED header classifiers through filename_for_rules(file_name, file_format, ...) so real filenames influence tier-2 rules.
• Define file-type extension tuples in header_classifier.py and import them into file_types.py (keeping selection and “trusted extension” logic aligned).
• Add/adjust tests and golden fixture output to reflect recovered classifications (notably sample.rnaseq.bam).

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
src/meta_disco/header_classifier.py	Uses filename_for_rules across classifiers; introduces shared extension tuples; threads file_format into BED classifier.
src/meta_disco/file_types.py	Imports extension tuples from header_classifier.py to keep pipeline selection aligned with classifier expectations.
scripts/fetch_bed_headers.py	Threads file_format into BED classification and reuses BED_EXTENSIONS for filtering (but currently misses file_format-based selection).
scripts/classify_bed_files.py	Reuses BED_EXTENSIONS for BED selection and passes file_format into BED classification.
tests/test_header_classifier.py	Adds tests asserting real filenames drive tier-2 rules (BAM rnaseq/hifi; VCF chm13) and that grafting works.
tests/test_bed_classification.py	Adds a test ensuring file_format rescues names lacking a .bed extension.
tests/fixtures/golden/expected_output.json	Updates golden output to reflect newly classified fields driven by the real filename.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

src/meta_disco/header_classifier.py:112

• classify_from_header() now ignores file_format entirely when choosing the synthetic filename. If callers pass file_format but omit/empty file_name (e.g. ClassifyPipeline.classify_single(file_name="", file_format=".cram")), the classifier will always behave like a BAM (sample.bam), and infer_assay_type() can’t apply the CRAM-specific size thresholds because ExtendedFileInfo.file_format is left unset. Consider falling back to sample{file_format} only when file_name is falsy, and set ExtendedFileInfo.file_format from the derived extension so assay inference can distinguish BAM vs CRAM without treating AnVIL file_format as an independent signal.

    # Use the real filename so its tokens reach the tier-2 filename rules. The
    # AnVIL file_format is not consulted: it is redundant with the name for
    # classification (#157), and in the corpus every selected BAM record's
    # file_name already carries a usable extension (#152).
    filename = file_name or "sample.bam"

src/meta_disco/header_classifier.py:199

• classify_from_vcf_header() now defaults to sample.vcf.gz whenever file_name is missing/empty, even if the caller provided file_format (previously this produced sample{file_format}). That can change which extension-scoped rules run for callers that only know file_format (e.g. .vcf vs .vcf.gz). Suggest keeping the same fallback behavior as before: use file_name when present, else synthesize sample{file_format} when it looks like an extension, else use the hard-coded default.

    # Use the real filename so its tokens reach the tier-2 filename rules. The
    # AnVIL file_format is not consulted: it is redundant with the name for
    # classification (#157), and in the corpus every selected VCF record's
    # file_name already carries a usable extension (#152).
    filename = file_name or "sample.vcf.gz"

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (2)

src/meta_disco/header_classifier.py:115

• classify_from_header now hard-codes file_format=".bam" and ignores the file_format argument. That breaks the documented header-only fallback behavior (e.g., a CRAM header-only call passing file_format=".cram" will still run BAM extension rules) and also makes the preceding comment inaccurate (“known .bam”). Use the provided file_format only as an extension fallback when file_name is empty, defaulting to ".bam".

    # the file_format we set — the known ".bam" — instead of a fabricated name.
    file_info = ExtendedFileInfo(
        filename=file_name or "",
        file_format=".bam",
        file_size=file_size,

src/meta_disco/header_classifier.py:200

• classify_from_vcf_header now hard-codes file_format=".vcf.gz" and ignores the file_format argument. For header-only calls this prevents callers from selecting ".vcf" vs ".vcf.gz" extension rules, and it makes the comment about the “known .vcf.gz” fallback too strong. Use file_format as the extension fallback when file_name is empty, defaulting to ".vcf.gz".

    # the file_format we set — the known ".vcf.gz" — instead of a fabricated name.
    file_info = ExtendedFileInfo(
        filename=file_name or "",
        file_format=".vcf.gz",
        file_size=file_size,

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

src/meta_disco/header_classifier.py:112

• classify_from_header still accepts file_format (".bam"/".cram"), but the new code hard-codes file_format=".bam" on ExtendedFileInfo. This breaks the documented fallback behavior for header-only/extensionless calls (and makes the surrounding comment inaccurate), because a caller-provided .cram fallback will be ignored and extension-scoped rules will run as .bam.

    # Use the real filename so its tokens reach the tier-2 filename rules. The
    # AnVIL file_format is redundant with the name and not consulted (#157). When
    # there is no name (a header-only call), the engine reads the extension from
    # the file_format we set — the known ".bam" — instead of a fabricated name.
    file_info = ExtendedFileInfo(

src/meta_disco/header_classifier.py:197

• classify_from_vcf_header still accepts file_format (".vcf"/".vcf.gz"), but the new code hard-codes file_format=".vcf.gz" on ExtendedFileInfo. That makes header-only/extensionless calls ignore a caller-provided .vcf fallback and contradicts the intent described in the comment (use the caller’s known extension when no filename is present).

    # Use the real filename so its tokens reach the tier-2 filename rules. The
    # AnVIL file_format is redundant with the name and not consulted (#157). When
    # there is no name (a header-only call), the engine reads the extension from
    # the file_format we set — the known ".vcf.gz" — instead of a fabricated name.
    file_info = ExtendedFileInfo(