Repository navigation
Fix FRAGPIPE_HEADLESS manifest, lint, test profiles and DIA post-processing; emit pg_matrix/diann_log - #3
Merged
an-altosian merged 13 commits intoOct 2, 2026
Conversation
Expose DIANN_FINALQUANTIFICATION pg_matrix and log as subworkflow emits so downstream QPX/AnnData export can consume them. Recorded as an nf-core patch (dia_proteomics_analysis.diff, modules.json patch entry).
…exactly manifest_content was declared but never read: every LFQ file fell back to experiment1 / bioreplicate 1 / DDA, so allinone runs collapsed all samples into one experiment and lost DDA+ and empty (fraction) bioreplicates. - manifest_content is authoritative when non-empty (file, experiment, bioreplicate, data type); file_experiment_map is the fallback (bioreplicate 1, DDA) - file names are matched on the whole first column instead of grep -F, which gave A1.mzML the experiment of XA1.mzML - fail loudly on a staged file without a row, a row for an unstaged file, duplicate file names, a TMT experiment missing from annotation_content, or both manifest_content and file_experiment_map empty - pass pasted values base64-encoded so they can neither expand in bash nor break the script indentation (versions.yml carried a stray END_VERSIONS) - single-plex TMT uses the per-experiment subdirectory layout of multi-plex - stub writes results/fragpipe-files.fp-manifest and results/experiment_annotation.tsv - tests: fake-FragPipe tests of the real script block (DDA+ manifest, fractions, exact file_experiment_map match, missing row, empty inputs)
…cs_pipeline `nf-core pipelines lint` (tools 3.5.2) crashed with `IndexError: list index out of range` in nf_core/subworkflows/lint/main_nf.py (`self.workflow_name = lines[0].split()[1]`): every entry in subworkflows/local/ is linted as a local subworkflow, and subworkflows/local/fragpipe_utils.nf is a functions-only file with no `workflow` block, so no workflow line was ever collected. It crashes the same way on unmodified dev. - Move shouldRunTool, getToolArgs, getToolModmasses, getToolField, getToolReportArgs and generateFragpipeManifest into the FUNCTIONS section of subworkflows/local/utils_nfcore_msproteomics_pipeline/main.nf, using fully-qualified groovy.json names (strict syntax rejects `import`). - Repoint all includes; delete subworkflows/local/fragpipe_utils.nf. - Delete lib/FragpipeUtils.nf: an unused, never-included stale copy.
nf-core's rocrate_readme_sync lint check compares the RO-Crate description with README.md; the committed description was stale.
- nextflow_config: params.diann_additional_analysis_args = null in nextflow.config and no schema default (nf-core fetch_wf_config maps a config '' to "null", so '' vs "" can never match); consumers in conf/modules.config all use `?: ''`, so behaviour is unchanged. - nextflow_config / schema_params: drop max_cpus, max_memory, max_time from nextflow_schema.json and the test profile; limits come from process.resourceLimits and no code reads params.max_*. - schema_params: drop params.proteomes = null from conf/reference_proteomes_ignored.config; nothing reads params.proteomes. - check_local_copy (utils_nfcore_pipeline): restore tests/main.nf.test to nf-core/modules f0b535b and remove the stale tests/main.workflow.nf.test (+ snap), renamed upstream in that commit. The remaining main.nf delta (imNotification kept for hook_url) must be recorded with `nf-core subworkflows patch utils_nfcore_pipeline`.
main.nf re-adds imNotification (removed upstream in nf-core/modules f0b535b), which the hook_url feature needs; nf-core lint check_local_copy failed without a recorded patch.
subworkflows/local/{utils_nfcore_pipeline,utils_nextflow_pipeline,utils_pipeline} were only included by each other; the pipeline uses the nf-core copies and utils_nfcore_msproteomics_pipeline (flagged by nf-core pipelines lint).
…ng; run integration test in the FragPipe container
…ms.databases)
The docs promised that --organism without --database downloads the UniProt
reference proteome, but the workflows read params.databases[organism].database
while conf/reference_proteomes.config defined an unused params.proteomes map
(snake_case keys, `fasta` field) and params.proteomes_ignore defaulted to true.
Every run without --database therefore failed.
- Mirror the nf-core igenomes pattern: proteomes_ignore defaults to false,
proteomes_base holds the UniProt REST prefix (schema default matches),
conf/reference_proteomes.config defines params.databases keyed by the
standard organism name ('Homo sapiens', ...) with a `database` URL,
conf/reference_proteomes_ignored.config sets params.databases = [:], and
`databases` is in validation.defaultIgnoreParams.
- getProteinDatabase() (utils_nfcore_msproteomics_pipeline): --database is
authoritative, else the organism's reference proteome, else fail with a
message naming --database and the available organisms. Used by DIA-NN;
WorkflowUtils.resolveFragPipeDatabase raises the same message.
- Channel.fromPath(..., glob: false) for the database: with globbing the `?`
in the UniProt URL dropped the whole query string.
- Function tests for the three resolution cases; docs and CHANGELOG updated.
…g protein-group FDR conf/test.config was the untouched nf-core template (viralrecon samplesheet, genome R64-1-1); -profile test is now a minimal DIA-NN test. test_tmtq searched the Erwinia TMT6 data against the human FASTA and had no FragPipe workflow; it now uses erwinia_carotovora.fasta and workflows/TMT6.workflow. test_tmt sets tmt_type TMT6; test_dda_lfq sets fragpipe_workflow explicitly. DIANNR and IQ no longer crash (subscript out of bounds / MissingFileException) when DIA-NN reports PG.Q.Value = 1 for every row on a small search space: they write header-only tables with a WARNING and fail loudly on an empty report. All test_* profiles and test verified end to end against the test data in nf-core/test-datasets#1953 (updated).
Member
|
Warning Newer version of the nf-core template is available. Your pipeline is using an old version of the nf-core template: 3.5.2. For more documentation on how to update your pipeline, please see the nf-core documentation and Synchronisation documentation. |
…pace) The linting workflow's pre-commit job has failed on dev since 2026-03-25. Formatting only: modules.json content is unchanged, and the 19 .nf files differ only in trailing whitespace on blank lines (git diff -w is empty).
nf-test.config loads profile 'test', and conf/test.config now sets params.database, so the organism-only and unknown-organism cases resolved to that database instead of the UniProt reference proteome (CI nf-test shards 3/7, 4/7). Local runs with --profile docker replaced the default profile and hid it; verified with --profile +docker (96/96 stubs).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes found while using msproteomics as the backend of a downstream pipeline (DIA-NN / FragPipe export to AnnData/MuData), plus everything
nf-core pipelines lintand the test profiles turned up. Every-profile test*now runs end to end.Depends on nf-core/test-datasets#1953 (msproteomics test data): the test profiles read their inputs from that branch.
Changes
dia_proteomics_analysis(nf-core subworkflow, recorded as an nf-core patch inmodules.json): emitpg_matrixanddiann_logfromDIANN_FINALQUANTIFICATION.FRAGPIPE_HEADLESS:manifest_contentwas declared but unused, so every file becameexperiment1/ bioreplicate 1 /DDA. It is now authoritative (keepsDDA+and empty fraction bioreplicates); file names match exactly (A1.mzMLno longer takesXA1.mzML's experiment); a staged file without a row, a row naming an unstaged file, duplicates, or both inputs empty fail the task.nf-core pipelines lint(tools 3.5.2 as pinned) crashed withIndexErrorin subworkflow linting:subworkflows/local/fragpipe_utils.nf, a functions-only file, was linted as a subworkflow. The helpers now live inutils_nfcore_msproteomics_pipeline; the unusedlib/FragpipeUtils.nfcopy and the unused duplicatesubworkflows/local/utils_*subworkflows are removed.diann_additional_analysis_argsdefault mismatch;max_cpus/max_memory/max_timeremoved from the schema (resourceLimits); unusedparams.proteomes;utils_nfcore_pipelinelocal edit (re-addsimNotificationforhook_url) recorded as an nf-core patch;ro-crate-metadata.jsonsynced with the README.--organismwithout--database) was never wired up (params.databaseswas undefined). It now follows the nf-core igenomes pattern (proteomes_ignore,conf/reference_proteomes.config);--databasestays authoritative; an unknown organism fails with a clear message.Channel.fromPathusedglob: falseso the UniProt query string survives.MERGE_SPLIT_SEARCH: fails clearly when the MSFragger jar is missing (it builtjava -jarwith an empty path); the integration test runs in a FragPipe container; the stub snapshot no longer captures the host Python version.DIANNR/IQ: no longer crash (subscript out of bounds/ missingmaxlfq.tsv) when DIA-NN reportsPG.Q.Value = 1for every row on a small search space; they write header-only tables with aWARNINGand fail loudly on an empty report.DIANNRalso no longer writes silent 0-byte tables.conf/test.configwas the untouched template (viralrecon samplesheet,genome = 'R64-1-1');-profile testis now a minimal DIA-NN test.test_tmtqsearched the Erwinia TMT6 data against the human FASTA and had no FragPipe workflow; it now useserwinia_carotovora.fastaandworkflows/TMT6.workflow.Testing
nf-core pipelines lint(3.5.2): 658 passed, 0 failed.nf-test test --tag stub --profile docker: full stub suite.fragpipe_headless(6),merge_split_search(3, integration in a FragPipe image),diannr(3),iq(3).test(44 protein groups),test_dia(23),test_dda_lfq(146 proteins),test_tmt,test_tmtq(210 proteins x 12 channels).