Fix FRAGPIPE_HEADLESS manifest, lint, test profiles and DIA post-processing; emit pg_matrix/diann_log

CHANGELOG.md

@@ -3,6 +3,31 @@
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## Unreleased
+
+### Fixed
+
+- `IQ`: no longer fails with "Missing output file(s) `maxlfq.tsv`" when no report row passes the FDR filter (`Q.Value`, `PG.Q.Value`, `Lib.Q.Value`, `Lib.PG.Q.Value`; e.g. `PG.Q.Value = 1` for every row on a 500-protein FASTA): `process_long_format()` wrote no file and R exited 0. A header-only `maxlfq.tsv` (`Protein.Group`, `Protein.Names`, `Genes`, the samples) is now written with a `WARNING: IQ: maxlfq.tsv: no report row passes ...` line giving the filter and the minimum q-values. A report with no rows, or with only contaminant rows, fails with `ERROR: IQ: ...`, and so does a run where rows pass but no `maxlfq.tsv` is written. New integration tests reuse the `DIANNR` 10-row report fixture; the `DIANNR` test now reads the task's `.command.err` (not the Nextflow console) for the warning.
+- `DIANNR`: no longer fails with `subscript out of bounds` in `diann_maxlfq` when no report row passes the FDR filter (e.g. DIA-NN reports `PG.Q.Value = 1` for every row on a small search space such as a 500-protein FASTA). `diann_maxlfq` is only called when rows pass; otherwise, and for the `diann_matrix` outputs that were silently written as 0-byte files, a header-only table (the sample columns) is written with a `WARNING: DIANNR: <file>: no report row passes ...` line. A report with no rows, or with only contaminant rows, fails with `ERROR: DIANNR: ...`; an empty table from rows that did pass fails too. New integration tests with a 10-row report fixture.
+- `MERGE_SPLIT_SEARCH`: the task fails at once with `ERROR: MERGE_SPLIT_SEARCH: no MSFragger*.jar found` (naming the searched `msfragger_dir` and `ext.fragpipe_tools_dir`) when no MSFragger jar is found; before, `java -jar` got an empty jar path and reported a misleading "Invalid or corrupt jarfile <histogram>.tsv". `merge_split_search.py` also rejects a `--msfragger_cmd` whose `-jar` path is missing.
+- `MERGE_SPLIT_SEARCH` tests: the integration test runs in the FragPipe image given by `FRAGPIPE_CONTAINER` (it ran on the host before); a new stub-tagged test covers the missing-jar error; the stub snapshot no longer records the host Python version.
+- `FRAGPIPE_HEADLESS`: `manifest_content` is used again and is authoritative (file, experiment, bioreplicate, data type), so allinone runs keep one experiment per sample, empty bioreplicates for fractions, and `DDA+`; before, every LFQ file fell back to `experiment1` / `1` / `DDA`.
+- `FRAGPIPE_HEADLESS`: file names are matched exactly against the manifest (or `file_experiment_map`) first column; the substring lookup (`grep -F`) gave `A1.mzML` the experiment of `XA1.mzML`.
+- `FRAGPIPE_HEADLESS`: the task fails loudly when a staged file has no row, a row names an unstaged file, a file is listed twice, a TMT file's experiment is not in `annotation_content`, or both `manifest_content` and `file_experiment_map` are empty (TMT files were silently dropped before).
+- `FRAGPIPE_HEADLESS`: `versions.yml` no longer carries a stray `END_VERSIONS` line when a multi-line value was pasted into the script (inputs are now passed base64-encoded, so no value can break indentation or expand in bash).
+- `nf-core pipelines lint`: `params.diann_additional_analysis_args` defaults to `null` (schema has no default); nf-core reads a config `''` as `null`, so `''` could never match the schema. All consumers use `?: ''`, so behaviour is unchanged.
+- `nf-core pipelines lint`: removed the `max_cpus` / `max_memory` / `max_time` schema params and the `test` profile's `params.max_*`; resource caps are set by `process.resourceLimits`, and nothing read the params.
+- `nf-core pipelines lint`: removed `params.proteomes = null` from `conf/reference_proteomes_ignored.config`; no code reads `params.proteomes`.
+- `utils_nfcore_pipeline`: tests match the installed nf-core/modules commit (`tests/main.nf.test` restored; stale `tests/main.workflow.nf.test` and its snapshot removed).
+- Reference proteomes: the documented UniProt download now works. `conf/reference_proteomes.config` defines `params.databases` (keyed by organism, e.g. `'Homo sapiens'`, with a `database` URL), which is what the workflows read; before it defined an unused `params.proteomes` and `params.proteomes_ignore` defaulted to `true`, so without `--database` every run failed. Mirrors nf-core iGenomes: `proteomes_ignore` defaults to `false`, `proteomes_base` holds the UniProt URL prefix, `conf/reference_proteomes_ignored.config` sets `params.databases = [:]`, and `databases` is in `validation.defaultIgnoreParams`. `--database` stays authoritative; an organism with no proteome and no `--database` fails with `No protein database for organism '<organism>': ...` listing the available organisms (`getProteinDatabase`, new function tests).
+- Database channels use `Channel.fromPath(..., glob: false)`; with globbing, the `?` in a UniProt REST URL dropped the query string.
+
+### Changed
+
+- `FRAGPIPE_HEADLESS`: single-plex TMT files are placed in `raw_files/<experiment>/` like multi-plex ones.
+- `FRAGPIPE_HEADLESS` stub writes `results/fragpipe-files.fp-manifest` and `results/experiment_annotation.tsv`.
+- FragPipe helper functions (`shouldRunTool`, `getToolArgs`, `getToolModmasses`, `getToolField`, `getToolReportArgs`, `generateFragpipeManifest`) moved from `subworkflows/local/fragpipe_utils.nf` into `subworkflows/local/utils_nfcore_msproteomics_pipeline/main.nf`; `nf-core pipelines lint` crashed (`IndexError` in the subworkflow `main_nf` check) on the functions-only file. The unused duplicate `lib/FragpipeUtils.nf` is removed.
+
 ## v1.0.0 - Initial Release
 
 ### Added

README.md

@@ -37,12 +37,12 @@ Instrument-specific settings are applied via optional config files (`-c conf/ins
 The `--mode` parameter selects the analysis engine.
 FragPipe sub-modes are controlled with `--tmt_mode` for TMT workflows.
 
-| Mode | Sub-mode | Description | Engine |
-| --- | --- | --- | --- |
-| `diann` | -- | DIA quantitative proteomics (standard, phospho) | [DIA-NN](https://github.com/vdemichev/DiaNN) |
-| `fragpipe` | *(none)* | Generic FragPipe workflows, configured by [workflow files](https://github.com/Nesvilab/FragPipe/tree/develop/workflows) | [FragPipe](https://fragpipe.nesvilab.org/) |
-| `fragpipe` | `--tmt_mode labelcheck` | TMT labeling efficiency QC | [FragPipe](https://fragpipe.nesvilab.org/) |
-| `fragpipe` | `--tmt_mode quant` | TMT isobaric quantification | [FragPipe](https://fragpipe.nesvilab.org/) |
+| Mode       | Sub-mode                | Description                                                                                                             | Engine                                       |
+| ---------- | ----------------------- | ----------------------------------------------------------------------------------------------------------------------- | -------------------------------------------- |
+| `diann`    | --                      | DIA quantitative proteomics (standard, phospho)                                                                         | [DIA-NN](https://github.com/vdemichev/DiaNN) |
+| `fragpipe` | _(none)_                | Generic FragPipe workflows, configured by [workflow files](https://github.com/Nesvilab/FragPipe/tree/develop/workflows) | [FragPipe](https://fragpipe.nesvilab.org/)   |
+| `fragpipe` | `--tmt_mode labelcheck` | TMT labeling efficiency QC                                                                                              | [FragPipe](https://fragpipe.nesvilab.org/)   |
+| `fragpipe` | `--tmt_mode quant`      | TMT isobaric quantification                                                                                             | [FragPipe](https://fragpipe.nesvilab.org/)   |
 
 DIA method variants (e.g., phospho) and instrument-specific settings are applied via `-c` config files.
 See [docs/usage.md](docs/usage.md) for full details.

conf/reference_proteomes.config

@@ -2,58 +2,61 @@
 ========================================================================================
     Reference Proteomes Configuration
 ========================================================================================
-    UniProt reference proteomes (Swiss-Prot reviewed + isoforms).
+    UniProt reference proteomes (Swiss-Prot reviewed + isoforms), keyed by the
+    organism name that --organism resolves to (see
+    WorkflowUtils.convertOrganismToStandardName). Loaded unless
+    --proteomes_ignore is set; the FASTA is downloaded from the UniProt REST API
+    at runtime when --database is not given. Mirrors the nf-core igenomes pattern
+    (params.genomes / conf/igenomes.config).
     These do NOT include contaminant sequences. For FragPipe workflows,
     contaminants are added automatically by Philosopher's database command.
     For DIA-NN workflows, append contaminants to your FASTA manually or
     use --database with a pre-prepared database that includes contaminants.
-    Override with --database to provide your own FASTA file.
+    --database always takes precedence over this map.
 ========================================================================================
 */
 
 params {
-    proteomes_base = 'https://rest.uniprot.org/uniprotkb/stream?format=fasta&includeIsoform=true&query=(reviewed:true)+AND+(proteome:'
-
-    proteomes {
-        'homo_sapiens' {
+    databases {
+        'Homo sapiens' {
             proteome_id = 'UP000005640'
             taxon_id    = 9606
-            fasta       = "${params.proteomes_base}UP000005640)"
+            database    = "${params.proteomes_base}UP000005640)"
         }
-        'mus_musculus' {
+        'Mus musculus' {
             proteome_id = 'UP000000589'
             taxon_id    = 10090
-            fasta       = "${params.proteomes_base}UP000000589)"
+            database    = "${params.proteomes_base}UP000000589)"
         }
-        'rattus_norvegicus' {
+        'Rattus norvegicus' {
             proteome_id = 'UP000002494'
             taxon_id    = 10116
-            fasta       = "${params.proteomes_base}UP000002494)"
+            database    = "${params.proteomes_base}UP000002494)"
         }
-        'saccharomyces_cerevisiae' {
+        'Saccharomyces cerevisiae' {
             proteome_id = 'UP000002311'
             taxon_id    = 559292
-            fasta       = "${params.proteomes_base}UP000002311)"
+            database    = "${params.proteomes_base}UP000002311)"
         }
-        'danio_rerio' {
+        'Danio rerio' {
             proteome_id = 'UP000000437'
             taxon_id    = 7955
-            fasta       = "${params.proteomes_base}UP000000437)"
+            database    = "${params.proteomes_base}UP000000437)"
         }
-        'drosophila_melanogaster' {
+        'Drosophila melanogaster' {
             proteome_id = 'UP000000803'
             taxon_id    = 7227
-            fasta       = "${params.proteomes_base}UP000000803)"
+            database    = "${params.proteomes_base}UP000000803)"
         }
-        'caenorhabditis_elegans' {
+        'Caenorhabditis elegans' {
             proteome_id = 'UP000001940'
             taxon_id    = 6239
-            fasta       = "${params.proteomes_base}UP000001940)"
+            database    = "${params.proteomes_base}UP000001940)"
         }
-        'escherichia_coli' {
+        'Escherichia coli' {
             proteome_id = 'UP000000625'
             taxon_id    = 83333
-            fasta       = "${params.proteomes_base}UP000000625)"
+            database    = "${params.proteomes_base}UP000000625)"
         }
     }
 }

conf/reference_proteomes_ignored.config

@@ -1,12 +1,9 @@
 /*
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-    Nextflow config file for iProteomes paths
+    Nextflow config file for UniProt reference proteomes
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-    Empty proteomes dictionary to use when iproteomes is ignored.
+    Empty databases dictionary to use when --proteomes_ignore is set.
 ----------------------------------------------------------------------------------------
 */
 
-params {
-    proteomes_base = null
-    proteomes      = null
-}
+params.databases = [:]

conf/test.config

@@ -2,7 +2,10 @@
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     Nextflow config file for running minimal tests
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-    Defines input files and everything required to run a fast and simple pipeline test.
+    Defines input files and everything required to run a fast and simple pipeline test:
+    a DIA-NN library-free search of one CPTAC DIA subset (spectra/dia_cptac_subset.mzML)
+    against a 1,000-entry human SwissProt subset, with fast search params
+    (charge 2-3, no missed cleavages, no variable mods).
 
     Use as follows:
         nextflow run nf-core/msproteomics -profile test,<docker/singularity> --outdir <OUTDIR>
@@ -16,15 +19,33 @@ process {
         memory: '15.GB',
         time: '1.h'
     ]
+    cpus   = 2
+    memory = '6.GB'
+    time   = '1.h'
 }
 
 params {
     config_profile_name        = 'Test profile'
     config_profile_description = 'Minimal test dataset to check pipeline function'
 
-    // Input data
-    // TODO nf-core: Specify the paths to your test data on nf-core/test-datasets
-    // TODO nf-core: Give any required params for the test so that command line flags are not needed
-    input  = params.pipelines_testdata_base_path + 'viralrecon/samplesheet/samplesheet_test_illumina_amplicon.csv'// Genome references
-    genome = 'R64-1-1'
+    test_mode = true
+
+    // Input data (absolute URLs: tests/nextflow.config redefines pipelines_testdata_base_path)
+    mode     = 'diann'
+    input    = 'https://raw.githubusercontent.com/nf-core/test-datasets/refs/heads/msproteomics/samplesheets/test_dia.csv'
+    database = 'https://raw.githubusercontent.com/nf-core/test-datasets/refs/heads/msproteomics/databases/human_sp_subset.fasta'
+    organism = 'homo_sapiens'
+
+    // Fast DIA-NN search for a 5-min FAIMS gradient
+    diann_min_pr_mz          = 350
+    diann_max_pr_mz          = 950
+    min_peptide_length       = 7
+    max_peptide_length       = 25
+    min_precursor_charge     = 2
+    max_precursor_charge     = 3
+    allowed_missed_cleavages = 0
+    max_mods                 = 0
+    diann_performance_mode   = true
+    diann_quick_mass_acc     = true
+    skip_post_msstats        = true
 }

conf/tests/test_dda_lfq.config

@@ -18,5 +18,6 @@ params {
     input          = 'https://raw.githubusercontent.com/nf-core/test-datasets/refs/heads/msproteomics/samplesheets/test_dda_lfq.csv'
     database       = 'https://raw.githubusercontent.com/nf-core/test-datasets/refs/heads/msproteomics/databases/human_sp_subset.fasta'
     organism       = 'homo_sapiens'
+    fragpipe_workflow = "${projectDir}/assets/LFQ-MBR.workflow"
     outdir         = 'results'
 }

conf/tests/test_tmt.config

@@ -16,6 +16,7 @@ params {
 
     mode           = 'fragpipe'
     tmt_mode       = 'labelcheck'
+    tmt_type       = 'TMT6'
     input          = 'https://raw.githubusercontent.com/nf-core/test-datasets/refs/heads/msproteomics/samplesheets/test_tmt.csv'
     database       = 'https://raw.githubusercontent.com/nf-core/test-datasets/refs/heads/msproteomics/databases/erwinia_carotovora.fasta'
     organism       = 'erwinia_carotovora'

conf/tests/test_tmtq.config

@@ -2,7 +2,8 @@
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     Nextflow config file for TMT quantification workflow tests
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-    Uses FragPipe TMT quantification with small dataset.
+    Uses FragPipe TMT quantification on two TMT6 Erwinia plexes (PRIDE PXD000001).
+    Requires --fragpipe_container (licensed FragPipe image).
 
     Use as follows:
         nextflow run nf-core/msproteomics -profile test_tmtq,docker --outdir <OUTDIR>
@@ -17,7 +18,8 @@ params {
     mode           = 'fragpipe'
     tmt_mode       = 'quant'
     input          = 'https://raw.githubusercontent.com/nf-core/test-datasets/refs/heads/msproteomics/samplesheets/test_tmtq.csv'
-    database       = 'https://raw.githubusercontent.com/nf-core/test-datasets/refs/heads/msproteomics/databases/human_sp_subset.fasta'
-    organism       = 'homo_sapiens'
+    database       = 'https://raw.githubusercontent.com/nf-core/test-datasets/refs/heads/msproteomics/databases/erwinia_carotovora.fasta'
+    organism       = 'erwinia_carotovora'
+    fragpipe_workflow = 'https://raw.githubusercontent.com/nf-core/test-datasets/refs/heads/msproteomics/workflows/TMT6.workflow'
     outdir         = 'results'
 }

docs/database_guide.md

@@ -4,7 +4,7 @@
 
 Protein identification in mass spectrometry proteomics requires a FASTA database of protein sequences to search spectra against.
 The nf-core/msproteomics pipeline supports both pre-built reference proteomes and custom user-provided databases.
-Pre-built databases are fetched automatically from UniProt based on organism metadata in the samplesheet, while custom databases can be supplied via the `--database` parameter.
+When `--database` is not given, the UniProt reference proteome for `--organism` is downloaded automatically at runtime; a custom database supplied via `--database` always takes precedence.
 
 ## Pre-Built Reference Proteomes
 
@@ -24,16 +24,26 @@ These databases contain Swiss-Prot reviewed proteins with isoforms, fetched from
 
 ### How pre-built databases are selected
 
-The pipeline auto-selects the correct reference proteome based on the organism metadata in your samplesheet SDRF.
-No additional parameters are needed when working with one of the eight supported organisms.
-If the organism in your samplesheet matches a pre-built proteome, the pipeline downloads the FASTA automatically at runtime.
+The reference proteome is selected by the `--organism` parameter (default `Homo sapiens`), not by the samplesheet.
+The aliases `human`/`hs`, `mouse`/`mm` and `yeast`/`sc` map to _Homo sapiens_, _Mus musculus_ and _Saccharomyces cerevisiae_.
+Any other organism must be given exactly as in the table above (e.g. `--organism 'Danio rerio'`).
+When `--database` is not given, the pipeline downloads the FASTA for that organism from UniProt at runtime.
 
-You can override the automatic selection by providing `--database /path/to/custom.fasta`.
+Resolution order:
+
+| Priority | Source                                                                         |
+| -------- | ------------------------------------------------------------------------------ |
+| 1        | `--philosopher_database` (FragPipe mode only)                                  |
+| 2        | `--database`                                                                   |
+| 3        | `params.databases[<organism>].database` from `conf/reference_proteomes.config` |
+
+If none applies (an organism outside the table and no `--database`), the pipeline stops before any task runs with `No protein database for organism '<organism>': pass --database <fasta> ...`, listing the available organisms.
 
 ### Configuration
 
-Pre-built proteome URLs are defined in `conf/reference_proteomes.config`.
-Each entry uses the UniProt REST API endpoint:
+Pre-built proteome URLs are defined in `conf/reference_proteomes.config` as the `params.databases` map, following the nf-core iGenomes pattern (`params.genomes` / `conf/igenomes.config`).
+It is loaded unless `--proteomes_ignore` is set; with `--proteomes_ignore`, `params.databases` is empty and `--database` is required.
+Each entry is `--proteomes_base` followed by the proteome ID and `)`, which gives the UniProt REST API endpoint:
 
 ```
 https://rest.uniprot.org/uniprotkb/stream?format=fasta&includeIsoform=true&query=(reviewed:true)+AND+(proteome:<PROTEOME_ID>)