A draft of a simple module to discuss `nf-core` by Claptar · Pull Request #1 · cellgeni/nf-modules

Claptar · 2025-05-02T18:53:11Z

This PR is my attempt to stick to `nf-core` best practices as much as possible

The following changes were introduced:

Created a new tool anndatautils for any modules related to AnnData manipulations
Added anndatautils/toh5ad module
Created tests/data folder for test data
Created tests/configs folder for all test related configs
Created nf-test.config file

⚠️ TODO: add conversion of .zarr file

…S_TOH5AD module

…ut structure

…h validation

…dencies

- Added new binary files: pbmc_1k.h5, barcodes.tsv.gz, features.tsv.gz, and matrix.mtx.gz to the anndatautils directory. - Updated the Nextflow test for the anndatautils/toh5ad module to validate the conversion of .h5 files to .h5ad format. - Improved assertions in the test to check for successful process execution and correct output file generation.

…ons for clarity

…and remove container specification

… add snapshot for output validation

…s for improved reliability

… work directory settings

…execution

- Added pbmc_1k.h5 file containing the PBMC 1K dataset. - Included barcodes.tsv.gz, features.tsv.gz, and matrix.mtx.gz files in the raw_feature_bc_matrix directory for the dataset.

…dd new config file for tests

…ta paths to use params for improved flexibility

Copilot

Pull Request Overview

This PR introduces the new tool module "anndatautils/toh5ad" designed for converting AnnData-compatible files into AnnData objects saved as .h5ad files.

New meta file defines module metadata
Module configuration for input/output and authorship is added

Files not reviewed (7)

modules/sanger/anndatautils/toh5ad/Dockerfile: Language not supported
modules/sanger/anndatautils/toh5ad/main.nf: Language not supported
modules/sanger/anndatautils/toh5ad/module.config: Language not supported
modules/sanger/anndatautils/toh5ad/tests/main.nf.test: Language not supported
modules/sanger/anndatautils/toh5ad/tests/main.nf.test.snap: Language not supported
nf-test.config: Language not supported
tests/configs/nf-test.config: Language not supported

Claptar · 2025-05-02T18:54:32Z

I still need to add .zarr file conversion

BioinfoTongLI · 2025-05-02T19:09:01Z

Dockerfile goes to dockage?

Docker file for this module is already on dockage. I think it would still be good to reference Dockerfile somehow. Either by putting it in the module directory or by leaving a link to dockage in meta.yml or README.md file

BioinfoTongLI · 2025-05-02T19:11:40Z

+        type: string
+        description: |
+          Sample ID to be used as obs_names in the AnnData object
+  - - input:


should be a single dash. Otherwise, it means sample_id, input and delimiter are the same level.
Also, maybe it's worth renaming input to be some thing more concrete? What type of data do you have in mind?

Fix dashes, thank you! Well, input can be either file or directory, so I struggled to come up with better name

BioinfoTongLI · 2025-05-02T19:14:17Z

+params {
+    publish_dir_mode                  = "copy"
+    singularity_pull_docker_container = false
+    test_data_base                    = "${projectDir}/tests/data/"


we may need a collective decision on this.

Agreed. I added this just as a draft to enable testing

…tions

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 9 out of 14 changed files in this pull request and generated 2 comments.

Suppressed comments (11)

modules/sanger/anndatautils/toh5ad/tests/main.nf.test:54

Typo in comment: "basec run health" → "basic run health".

            // basec run health

modules/sanger/anndatautils/toh5ad/tests/main.nf.test:90

Typo in comment: "basec run health" → "basic run health".

            // basec run health

tests/config/nf-test.config:21

This singularity profile hard-codes site-specific bind mounts and a cache directory (/lustre, /nfs, /nfs/cellgeni/...). That makes the test config non-portable and likely to fail in CI or on other environments.

        singularity.enabled    = true
        singularity.autoMounts = true
        singularity.runOptions = '-B /lustre,/nfs'
        singularity.cacheDir   = '/nfs/cellgeni/singularity/images/'
    }

modules/sanger/anndatautils/toh5ad/main.nf:8

path(input, name: 'input/*') uses an option name that isn't used elsewhere in this repo for staging paths (e.g. other modules use stageAs:). This may not stage inputs into input/* as intended.

        tuple val(sample_id), path(input, name: 'input/*')

modules/sanger/anndatautils/toh5ad/main.nf:5

The module container is pinned to :latest, which is non-reproducible and inconsistent with the rest of this repo (no other module uses :latest). Please pin to a specific tag or digest.

    container 'docker://quay.io/cellgeni/metacells-python:latest'

modules/sanger/anndatautils/toh5ad/meta.yml:37

The metadata says sample_id is used as obs_names, but the implementation stores it in adata.obs["sample"] (and only affects obs.index when --delimiter is set). This is misleading for module users.

        description: |
          Sample ID to be used as obs_names in the AnnData object
    - input:

modules/sanger/anndatautils/toh5ad/meta.yml:44

The metadata describes the delimiter behavior as sample_id + delimiter + barcode, but the script builds barcode + delimiter + sample_id (and names the index barcode_sample). The meta.yml should match the implemented behavior.

        type: string
        description: |
          Delimiter to be used to create obs_names in the format sample_id + delimiter + barcode

modules/sanger/anndatautils/toh5ad/meta.yml:52

The output metadata says sample_id was used as obs_names, but the script uses it as an obs["sample"] value (and may incorporate it into obs_names only when delimiter is set).

        type: string
        description: |
          Sample ID that was used as obs_names in the AnnData object
    - "*.h5ad":

modules/sanger/anndatautils/toh5ad/Dockerfile:33

This Dockerfile installs several large notebook/interactive packages (e.g. jupyterlab, notebook, ipywidgets, papermill) that aren't needed by convert_to_h5ad.py and will significantly increase image size and build time.

    uv pip install --no-cache-dir \
    scanpy \
    numpy \
    pandas \
    muon \
    scikit-network \
    jupyterlab \
    notebook \
    tqdm \
    ipywidgets \
    papermill

modules/sanger/anndatautils/toh5ad/tests/main.nf.test:9

Trailing whitespace after tag "anndatautils".

    tag "anndatautils"

modules/sanger/anndatautils/toh5ad/tests/main.nf.test:23

Typo in comment: "basec run health" → "basic run health".

This issue also appears in the following locations of the same file:

line 54
line 90

            // basec run health

+config {
+    testsDir "."
+    workDir ".nf-test"
+    configFile "./tests/configs/nf-test.config"


+process {
+    withName: "*ANNDATAUTILS_TOH5AD" {
+        queue      = 'normal'
+        cpus       = 4
+        memory     = { 16.GB + 32.GB * (task.attempt - 1) }
+        publishDir = [
+            mode     : params.publish_mode,
+            path     : "${outputDir}/adata/raw",
+            overwrite: true
+        ]
+    }
+}


cellgeni added 16 commits April 30, 2025 16:08

add ANNDATAUTILS_TOH5AD module with configuration and metadata

8efeb53

refactor input type in meta.yml and add initial tests for ANNDATAUTIL…

de529da

…S_TOH5AD module

refactor meta.yml by removing unused tool entries and cleaning up inp…

822a9b8

…ut structure

add convert_to_h5ad script for loading and saving AnnData objects wit…

0891679

…h validation

add Dockerfile for setting up Python environment with necessary depen…

602d7e5

…dencies

Enhance ANNDATAUTILS_TOH5AD process to emit versions.yml and h5ad output

460e82e

Refactor meta.yml to standardize keywords and improve input descripti…

a10b328

…ons for clarity

Update ANNDATAUTILS_TOH5AD process to use wildcard for name matching …

f18c87f

…and remove container specification

Refactor tests for ANNDATAUTILS_TOH5AD process to improve clarity and…

fb2cfd5

… add snapshot for output validation

Fix quoting in script and stub sections of ANNDATAUTILS_TOH5AD proces…

24213a8

…s for improved reliability

Add initial configuration for nf-test module with tests directory and…

fa36589

… work directory settings

Fix process name matching for ANNDATAUTILS_TOH5AD to ensure accurate …

292f451

…execution

Add raw data files for PBMC 1K dataset

1ddd9fe

- Added pbmc_1k.h5 file containing the PBMC 1K dataset. - Included barcodes.tsv.gz, features.tsv.gz, and matrix.mtx.gz files in the raw_feature_bc_matrix directory for the dataset.

Update nf-test configuration: set testsDir to current directory and a…

0634482

…dd new config file for tests

Refactor ANNDATAUTILS_TOH5AD process name matching and update test da…

80c5f4c

…ta paths to use params for improved flexibility

Claptar requested review from BioinfoTongLI, YinshuiChang, cakirb, Copilot and iaaka May 2, 2025 18:53

Copilot AI reviewed May 2, 2025

View reviewed changes

Comment thread modules/sanger/anndatautils/toh5ad/meta.yml Outdated

BioinfoTongLI reviewed May 2, 2025

View reviewed changes

Claptar and others added 2 commits May 3, 2025 11:45

Fix indentation and structure in meta.yml for input and output defini…

4e34040

…tions

Update modules/sanger/anndatautils/toh5ad/meta.yml

9c207ec

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Claptar self-assigned this May 3, 2025

Claptar changed the title ~~Anndatautils toh5ad~~ A draft of a simple module to discuss nf-core May 3, 2025

Claptar added 5 commits May 3, 2025 12:23

Changed typo in directory name

e37e4c4

Added tool description to meta.yml file

5ed50e3

fix doi line

77ac7ce

Removed TODO comment from tests

992558b

Fix output emit syntax in ANNDATAUTILS_TOH5AD process

7353d82

BioinfoTongLI requested a lite review from Copilot August 4, 2026 12:19

Copilot started reviewing on behalf of BioinfoTongLI August 4, 2026 12:19 View session

Copilot AI reviewed Aug 4, 2026

View reviewed changes

Navigation Menu

Uh oh!

A draft of a simple module to discuss `nf-core` - #1

A draft of a simple module to discuss `nf-core`#1
Claptar wants to merge 23 commits into
mainfrom
anndatautils_toh5ad

Claptar commented May 2, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Claptar commented May 2, 2025

Uh oh!

BioinfoTongLI May 2, 2025

Uh oh!

Claptar May 3, 2025

Uh oh!

BioinfoTongLI May 2, 2025

Uh oh!

Claptar May 3, 2025

Uh oh!

BioinfoTongLI May 2, 2025

Uh oh!

Claptar May 3, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

Claptar commented May 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

This PR is my attempt to stick to nf-core best practices as much as possible

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Claptar commented May 2, 2025

Uh oh!

BioinfoTongLI May 2, 2025

Choose a reason for hiding this comment

Uh oh!

Claptar May 3, 2025

Choose a reason for hiding this comment

Uh oh!

BioinfoTongLI May 2, 2025

Choose a reason for hiding this comment

Uh oh!

Claptar May 3, 2025

Choose a reason for hiding this comment

Uh oh!

BioinfoTongLI May 2, 2025

Choose a reason for hiding this comment

Uh oh!

Claptar May 3, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants

Claptar commented May 2, 2025 •

edited

Loading

This PR is my attempt to stick to `nf-core` best practices as much as possible