Updated nextflow modules and better configuration

[cgoina]

@krokicki Improved configuration for spark tasks so that they use the proper container. There are currently 3 pipeline steps that use spark - stitching, spot extraction and multiscale and they all need to use the appropriate container otherwise there may be strange failures. This PR not only updates to the latest spark workflow but it also improved the configuration for stitching and spot extraction

[github-actions]

nf-core pipelines lint overall result: Failed ❌

Posted for pipeline commit 6f562fc

+| ✅ 213 tests passed       |+
#| ❔  11 tests were ignored |#
!| ❗  23 tests had warnings |!
-| ❌  56 tests failed       |-

Details

❌ Test failures:

• files_exist - File not found: assets/nf-core-easifish_logo_light.png
• files_exist - File not found: docs/images/nf-core-easifish_logo_light.png
• files_exist - File not found: docs/images/nf-core-easifish_logo_dark.png
• files_exist - File must be removed: lib/NfcoreTemplate.groovy
• files_exist - File must be removed: lib/Utils.groovy
• files_exist - File must be removed: lib/WorkflowMain.groovy
• files_exist - File must be removed: lib/nfcore_external_java_deps.jar
• nextflow_config - Plugin 'nf-validation' does not have a pinned version
• nextflow_config - Config variable (incorrectly) found: params.max_cpus
• nextflow_config - Config variable (incorrectly) found: params.max_memory
• nextflow_config - Config variable (incorrectly) found: params.max_time
• nextflow_config - Config default value incorrect: params.bigstream_config is set as ${projectDir}/conf/bigstream_config.yml in nextflow_schema.json but is /home/runner/work/easifish/easifish/conf/bigstream_config.yml in nextflow.config.
• nextflow_config - Config default value incorrect: params.dask_config is set as ${projectDir}/conf/dask_config.yml in nextflow_schema.json but is /home/runner/work/easifish/easifish/conf/dask_config.yml in nextflow.config.
• nextflow_config - Config default value incorrect: params.local_align_worker_cpus is set as 3 in nextflow_schema.json but is 1 in nextflow.config.
• nextflow_config - Config default value incorrect: params.bigstream_inv_use_root is set as true in nextflow_schema.json but is false in nextflow.config.
• nextflow_config - Config default value incorrect: params.multiscale_spark_workers is set as 1 in nextflow_schema.json but is 0 in nextflow.config.
• nextflow_config - Config default value incorrect: params.multiscale_spark_worker_cores is set as 3 in nextflow_schema.json but is 0 in nextflow.config.
• nextflow_config - Config default value incorrect: params.multiscale_spark_driver_mem_gb is set as 1 in nextflow_schema.json but is 0 in nextflow.config.
• nextflow_config - Config default value incorrect: params.cellpose_log_config is set as ${projectDir}/conf/cellpose_logging_config.ini in nextflow_schema.json but is /home/runner/work/easifish/easifish/conf/cellpose_logging_config.ini in nextflow.config.
• nextflow_config - Config default value incorrect: params.cellpose_dask_config is set as ${projectDir}/conf/dask_config.yml in nextflow_schema.json but is /home/runner/work/easifish/easifish/conf/dask_config.yml in nextflow.config.
• nextflow_config - Config default value incorrect: params.cellpose_dask_workers is set as 10 in nextflow_schema.json but is 8 in nextflow.config.
• nextflow_config - Config default value incorrect: params.user_id is set as 0 in nextflow_schema.json but is 1001 in nextflow.config.
• nextflow_config - Config default value incorrect: params.group_id is set as 0 in nextflow_schema.json but is 118 in nextflow.config.
• nextflow_config - Config default value incorrect: params.singularity_cache_dir is set as $HOME/.singularity_cache in nextflow_schema.json but is /home/runner/.singularity_cache in nextflow.config.
• files_unchanged - .github/CONTRIBUTING.md does not match the template
• files_unchanged - .github/PULL_REQUEST_TEMPLATE.md does not match the template
• files_unchanged - .github/workflows/branch.yml does not match the template
• files_unchanged - .github/workflows/linting.yml does not match the template
• actions_ci - Minimum pipeline NF version '23.04.0' is not tested in '.github/workflows/ci.yml'
• schema_params - Param spot_extraction_ids from nextflow config not found in nextflow_schema.json
• schema_params - Param skip_spot_extraction from nextflow config not found in nextflow_schema.json
• schema_params - Param distributed_spot_extraction from nextflow config not found in nextflow_schema.json
• schema_params - Param spot_extraction_subdir from nextflow config not found in nextflow_schema.json
• schema_params - Param spot_channels from nextflow config not found in nextflow_schema.json
• schema_params - Param spot_scales from nextflow config not found in nextflow_schema.json
• schema_params - Param spot_subpaths from nextflow config not found in nextflow_schema.json
• schema_params - Param rsfish_min_intensity from nextflow config not found in nextflow_schema.json
• schema_params - Param rsfish_max_intensity from nextflow config not found in nextflow_schema.json
• schema_params - Param rsfish_anisotropy from nextflow config not found in nextflow_schema.json
• schema_params - Param rsfish_sigma from nextflow config not found in nextflow_schema.json
• schema_params - Param rsfish_threshold from nextflow config not found in nextflow_schema.json
• schema_params - Param rsfish_background from nextflow config not found in nextflow_schema.json
• schema_params - Param rsfish_intensity from nextflow config not found in nextflow_schema.json
• schema_params - Param rsfish_spark_workers from nextflow config not found in nextflow_schema.json
• schema_params - Param rsfish_min_spark_workers from nextflow config not found in nextflow_schema.json
• schema_params - Param rsfish_spark_worker_cores from nextflow config not found in nextflow_schema.json
• schema_params - Param rsfish_spark_gb_per_core from nextflow config not found in nextflow_schema.json
• schema_params - Param rsfish_spark_driver_cores from nextflow config not found in nextflow_schema.json
• schema_params - Param rsfish_spark_driver_mem_gb from nextflow config not found in nextflow_schema.json
• schema_params - Param skip_warp_spots from nextflow config not found in nextflow_schema.json
• schema_params - Param warped_spots_subdir from nextflow config not found in nextflow_schema.json
• schema_params - Param warp_spots_cpus from nextflow config not found in nextflow_schema.json
• schema_params - Param warp_spots_mem_in_gb from nextflow config not found in nextflow_schema.json
• schema_params - Default value for param bigstream_inv_use_root invalid: Schema default (True) does not match the config default (false)
• modules_json - Your modules.json file is outdated. It will be automatically generated by running any module command.
• multiqc_config - assets/multiqc_config.yml not found.

❗ Test warnings:

• files_exist - File not found: assets/multiqc_config.yml
• files_exist - File not found: conf/igenomes.config
• files_exist - File not found: conf/igenomes_ignored.config
• files_exist - File not found: .github/workflows/awstest.yml
• files_exist - File not found: .github/workflows/awsfulltest.yml
• files_exist - File not found: ro-crate-metadata.json
• nextflow_config - nf-validation has been detected in the pipeline. Please migrate to nf-schema: https://nextflow-io.github.io/nf-schema/latest/migration_guide/
• pipeline_todos - TODO string in README.md: Add citation for pipeline after first release. Uncomment lines below and update Zenodo doi and badge at the top of this file.
• schema_description - No description provided in schema for parameter: stitching_result_container
• schema_description - No description provided in schema for parameter: sample_anisotropy
• schema_description - No description provided in schema for parameter: sample_expansion_factor
• schema_description - No description provided in schema for parameter: sample_voxel_spacing
• schema_description - No description provided in schema for parameter: sample_z_axis
• schema_description - No description provided in schema for parameter: cellpose_verbose
• schema_description - No description provided in schema for parameter: cellpose_preprocessing_steps
• schema_description - No description provided in schema for parameter: cellpose_preprocessing_params_file
• schema_description - No description provided in schema for parameter: cellpose_worker_runtime_opts
• schema_description - No description provided in schema for parameter: cellpose_worker_lsf_opts
• local_component_structure - samplesheet_check.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
• local_component_structure - cellpose_segmentation.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
• local_component_structure - multiscale.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
• local_component_structure - stitching.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
• local_component_structure - input_check.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure

❔ Tests ignored:

• files_exist - File is ignored: CODE_OF_CONDUCT.md
• files_exist - File is ignored: .github/ISSUE_TEMPLATE/config.yml
• nextflow_config - Config variable ignored: manifest.name
• nextflow_config - Config variable ignored: manifest.homePage
• files_unchanged - File ignored due to lint config: CODE_OF_CONDUCT.md
• files_unchanged - File ignored due to lint config: LICENSE or LICENSE.md or LICENCE or LICENCE.md
• files_unchanged - File ignored due to lint config: .github/ISSUE_TEMPLATE/bug_report.yml
• files_unchanged - File does not exist: assets/nf-core-easifish_logo_light.png
• files_unchanged - File does not exist: docs/images/nf-core-easifish_logo_light.png
• files_unchanged - File does not exist: docs/images/nf-core-easifish_logo_dark.png
• actions_awstest - 'awstest.yml' workflow not found: /home/runner/work/easifish/easifish/.github/workflows/awstest.yml

✅ Tests passed:

• files_exist - File found: .gitattributes
• files_exist - File found: .gitignore
• files_exist - File found: .nf-core.yml
• files_exist - File found: .editorconfig
• files_exist - File found: .prettierignore
• files_exist - File found: .prettierrc.yml
• files_exist - File found: CHANGELOG.md
• files_exist - File found: CITATIONS.md
• files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
• files_exist - File found: nextflow_schema.json
• files_exist - File found: nextflow.config
• files_exist - File found: README.md
• files_exist - File found: .github/.dockstore.yml
• files_exist - File found: .github/CONTRIBUTING.md
• files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
• files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
• files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
• files_exist - File found: .github/workflows/branch.yml
• files_exist - File found: .github/workflows/ci.yml
• files_exist - File found: .github/workflows/linting_comment.yml
• files_exist - File found: .github/workflows/linting.yml
• files_exist - File found: assets/email_template.html
• files_exist - File found: assets/email_template.txt
• files_exist - File found: assets/sendmail_template.txt
• files_exist - File found: conf/modules.config
• files_exist - File found: conf/test.config
• files_exist - File found: conf/test_full.config
• files_exist - File found: docs/output.md
• files_exist - File found: docs/README.md
• files_exist - File found: docs/README.md
• files_exist - File found: docs/usage.md
• files_exist - File found: main.nf
• files_exist - File found: conf/base.config
• files_exist - File found: modules.json
• files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
• files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
• files_exist - File not found check: .github/workflows/push_dockerhub.yml
• files_exist - File not found check: .markdownlint.yml
• files_exist - File not found check: .nf-core.yaml
• files_exist - File not found check: .yamllint.yml
• files_exist - File not found check: bin/markdown_to_html.r
• files_exist - File not found check: conf/aws.config
• files_exist - File not found check: docs/images/nf-core-easifish_logo.png
• files_exist - File not found check: lib/Checks.groovy
• files_exist - File not found check: lib/Completion.groovy
• files_exist - File not found check: lib/Workflow.groovy
• files_exist - File not found check: lib/WorkflowEasifish.groovy
• files_exist - File not found check: parameters.settings.json
• files_exist - File not found check: pipeline_template.yml
• files_exist - File not found check: Singularity
• files_exist - File not found check: .travis.yml
• nextflow_config - Found nf-validation plugin
• nextflow_config - Config variable found: manifest.nextflowVersion
• nextflow_config - Config variable found: manifest.description
• nextflow_config - Config variable found: manifest.version
• nextflow_config - Config variable found: timeline.enabled
• nextflow_config - Config variable found: trace.enabled
• nextflow_config - Config variable found: report.enabled
• nextflow_config - Config variable found: dag.enabled
• nextflow_config - Config variable found: process.cpus
• nextflow_config - Config variable found: process.memory
• nextflow_config - Config variable found: process.time
• nextflow_config - Config variable found: params.outdir
• nextflow_config - Config variable found: params.input
• nextflow_config - Config variable found: manifest.mainScript
• nextflow_config - Config variable found: timeline.file
• nextflow_config - Config variable found: trace.file
• nextflow_config - Config variable found: report.file
• nextflow_config - Config variable found: dag.file
• nextflow_config - Config variable (correctly) not found: params.nf_required_version
• nextflow_config - Config variable (correctly) not found: params.container
• nextflow_config - Config variable (correctly) not found: params.singleEnd
• nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
• nextflow_config - Config variable (correctly) not found: params.name
• nextflow_config - Config variable (correctly) not found: params.enable_conda
• nextflow_config - Config timeline.enabled had correct value: true
• nextflow_config - Config report.enabled had correct value: true
• nextflow_config - Config trace.enabled had correct value: true
• nextflow_config - Config dag.enabled had correct value: true
• nextflow_config - Config dag.file ended with .html
• nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
• nextflow_config - Config manifest.version ends in dev: 1.0dev
• nextflow_config - Config params.custom_config_version is set to master
• nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
• nextflow_config - Lines for loading custom profiles found
• nextflow_config - nextflow.config contains configuration profile test
• nextflow_config - Config default value correct: params.resolution= 0.23,0.23,0.42
• nextflow_config - Config default value correct: params.stitching_result_container= stitched.n5
• nextflow_config - Config default value correct: params.stitching_channel= all
• nextflow_config - Config default value correct: params.axis_mapping= -x,y,z
• nextflow_config - Config default value correct: params.stitching_block_size= 128,128,64
• nextflow_config - Config default value correct: params.final_block_size_xy= 512
• nextflow_config - Config default value correct: params.stitching_mode= incremental
• nextflow_config - Config default value correct: params.stitching_padding= 0,0,0
• nextflow_config - Config default value correct: params.stitching_blur_sigma= 2
• nextflow_config - Config default value correct: params.spark_cluster= true
• nextflow_config - Config default value correct: params.spark_workers= 10
• nextflow_config - Config default value correct: params.min_spark_workers= 5
• nextflow_config - Config default value correct: params.spark_worker_cores= 1
• nextflow_config - Config default value correct: params.spark_gb_per_core= 15
• nextflow_config - Config default value correct: params.spark_driver_cores= 1
• nextflow_config - Config default value correct: params.spark_driver_mem_gb= 12
• nextflow_config - Config default value correct: params.reuse_global_affine= true
• nextflow_config - Config default value correct: params.global_steps= affine
• nextflow_config - Config default value correct: params.global_scale= s3
• nextflow_config - Config default value correct: params.local_inverse_cpus= 1
• nextflow_config - Config default value correct: params.local_inverse_mem_gb= 0
• nextflow_config - Config default value correct: params.local_deform_cpus= 1
• nextflow_config - Config default value correct: params.local_deform_mem_gb= 0
• nextflow_config - Config default value correct: params.local_steps= affine,deform
• nextflow_config - Config default value correct: params.local_scale= s2
• nextflow_config - Config default value correct: params.local_transform_name= deform.n5
• nextflow_config - Config default value correct: params.local_inv_transform_name= inv-deform.n5
• nextflow_config - Config default value correct: params.with_dask_cluster= true
• nextflow_config - Config default value correct: params.warped_scales= s0
• nextflow_config - Config default value correct: params.global_align_cpus= 3
• nextflow_config - Config default value correct: params.global_align_mem_gb= 0
• nextflow_config - Config default value correct: params.local_align_cpus= 1
• nextflow_config - Config default value correct: params.local_align_mem_gb= 0
• nextflow_config - Config default value correct: params.local_align_workers= 10
• nextflow_config - Config default value correct: params.local_align_min_workers= 1
• nextflow_config - Config default value correct: params.local_align_worker_mem_gb= 0
• nextflow_config - Config default value correct: params.bigstream_local_overlap_factor= 0.125
• nextflow_config - Config default value correct: params.bigstream_inv_step= 1.0
• nextflow_config - Config default value correct: params.bigstream_inv_iterations= 20
• nextflow_config - Config default value correct: params.bigstream_inv_shrink_spacings=
• nextflow_config - Config default value correct: params.bigstream_inv_smooth_sigmas= 0.
• nextflow_config - Config default value correct: params.bigstream_inv_step_cut_factor= 0.5
• nextflow_config - Config default value correct: params.bigstream_inv_pad= 0.1
• nextflow_config - Config default value correct: params.bigstream_dashboard_port= 0
• nextflow_config - Config default value correct: params.bigstream_scheduler_port= 0
• nextflow_config - Config default value correct: params.global_registration_container= global-registration.n5
• nextflow_config - Config default value correct: params.local_registration_container= registration.n5
• nextflow_config - Config default value correct: params.registration_subdir= registration
• nextflow_config - Config default value correct: params.custom_config_version= master
• nextflow_config - Config default value correct: params.custom_config_base= https://raw.githubusercontent.com/nf-core/configs/master
• nextflow_config - Config default value correct: params.lsf_queue_size= 500
• nextflow_config - Config default value correct: params.default_mem_gb_per_cpu= 15
• nextflow_config - Config default value correct: params.max_cpus= 500
• nextflow_config - Config default value correct: params.max_memory= 1024.GB
• nextflow_config - Config default value correct: params.max_time= 240.h
• nextflow_config - Config default value correct: params.multiscale_with_spark_cluster= true
• nextflow_config - Config default value correct: params.multiscale_min_spark_workers= 1
• nextflow_config - Config default value correct: params.multiscale_spark_gb_per_core= 15
• nextflow_config - Config default value correct: params.multiscale_spark_driver_cores= 1
• nextflow_config - Config default value correct: params.multiscale_resolution= 0.23,0.23,0.42
• nextflow_config - Config default value correct: params.segmentation_ids= t1
• nextflow_config - Config default value correct: params.seg_channels= c1
• nextflow_config - Config default value correct: params.seg_scales= s2
• nextflow_config - Config default value correct: params.segmentation_subdir= segmentation
• nextflow_config - Config default value correct: params.segmentation_imgname= segmentation.n5
• nextflow_config - Config default value correct: params.sample_anisotropy= 1.0
• nextflow_config - Config default value correct: params.sample_expansion_factor= 1.0
• nextflow_config - Config default value correct: params.sample_z_axis= 0
• nextflow_config - Config default value correct: params.distributed_cellpose= true
• nextflow_config - Config default value correct: params.cellpose_model= cyto3
• nextflow_config - Config default value correct: params.distributed_cellpose_blocksize= 128,128,128
• nextflow_config - Config default value correct: params.cellpose_use_gpu= true
• nextflow_config - Config default value correct: params.cellpose_device= 0
• nextflow_config - Config default value correct: params.cellpose_diameter= 30
• nextflow_config - Config default value correct: params.cellpose_eval_channels= 0,0
• nextflow_config - Config default value correct: params.cellpose_min_size= 5
• nextflow_config - Config default value correct: params.cellprob_threshold= 1.0
• nextflow_config - Config default value correct: params.cellpose_iou_threshold= 0.05
• nextflow_config - Config default value correct: params.cellpose_labels_distance_th= 1.0
• nextflow_config - Config default value correct: params.cellpose_save_intermediate_labels= true
• nextflow_config - Config default value correct: params.cellpose_merge_iou_depth= 1
• nextflow_config - Config default value correct: params.cellpose_segmentation_cpus= 1
• nextflow_config - Config default value correct: params.cellpose_segmentation_mem_gb= 0
• nextflow_config - Config default value correct: params.cellpose_dask_min_workers= 1
• nextflow_config - Config default value correct: params.cellpose_dask_worker_cpus= 1
• nextflow_config - Config default value correct: params.cellpose_dask_worker_mem_gb= 0
• nextflow_config - Config default value correct: params.cellpose_scheduler_port= 0
• nextflow_config - Config default value correct: params.cellpose_dashboard_port= 0
• nextflow_config - Config default value correct: params.cellpose_worker_lsf_opts= -q gpu_l4 -gpu "num=1
• nextflow_config - Config default value correct: params.publish_dir_mode= copy
• nextflow_config - Config default value correct: params.max_email_size= 25.MB
• nextflow_config - Config default value correct: params.validate_params= true
• nextflow_config - Config default value correct: params.workdir= work
• files_unchanged - .gitattributes matches the template
• files_unchanged - .prettierrc.yml matches the template
• files_unchanged - .github/.dockstore.yml matches the template
• files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
• files_unchanged - .github/workflows/linting_comment.yml matches the template
• files_unchanged - assets/email_template.html matches the template
• files_unchanged - assets/email_template.txt matches the template
• files_unchanged - assets/sendmail_template.txt matches the template
• files_unchanged - docs/README.md matches the template
• files_unchanged - .gitignore matches the template
• files_unchanged - .prettierignore matches the template
• actions_ci - '.github/workflows/ci.yml' is triggered on expected events
• readme - README Zenodo placeholder was replaced with DOI.
• plugin_includes - No wrong validation plugin imports have been found
• pipeline_name_conventions - Name adheres to nf-core convention
• template_strings - Did not find any Jinja template strings (0 files)
• schema_lint - Schema lint passed
• schema_lint - Schema title + description lint passed
• schema_lint - Input mimetype lint passed: 'text/csv'
• system_exit - No System.exit calls found
• actions_schema_validation - Workflow validation passed: ci.yml
• actions_schema_validation - Workflow validation passed: release-announcments.yml
• actions_schema_validation - Workflow validation passed: branch.yml
• actions_schema_validation - Workflow validation passed: linting_comment.yml
• actions_schema_validation - Workflow validation passed: linting.yml
• actions_schema_validation - Workflow validation passed: clean-up.yml
• actions_schema_validation - Workflow validation passed: fix-linting.yml
• merge_markers - No merge markers found in pipeline files
• modules_json - Only installed modules found in modules.json
• modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'
• base_config - conf/base.config found and not ignored.
• modules_config - conf/modules.config found and not ignored.
• nfcore_yml - Repository type in .nf-core.yml is valid: pipeline
• nfcore_yml - nf-core version in .nf-core.yml is set to the latest version: 3.2.0

Run details

• nf-core/tools version 3.2.0
• Run at 2025-04-16 13:46:28