usage of easifish on own data

[tatianawoller]

Hi,

My colleagues (@bpavie and @irelamb) are trying to run your pipeline easifish on our lightsheet dataset using our university HPC within a slurm job.
What we run

nextflow run easifish --input /dodrio/scratch/projects/2024_300/twoller/input_easifish/samplesheet.csv --datadir /dodrio/scratch/projects/2024_300/twoller/input_easifish -profile singularity --outdir test_easifish/ --skip spot_extraction,segmentation,registration,warp_spots,measure_intensities,assign_spots --reg_ch dummy --singularity_cache_dir /dodrio/scratch/projects/2024_300/twoller/.apptainer/cache --resolution 0.93,0.93,4.05

The associated error

ERROR ~ Error executing process > 'EASIFISH:STITCHING:BIGSTITCHER:RESAVE (t1)'

Caused by:
  Process `EASIFISH:STITCHING:BIGSTITCHER:RESAVE (t1)` terminated with an error exit status (1)


Command executed:

  CMD=(
      /opt/scripts/runapp.sh
      "singularity"
      "/dodrio/scratch/projects/2024_300/twoller/work/3605c4db-f7bb-405b-965f-eda05087dad7/stitching/spark/t1"
      "spark://10.141.41.10:7077"
      /app/app.jar
      net.preibisch.bigstitcher.spark.SparkResaveN5
      10
      1
      "15g"
      1
      "14g"
      --spark-conf "spark.driver.extraClassPath=/app/app.jar"
      --spark-conf "spark.executor.extraClassPath=/app/app.jar"
      --spark-conf "spark.jars.ivy=${SPARK_WORK_DIR}"
      --spark-conf "spark.driver.extraJavaOptions=-Dnative.libpath.verbose=true"
      -x /dodrio/scratch/projects/2024_300/twoller/test_easifish/stitching/t1/dataset.xml -o /dodrio/scratch/projects/2024_300/twoller/test_easifish/stitching/t1/dataset.zarr
  )
  echo "CMD: ${CMD[@]}"
  (exec "${CMD[@]}")

We succeeded to run the stitching part of your easifish pipeline on one of your dataset.
Is the stitching_padding expressed in percentage or in number ([0,1.0])?
Besides the resolution, which parameters would you recommend to specify?
Let us know whether it is easier to answer by issue, by mail or through a short call.

Thank you in advance and kind regards,

Benjamin, Irene and Tatiana

[cgoina]

@tatianawoller, @bpavie and @irelamb, stitching_padding parameter is used with the old 'SaalfeldStitcher' stitcher and it's the tile padding and the values are in voxels "x-padding,y-padding,z-padding". You used BigStitcher which is the one we recommend because the old one is not supported very well. Also the saalfeld stitcher only generates N5 and lately I haven't done a lot of testing for n5 data. It would help if you could find the actual error somewhere in the logs. Nextflow creates a lot of logs and especially with Spark finding the proper log is not easy.

[tatianawoller]

Hi @cgoina,

Thank you a lot for the swift and clear answer.
So, here is an overview at which stage the pipeline stopped

executor >  local (27)
[60/b91ac8] EAS…ET_CHECK (samplesheet.csv) | 1 of 1 ✔
[-        ] EASIFISH:INPUT_CHECK:DOWNLOAD  -
[32/c0abf2] EAS…LINK (organoid_subset.czi) | 1 of 1 ✔
[c6/e2f051] EAS…:PREPARE_SPARK_CONFIG (t1) | 1 of 1 ✔
[e9/570143] EAS…RT:SPARK_STARTMANAGER (t1) | 0 of 1
[6c/cdd0c4] EAS…:SPARK_WAITFORMANAGER (t1) | 1 of 1 ✔
[e4/924864] EAS…:SPARK_STARTWORKER (t1:10) | 0 of 10
[-        ] EAS…:SPARK_START:SPARK_CLEANUP -
[ba/16ce8b] EAS…SPARK_WAITFORWORKER (t1:2) | 10 of 10 ✔
[-        ] EAS…ER:STITCH_EXISTING_DATASET -
[68/9b324e] EAS…ITCHER:CREATE_DATASET (t1) | 1 of 1 ✔
[58/52ec85] EAS…NG:BIGSTITCHER:RESAVE (t1) | 0 of 1 ✘
[-        ] EAS…TITCHER:STITCH_NEW_DATASET -
[-        ] EAS…GSTITCHER:CREATE_CONTAINER -
[-        ] EAS…STITCHING:BIGSTITCHER:FUSE -
[-        ] EAS…SPARK_STOP:SPARK_TERMINATE -
[-        ] EAS…TION:BIGSTREAM_GLOBALALIGN -
[-        ] EAS…SK:DASK_START:DASK_PREPARE -
[-        ] EAS…SK_START:DASK_STARTMANAGER -
[-        ] EAS…_START:DASK_WAITFORMANAGER -

and here is the log file associated to the process EASIFISH:STITCHING:BIGSTITCHER:RESAVE (t1).
command.log

Thank you in advance,

[cgoina]

@tatianawoller the logs do not have enough information to tell you what happened. Do you have any other way to validate your CZI that is correct? Or can you share it with us somehow?

[tatianawoller]

@cgoina I have just shared the file through globus. Let me know if you can access it. Thank you in advance

[cgoina]

@tatianawoller I got the CZI and I was able to run the entire stitching. I am attaching here my input parameters. In input.json I disabled all steps except for stitching. You can also see exactly the parameters that I used for Bigstitcher. Make sure you use the latest version of the pipeline because the way the BigStitcher steps are handled have changed recently, mostly in order to support all possible knobs from BigSticher-Spark tools.

bigstitcher_config.yml
input.json
samplesheet.csv

[bpavie]

@cgoina , thanks for your fast feedback !

We are still a bit unsure how to run the workflow in order to obtain this result.

We think to submit a slurm job like this:

#!/bin/bash
#SBATCH -A 2024_300
#SBATCH --job-name=easifish
#SBATCH --partition=cpu_milan
#SBATCH --nodes="1"
#SBATCH --ntasks-per-node="1"
#SBATCH --cpus-per-task=16
#SBATCH --mem-per-cpu=4G
#SBATCH --output=easifish.out
#SBATCH --error=easifish.err
#SBATCH --time=04:00:00

module load Nextflow
nextflow run /dodrio/scratch/projects/2024_300/twoller/easifish --input /dodrio/scratch/projects/2024_300/twoller/input_easifish/samplesheet.csv --datadir /dodrio/scratch/projects/2024_300/twoller/input_easifish -profile singularity --outdir test_easifish/   --singularity_cache_dir /dodrio/scratch/projects/2024_300/twoller/.apptainer/cache --resolution 0.93,0.93,4.41 --bigstitcher_config  /dodrio/scratch/projects/2024_300/twoller/input_easifish/bigstitcher_config.yml --skip spot_extraction,segmentation,registration,warp_spots,measure_intensities,assign_spots --reg_ch dummy

Which will use the bigstitcher_config.yml and samplesheet.csv files hat you provided us.

However, we are not sure how to use the input.json that you added, we didn't see where/when it should be used, could you give us more information on that ?

Best,
Benjamin

[tatianawoller]

Hi @cgoina,

Thank you for the tips and explanation.
We managed to run the pipeline with the input.json file (where we have modified the paths) within a slurm jobs as follows:

nextflow run easifish  --resolution 0.93,0.93,4.41 --bigstitcher_config  /path/input_easifish/bigstitcher_config.yml -params-file /path/input_easifish/input.json

with the error

Command output:
  CMD: /opt/scripts/waitforworker.sh spark://10.141.42.93:7077 /path/script_easyfish/work/b08e256b-fb3b-43df-8928-99fe97759bb1/stitching/spark/r0/sparkworker-11.log /path/script_easyfish/work/b08e256b-fb3b-43df-8928-99fe97759bb1/stitching/spark/r0/terminate-spark 2 7200
  Timed out after 7202 seconds while waiting for Spark manager <- spark://10.141.42.93:7077

Command error:
  INFO:    Environment variable SINGULARITYENV_TMPDIR is set, but APPTAINERENV_TMPDIR is preferred
  INFO:    Environment variable SINGULARITYENV_NXF_TASK_WORKDIR is set, but APPTAINERENV_NXF_TASK_WORKDIR is preferred
  INFO:    Environment variable SINGULARITYENV_NXF_DEBUG is set, but APPTAINERENV_NXF_DEBUG is preferred
  CMD: /opt/scripts/waitforworker.sh spark://10.141.42.93:7077 /path/script_easyfish/work/b08e256b-fb3b-43df-8928-99fe97759bb1/stitching/spark/r0/sparkworker-11.log /path/script_easyfish/work/b08e256b-fb3b-43df-8928-99fe97759bb1/stitching/spark/r0/terminate-spark 2 7200
  Timed out after 7202 seconds while waiting for Spark manager <- spark://10.141.42.93:7077
  cat: /dath/script_easyfish/work/b08e256b-fb3b-43df-8928-99fe97759bb1/stitching/spark/r0/sparkworker-11.log: No such file or directory

However, it seems we have a configuration regarding the resources of the spark cluster and our own HPC.
We are using the following partition :

dodrio/cpu_milan_rhel9: phase 2 main partition:
384 workernodes, each with:
2x 64-core AMD Epyc 7763 CPU 2.45 GHz (“Milan” microarchitecture, 128 cores per node)
238 GiB usable RAM (~1.85GB/core), no swap
480 GB SSD local disk

How do you determine the configuration, resources of the spark cluster or do you have any other recommendation?

Thank you in advance and kind regards,

[cgoina]

@tatianawoller , @bpavie that input.json contains all command line parameters that I want to provide to the pipeline. My command line looks like this

nextflow run main.nf -params-file input.json -c conf/kubernetes.config -profile kubernetes 

The command above uses a kubernetes cluster specified by the -profile kubernetes. In my case the kubernetes profile is defined in the config file conf/kubernetes.config that is checked in. If I run this on the Janelia cluster I use this command

nextflow run main.nf -params-file input.json -profile janelia 

In this case the cluster configuration and the executor are defined in a configuration uploaded to https://github.com/nf-core/configs I checked and I haven't seen any configuration for VIB or probably I used a wrong name but if you don't have one uploaded you can always create a local configuration file that configures the slurm executor. If you look under the conf folder from the github repository I mentioned, you can find many configurations that use slurm (crick.config, jax.config,...). You can use that as a template to define the local configuration for your slurm cluster. I don't know why in your case the application timed out - I don't know exactly which container runtime Nextflow tries to use by default. You can try to use -profile singularity or -profile apptainer whichever is available and see if that works out of the box since you are already on a machine powerful enough to run this pipeline.

[bpavie]

Hi @cgoina

Thanks for the help, we succeed to obtain the stiched dataset in ome.zarr.

Here is the script we used to submit the SLURM job:

#!/bin/bash
#SBATCH -A 2024_300
#SBATCH --job-name=easifish
#SBATCH --partition=cpu_milan
#SBATCH --nodes="1"
#SBATCH --ntasks-per-node="1"
#SBATCH --cpus-per-task=4
#SBATCH --mem-per-cpu=4G
#SBATCH --output=easifish.out
#SBATCH --error=easifish.err
#SBATCH --time=04:00:00
 
module load Nextflow
nextflow run /scratch/projects/easifish  -profile vsc_ugent,apptainer   --apptainer_cache_dir /scratch/projects/.apptainer/cache  -params-file /scratch/projects/input_easifish/input.json -w /scratch/work --max_time '71.h' --tier1_project "ls_project"

We used the institutional nextflow config from VSC Ugent (vsc_ugent) as we are relying on the regional cluster.

We noticed that the pixel size was correctly set in the output of the ome.zarr for the BigDataViewer (so it seems that it's read directly from the CZI metadata). We will also try to visualize it with Webknossos.

Our next step will be to run then the cellpose segmentation (and skipping the registration).

One last things, we tried to run it locally via WSL on windows but we wee wondering if we would need a profile to run it on a local workstation

Thanks again for the help!

Benjamin

[cgoina]

Hi @bpavie - I am glad you were able to run the stitcher. To run locally on WSL if you have apptainer installed just use any of the apptainer, docker or podman profile. I am running this on my mac laptop (with the test dataset) using podman profile for example. You can also run segmentation only using conda and the conda config - which I also use locally to make sure the cellpose can use the MPS device for segmentation.

[tatianawoller]

Hi @cgoina,

Thank you a lot for all your help.
I played a bit the spark configuration parameters so that the jobs would be less in the queue.
How do you adapt the spark and dask configuration parameters as a function of the size of the dataset ?

Thank you in advance and kind regards,

[irelamb]

Hi @cgoina ,
Thank you for all your help regarding the stitching. We are testing it on a whole mouse brain data set. We were wondering if easifish contains a step to register the brain to an ATLAS or not. If I understand well , the registration step with Bigstream is used to register multiple images together, from multiple staining rounds for example. Can it be used to align channels from the same multi-channel image? Could you share an example Bigstream yml configuration file with us?

Thank you in advance and kind regards,

[cgoina]

@irelamb Bigstream can align any moving 3-D volume to any fixed 3-D volume - I never tested it with 2-D - and 2-D might generate errors because of how the data is read always as 3D blocks. I have not tested in the pipeline the case where the fix and moving volume come from the same zarr - just different channels but it might work - I don't know. Also regarding aligning to an ATLAS that probably requires putting the ATLAS in a directory structure that mimics round 0 or the reference round (doesn't have to be round 0 - it can have any name).

[irelamb]

@cgoina thank you very much for the fast reply. It is clear what Bigstream is made for. I am also not sure that it would be suitable for channel alignment or atlas registration.

We were testing the stitching with easifish on a larger data set (320 GB) with 2 channels, but we get an error at the RESAVE step:

Process EASIFISH:STITCHING:BIGSTITCHER:RESAVE (r0) terminated with an error exit status (1)

Looking more carefully, the error is:

/opt/spark/bin/spark-class org.apache.spark.deploy.SparkSubmit --properties-file /dodrio/scratch/projects/2024_300/ilamberti/scripts_slurm/work/485ca4f2-2377-41df-8c68-bf22dc37c242/stitching/spark/r0/spark-defaults.conf --conf spark.driver.host=10.141.41.20 --conf spark.driver.bindAddress=10.141.41.20 --master spark://10.141.41.20:7077 --class net.preibisch.bigstitcher.spark.SparkResaveN5 --conf spark.files.openCostInBytes=0 --conf spark.default.parallelism=10 --conf spark.driver.extraClassPath=/app/app.jar --conf spark.executor.extraClassPath=/app/app.jar --conf spark.jars.ivy=/opt/spark//work-dir --conf spark.driver.extraJavaOptions=-Dnative.libpath.verbose=true --executor-cores 4 --executor-memory 30g --driver-cores 1 --driver-memory 15g /app/app.jar -x /dodrio/scratch/projects/2024_300/ilamberti/easifish_results/stitching/r0/dataset.xml -o /dodrio/scratch/projects/2024_300/ilamberti/easifish_results/stitching/r0/dataset.zarr -s ZARR2
  java.lang.ArrayIndexOutOfBoundsException: Index 3 out of bounds for length 3

We are unsure about what causes the error. Do you have any suggestion about it?

We get the same error when trying to stitch the same data set that you processed (the organoid) but with 3 channels instead of 2. With respect to your input files, we modified the number of channels and the spark configuration parameters (and the paths).

Let me know if you need further information. Thank you very much in advance.

[cgoina]

@irelamb do you have a stack trace not just the exception? When you process the CZI for the organoid with 3 channels - does your CZI have all 3 channels? BigStitcher converts the CZI to view setups that it registers and fuses together and now the only place where channel is important for BigStitcher is in bigstitcher_config yaml. The points of interest detection detects the points only in one channel and it registers only one channel then copies the transformations to the other channels. Keep in mind channels are 0-based so for 3 channels you have (0,1,2) so if you use 3rd channel make sure you set channelId=2. If you upload your input.json and bigstitcher config I can review it