refactor: make config paths relative to a base_path

[dgruano]

What & why

As discussed with @Sergio-ote , and related to #3, linting test currently fails on the workflow catalog:

Using workflow specific profile profiles/default for setting default command line arguments.
[validate_input / decoupled] Missing contract h5ad for 3 sample(s) in the input directory:
  - sample1: /scratch/CBiB/soterino/smk_visiumhd_results/Samples/sample1.h5ad
  - sample2: /scratch/CBiB/soterino/smk_visiumhd_results/Samples/sample2.h5ad
  - sample3: /scratch/CBiB/soterino/smk_visiumhd_results/Samples/sample3.h5ad
Every sample in the core sheet must already have a contract present as <contract_dir>/<sample>.h5ad (mode: decoupled does not build them). Fix the sheet or contract_dir, or set mode to a head technology (e.g. visiumhd)

This is a custom error raised correctly as a sanity check that each declared sample has an associated file under <contract_dir>/.h5ad. However, this is a check that should not be triggered during linting.

The easy way to fix this would be to place the sanity-check code under a onstart: block, which would relegate the check to run only during a real run (not with snakemake -n or snakemake --lint. Problem solved.

However, we realised the underlying problem is that current configuration points to files that do not exist in the repo (Sergio's scratch and development environment) instead of using proper defaults, such as .test dataset paths.

The goals of this PR are to:

1. Declare default datasets
2. Improve path configuration
3. Solve linting problem

To address the first item, I did:

• Define base_path in config.yaml
• Incldue base_path in config.schema.yaml
• Define all project paths relative to base_path
• Include instructions on how to correctly define such paths
• Add function to parse all config variables and expand base_path
• Include common.smk before config loading and processing
• Validate config schema after expanding base_path

I am creating this PR as a draft for improved collaboration.

Validation

• Workflow lint passes against .test/
• Decoupled .test/ dry-run builds the CoreBlock DAG
• Affected HeadBlock mode was dry-run with a complete mode-specific config and representative inputs, if applicable
• Config validates against workflow/schemas/config.schema.yaml
• Docs are modified according to the config modifications
• mkdocs build --strict passes, if documentation was changed
• Parameter names match between .smk rules and scripts; documentation was updated if behaviour changed
• Configuration TODO is addressed (i.e., Dani's custom paths are reset to defaults before merge)

[dgruano]

Current state of the branch should solve the missing sample files issue. Indeed, triggering the default runs in the .test directory is a good approach, as it configures the workflow minimal working example.

In my opinion, we could:

• Leave .test as the default working directory of profile/default

• On the root config/config.yaml, leave the necessary configuration paths with a clear placeholder.

• Modify the docs to include this information. User will have to:

  • Comment out or modify the directory variable under default/profile (they have to configure this profile in any case)
  • Fill up the appropriate paths and variables under the root config.yaml

  Thoughts @Sergio-ote ?

[Sergio-ote]

Hey @dgruano, sorry it took me some time to review this, great additions as always! Here a couple of comments:

Keeping .test as the default makes sense for catalog linting. Could we add a comment beside directory: .test in profiles/default/config.yaml explaining that it loads .test/config/config.yaml? For their own data, users should switch to directory: . when running from the repository root, then configure the root config and sample sheets. A short note in the quickstart would also help.

Also, one functional issue. The expansion helper prints to stdout, so --rulegraph produces expansion messages before the DOT graph. Could we remove those prints or send them to stderr?

Last, you left a #TODO in the config with your personal paths, remember to change it.

[dgruano]

Thanks for your comments @Sergio-ote ! I accepted your suggestions and am working on cleaning up the documentation about the paths.

Looking closer, I see three types of paths to configure:

• Pipeline output paths: these should be set relative to base_dir, as they are outputs of one specific pipeline run. We provide a default configuration so users only have to set base_dir.
• Pipeline input paths: these inputs are specific to the pipeline, and should be set by the user. We could leave them as (a) our defaults, (b) as placeholders (e.g. /path/to/input/file) or (c) as a default path under <base_dir>/inputs so users only have to copy paste them inside. Do you have any preference?
• Absolute input paths: external tools and references. These are all set as absolute paths, no question.

Also, while documenting the changes I noticed two things I'd like to discuss:

• If gene_exploration.queries is not set, it defaults to []. This is okay for Snakemake, but not for the underlying script, that fails abruptly. Some options here:
  • Default to config/gene_queries.tsv so users will see output with out default genes.
  • Execute gracefully with placeholder empty output.
  • Handle gracefully inside the script, by raising an error explaining that a file is needed. I don't like this idea, because an error due to a missing file should be thrown by snakemake.
  • Let Snakemake throw an error due to missing input file
  • Write custom logic to raise a custom error

Happy to chat about this

[Sergio-ote]

Hey @dgruano, thanks again for your comments, really useful.

The base_dir is a great idea, and for input paths I would leave the placeholders like /path/to/input/file.h5ad to specify the format (like the case of ingest_ref) or /path/to/dir/ when expecting a directory with files (like the external metadata).

For the exploration queries, I think it is a good idea to set it as config/gene_queries.tsv. Importantly, I'd add some genes present in the .test/data objects in case we decide to extend the CI/CD in the future.

[Sergio-ote]

Hey @dgruano, I have just merged PR #8, so this stack should be updated with respect to dev and tested following the CONTRIBUTING.md with the tools you developed.

Once merged into dev, I'll proceed to do some tests with real data before merging into main.

Thanks a lot for the help here, really pushing the work quality!

[dgruano]

I will skip for this once the "separation of concerns": I have edited the development environment to include graphviz as a dependency (it ships dot, needed to create the visual rulegraph).