Skip to content

Make submit_slurm_job() an S3 generic with shipped job templates - #32

Open
tariqmb wants to merge 4 commits into
mainfrom
refactor/submit-slurm-job-generic
Open

tariqmb wants to merge 4 commits into
mainfrom
refactor/submit-slurm-job-generic

Conversation

@tariqmb

@tariqmb tariqmb commented Aug 7, 2026

Copy link
Copy Markdown

Summary

Turns submit_slurm_job() into an S3 generic. What runs on the compute node is
decided by what you pass as input, and the command itself lives in the job
template, not in R. Each method's only job is to assemble the variables its
template needs and hand them to one shared renderer.

input what happens
bbi_nonmem_model renders the shipped nonmem-bbi template (bbi nonmem run local)
hyperion_nonmem_model delegated to hyperion::submit_model_to_slurm(), which carries its own template and finds its own pharos config
a .R / .qmd path renders the shipped rscript / quarto template by extension; a character vector submits one job per path
anything else supply your own template + template_opts
any other class submit_slurm_job.default() errors with what it does accept

New submit_rendered_job() is the shared worker: fill the whisker template,
write the script under submission_root, chmod, sbatch. It knows nothing
about what runs on the node, and paths flow through untouched — pharos-style,
the template owns any path handling.

Templates

Four templates now ship in inst/templates/ (based on the pharos slurm
template): nonmem-bbi, nonmem-pharos, quarto, rscript. Beyond your own
template_opts, a template can use {{job_name}}, {{partition}}, {{ncpu}},
{{parallel}}, {{num_mpi_cpus}}, {{account}}, {{log_path}}, and per method
{{model_path}}, {{config_path}} or {{script_path}}.

Breaking changes

  • Argument renames on the exported submit_slurm_job():
    .modinput, slurm_job_template_pathtemplate,
    bbi_config_pathconfig_path, slurm_template_optstemplate_opts.
  • Options dropped. slurmtools.slurm_job_template_path and
    slurmtools.bbi_config_path are no longer read — template now defaults to
    the shipped template for the input type, and config_path defaults to
    options('slurmtools.config_path'). The .onAttach message was updated to
    match.
  • A bare .mod/.ctl path is now refused with a pointer to
    bbr::read_model() / hyperion::read_model(). Previously a bare path was
    silently coerced into a list(absolute_model_path = ...) and run through the
    bbi path, which guesses the engine from a filename. Choosing the engine is the
    caller's decision, so it now requires a model object.
  • Deprecated internal submit_nonmem_model() and R/submit-model.R are
    removed, along with the lifecycle dependency. (It was @keywords internal
    and never exported, so no public surface goes away.)

Partition handling

Split the resolve/validate step out of the cpu-fit check, and put the cache
lookup in one place:

  • cached_partition_table() — the cache-or-lookup step, shared.
  • validate_partition() — resolves the partition = get_slurm_partitions()
    default (takes the first/smallest) and confirms it's a real partition.
  • check_slurm_partitions() — unchanged behaviour, now expects an
    already-resolved partition and returns it invisibly.

This replaces the old match.arg(partition), which only worked because the
default happened to be the full vector.

Vignette

Running-nonmem.Rmd executes live during check, and was still written against
the old API — R CMD check failed at vignette build. Ported it and its two
local template files:

  • slurm_job_template_pathtemplate, slurm_template_opts
    template_opts, options('slurmtools.bbi_config_path')
    options('slurmtools.config_path'); the "default template variables" section
    now documents what the methods actually inject.
  • The vignette templates used variables the methods no longer supply
    ({{bbi_exe_path}}, {{bbi_config_path}}, {{project_name}},
    {{project_path}}) — whisker renders those empty, i.e. a job that submits
    fine and dies on the node. They now invoke bbi from PATH and use
    {{config_path}} / {{#account}} / {{log_path}}.
  • Two live-execution safety fixes: monitoring no longer hardcodes
    user = "matthews", and the cleanup chunk now cancels only the two job ids
    the vignette submitted
    (parsed from submission$stdout) instead of every
    RUNNING job of a hardcoded user. The old cleanup also called
    cancel_slurm_job(job, confirm = FALSE) — the argument is auto_confirm, so
    it had been silently erroring inside tryCatch and never cancelled anything.

Tests

New tests/testthat/test-submit-slurm-job.R covers template selection per
extension, path pass-through, vectorised submission, script write + chmod,
relative submission_root/template, template_opts overriding
method-built values, the bare-control-stream refusal, unsupported extensions,
user-supplied templates for arbitrary file types, missing files, the .default
error, the bbi template render, and partition validation.

devtools::test(): 39 passing, 0 failures, 1 skip (the
hyperion-not-installed branch, which can't fire where hyperion is installed).

devtools::check(): 0 errors, 0 notes, 1 warning (qpdf not installed on
the check machine — no PDFs in the package, so environmental only). Note the
vignette build really submits (and then cancels) two tiny 1001 bbi runs on
the cluster.

Housekeeping

  • rproject.toml: project name was hyperionslurmtools; prism snapshot
    bumped to 2026-06-30; drop dependencies_only so the package itself installs.
  • hyperion added to Suggests.
  • air.toml added (excludes rv/scripts/ from formatting).
  • .gitignore: root-level accessory .qmd/.html design notes and their render
    artifacts, plus manual-test/ scratch.
  • .Rbuildignore: same accessory files — they're out of git but R CMD check
    builds from the working tree, and they caused 3 of the 4 check notes.
  • here moved Imports → Suggests: its last R/ user was the removed
    submit-model.R; the vignette still uses it.

Note for reviewers

The .onAttach message reports slurmtools.submission_root but not
slurmtools.config_path, even though the bbi method defaults to it. Happy to
add it if we want that surfaced at load.

@tariqmb
tariqmb force-pushed the refactor/submit-slurm-job-generic branch from 6c69f3c to 7ee0a6c Compare August 7, 2026 15:59
{{/ntfy}}


{{#parallel}}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why did this lose {{ bbi_exe_path }}?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe the refactored methods stopped filling in bbi_exe_path, and giving {{bbi_exe_path}} here would have been empty, causing a failure on node after submission. But it is templated again now (in re to your other comment), so i can re-insert it.

Comment thread inst/templates/nonmem-bbi.tmpl Outdated
#SBATCH --output={{log_path}}

{{#parallel}}
exec bbi nonmem run local {{model_path}}.mod --parallel --threads={{ncpu}} --config {{config_path}}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmm not sure how I feel about exec bbi seems like it needs to be templated...

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

assuming you want the same for all the other templates (exec pharos nonmem, exec rscript etc)

…ompute and login node have same container image
@tariqmb

tariqmb commented Aug 7, 2026

Copy link
Copy Markdown
Author

submit_to_slurm generic functionality:

  1. able to submit r scripts using submit_to_slurm()
Screenshot 2026-08-07 at 1 13 27 PM Screenshot 2026-08-07 at 1 24 58 PM
  1. able to submit a job to render a qmd
Screenshot 2026-08-07 at 1 29 55 PM
  1. parallel job submission maintained, correct template matching
Screenshot 2026-08-07 at 1 31 18 PM
  1. able to submit a bbi model with the shipped nonmem-bbi template
Screenshot 2026-08-07 at 1 50 31 PM Screenshot 2026-08-07 at 1 56 44 PM
  1. able to submit hyperion model
Screenshot 2026-08-07 at 2 03 31 PM
  1. able to supply own template to run other processes
Screenshot 2026-08-07 at 2 05 39 PM
  1. Won't accept bare paths
Screenshot 2026-08-07 at 2 07 36 PM
  1. similarly, numerics don't qualify as accepted structures
Screenshot 2026-08-07 at 2 15 28 PM
  1. nor are .txts (without a template)
Screenshot 2026-08-07 at 2 17 17 PM
  1. partitions must be valid
Screenshot 2026-08-07 at 2 19 21 PM

@dpastoor

dpastoor commented Aug 8, 2026

Copy link
Copy Markdown
Member

What's the justification for making this an s3 method at all. I do not generally like the boilerplate unless we have a good reason for it. I'm also not sure the way this is designed is the right mental model of if we were to do a generic interface where things would register. Eg we don't want a bunch of concrete implementations consolidated here, more likely we want to provide a set of patterns and functions that makes it easier to use slurm from the tool you care about without necessarily pulling in too much of the ecosystem.

@tariqmb

tariqmb commented Aug 10, 2026

Copy link
Copy Markdown
Author

It was inspired by a client meeting, not a direct request, just thought it would make the process simpler for the next client(s). I will work on de-generalizing the function, and make a template/pattern that can be adopted by multiple tools. I am doing this on a new branch, and will push after doing some proofing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants