hpc.qmd

---
title: "High-Performance Computing (Argon Supercomputer)"
---

# Overview

Analysis scripts for High Performance Computing (HPC) at the University of Iowa.
Currently uses Argon and IDAS.

## Argon vs. IDAS

Both Argon and IDAS rely on HPC to run intensive, large-scale scripts and processing jobs.
See the [Argon vs. IDAS comparison page](https://uiowa.atlassian.net/wiki/spaces/hpcdocs/pages/540606569/Comparing+IDAS+and+Argon) (archived at <https://perma.cc/Q7G5-4JA3>) for more information about which option to use.

Argon runs on CentOS Linux, which requires users to have a basic understanding of command-line interface (CLI).
IDAS is similar to a remote desktop in that users can open and interact with Rstudio, Jupyter Notebook, and Visual Studio Code while benefiting from the higher processing capabilities of the HPC system.
This documentation focuses primarily on Argon.

## Links to Documentation

Workflow to request Argon/HPC access: <https://workflow.uiowa.edu/entry/new/3282>

Workflow to request IDAS access: <https://workflow.uiowa.edu/entry/new/7941>

Cluster Systems Documentation:
<https://uiowa.atlassian.net/wiki/spaces/hpcdocs/pages/76513411/Cluster+Systems+Documentation> (archived at <https://perma.cc/EKB8-ZKR7>)

Argon Cluster Documentation:
<https://uiowa.atlassian.net/wiki/spaces/hpcdocs/pages/76513466/Argon+Cluster> (archived at <https://perma.cc/6HST-VV6Y>)

IDAS Documentation: <https://uiowa.atlassian.net/wiki/spaces/hpcdocs/pages/76513404/Interactive+Data+Analytics+Service+Documentation> (archived at <https://perma.cc/2HT8-7FWU>)

Beginner Overview of HPC: <https://carpentries-incubator.github.io/hpc-intro/index.html> (archived at <https://perma.cc/Y8GR-QU3W>)

Beginner Overview of Linux Command Line: <https://ubuntu.com/desktop/docs/en/latest/tutorial/the-linux-command-line-for-beginners/> (archived at <https://perma.cc/73G9-B8QK>)

## Data Storage

Significant amounts of data storage are provided on Argon, but data are not backed up in any way unless special arrangements are made.
Standard (free) Argon accounts include 1 TB of storage, and standard IDAS accounts include 100 GB of storage.
It is the responsibility of the user to back up important information.

## Sensitive Data

User agrees to refrain from storing Restricted data on HPC resources.
Data is classified as Restricted when the unauthorized disclosure, alteration or destruction of that data could cause a significant level of risk to the University or its affiliates.
Examples of Restricted data include data protected by state or federal privacy regulations and data pertaining to identified human subjects that has not been deidentified.

## Fees

At present, there are no fees for the use of the Argon cluster or IDAS for low-priority usage.
For large users, or those who want access to dedicated resources, the option of purchasing or renting supplemental system hardware may be available.
If you are interested in dedicated hardware, contact [research-computing@uiowa.edu](mailto:research-computing@uiowa.edu).

# Accessing Argon via SSH

SSH, or **S**ecure **SH**ell, is a tool that allows users to access and operate remote servers (i.e., other computers).
You do not need to download or install anything to use SSH.
Users can implement SSH via the terminal or via an emulator software, such as SecureCRT or PuTTY.

Emulators allow users to enter commands into the terminal and often include GUIs that show the file structure of the remote server.
This can be helpful for users who have not used Linux, which operates via command line input and does not have an equivalent to Windows File Explorer or Mac Finder.

With SSH access, you can view and edit files as well as run scripts and submit jobs.

## Using the Terminal

1. Open Command Prompt (Windows) or Terminal (MacOS)
1. If you are on campus and/or using the VPN, enter `ssh hawkid@argon.hpc.uiowa.edu`
    - If you are off campus and not using the VPN, enter `ssh -p 40 hawkid@argon.hpc.uiowa.edu`
1. Enter your HawkID password when prompted
    - Note: The individual characters of your password will not appear as you type--this is not a bug!
1. Complete the two-factor authentication

## Using SecureCRT

SecureCRT is the University's choice of emulator, and it has several customizable settings for more advanced users.

1. Download SecureCRT from UIowa Informational Technology Services: <https://its.uiowa.edu/securecrt>
1. Enter `argon.hpc.uiowa.edu` as the Hostname and your HawkID as the Username
    - Do not modify any other fields/settings
1. Click 'Connect'
1. Enter your HawkID password when prompted
1. Complete the two-factor authentication

## Using PuTTY

PuTTY is another popular emulator that is more lightweight but less customizable than SecureCRT.
It is not available through ITS, but it is free to download from the creators.

1. Download PuTTY: <https://www.chiark.greenend.org.uk/~sgtatham/putty/>
1. Enter `argon.hpc.uiowa.edu` as the Host Name
1. Click 'Connect'
    - The first time you connect, click Accept on the pop-up alert
1. Enter your HawkID after the `login as` prompt
1. Enter your HawkID password when prompted
1. Complete the two-factor authentication

## After Log-In

After logging in, the terminal or emulator will take a moment to connect to Argon.
Once a connection is established, you will see `[hawkid@argon-itf-login-# ~]$` as the final line of text.

# Map Argon Drive to Local Computer

Mapping your Argon directory to your computer allows you to access your Argon files without CLI.
You won't be able to submit jobs, but it makes editing and moving files much more convenient for those who are new to Linux.

## Windows

1. Open File Explorer
1. In the sidebar on the left, right click 'This PC', then select 'Map Network Drive'
1. Enter `\\data.hpc.uiowa.edu\argon_home` as the Folder
1. Ensure that the 'Reconnect at sign-in' box is checked
1. Click 'Finish'

## Mac

1. Open Finder
1. In the menus along the top, select 'Go' then 'Connect to Server...'
1. Enter `smb://IOWA;<hawkid>:*@data.hpc.uiowa.edu/argon_home` as the Server Address
1. Click 'Connect'
1. Log in using your HawkID username and password if prompted

# Argon Navigation

## Quick Reference

Below is a list of common commands with a brief explanation of what each does.
More detailed information for each command is located in the corresponding section of the page.

Accessing Argon

- `ssh hawkid@argon.hpc.uiowa.edu` logs in to Argon

[Nodes](#sec-nodes)

- `qlogin` requests a computing node
- `exit` closes out of whichever node/environment/connection you are in

[File Systems](#sec-file-systems)

- `pwd` shows the path to the current folder (the working directory)
- `cd insert/file/path` brings you to the provided file path (sets the working directory)
    - `cd ~` returns to home directory (your Argon folder)
    - `cd ..` returns to the folder "above" the current folder (i.e., the folder that the current folder is in)
    - `cd /Shared/lss_itpetersen/Lab` goes to the lab drive
- `ls` lists the files and folders in your current folder
- `nano filename.extension` opens or creates a file
- `dos2unix filename.extension` converts CRLF (Windows) line endings to LF (Linux) line endings
- `rm filename.extension` deletes the specified file
- `cp filename.extension newFileOrFolder` copies the contents of the specified file into the provided location (folder or file)
- `mv filename.extension new/folder/path` moves the specified file to the provided location
    - `mv filename.extension newFileName.extension` renames the specified file to the new name

[Modules](#sec-modules)

- `module load moduleName` loads the specifed module
- `module list` lists the modules loaded in the current session/environment
- `module avail optionalSearchTerm` lists the modules available to use, narrowing down to modules matching the search term if provided

[Containers](#sec-containers)

- `apptainer build container.sif defFile.def` builds a container named 'container.sif' based on the definition file named 'defFile.def'
- `apptainer run container.sif` runs the default command in the specified container
    - `apptainer run --app appName container.sif` runs the specified app's default command in the specified container
- `apptainer exec container.sif commandName commandArguments` runs the specified command (with any provided arguments) in the specified container

[Jobs](#sec-jobs)

- `qsub jobFile.job` submits the specified job file to the queue
- `qstat -u hawkid` lists the active jobs associated with the specified HawkID, including job numbers
- `qdel jobNumber` cancels the job associated with the specified job number

Miscellaneous

- `Rscript path/to/script.R` runs the specified `R` script (if an `R` module is loaded)
    - `Rscript path/to/script.R outputFile.txt` runs the specified `R` script and saves the console output to the specified output file
    - See the [HPC documentation](https://uiowa.atlassian.net/wiki/spaces/hpcdocs/pages/76514707/R+Programs+in+Batch+mode+for+HPC) (archived at <https://perma.cc/99JQ-43ZG>) for more information

## Nodes {#sec-nodes}

After logging in, you are assigned to a *log-in node*.
There are limited log-in nodes, and they are shared by all Argon users.
You can create, edit, and delete files, and you can submit jobs for processing, but you should not do any actual computing from the log-in node.
To do more intensive work, you will need to request a *computing node* by entering `qlogin` into the terminal or emulator.

You can think of the login nodes as the host stand at a restaurant--you go there when you first arrive (log in) and ask for a table (computing node), and the host (queue manager) seats you when one is available.
However, it would be rude and inconvenient to order your food and eat it (run a bunch of scripts) at the host stand.

## File Systems {#sec-file-systems}

To interact with files in Argon, you must be able to navigate through the file directory.
On Windows and MacOS, File Explorer and Finder allow users to locate the files they need.
In Argon (and other Linux-based systems), users navigate through directories (folders) using the command line.
There are a few commands that are helpful for moving through folders:

- `pwd` shows the full file path to the current folder
    - When you first log in to Argon, you will be in your home directory, `/Users/hawkid`
    - The home directory can be abbreviated to `~`
- `cd insert/file/path` allows you to move to different folders
    - If you are moving to a folder that is somewhere inside of the current folder, you can use the relative file path
        - Example: To move from the main `Lab` folder in the lab drive to the Studies folder, enter `cd Studies`
        - Example: To move from the main `Lab` folder in the lab drive to your Members folder, enter `cd Members/hawkid`
    - If you are moving to a folder that is outside of the current folder, you must use the absolute (full) file path
        - Example: To move from your Argon home directory to the Studies folder on the lab drive, enter `cd /Shared/lss_itpetersen/Lab/Studies`
    - If the folder name has any spaces in it, put quotation marks around the file path
        - Example: `cd "Studies/School Readiness Study"`
    - `cd ~` will bring you back to your home directory (i.e., your Argon storage folder)
    - `cd ..` will bring you to the folder "above" the folder that you are currently in
        - Example: If you are in the `School Readiness Study` folder in the lab drive, `cd ..` will bring you to the `Studies` folder
- `ls` lists all of the files and folders in your current folder
    - All folders contain `.` (the current folder) and `..` (the parent folder)
    - Below are some common arguments used with `ls` (arguments can be combined):
        - `ls -A` lists all files and folders, including hidden ones but excluding `.` and `..`
        - `ls -d` lists folders only
        - `ls -l` lists folder contents with permissions, owner, size, and modification date
        - `ls -R` lists contents of current folder and all subfolders
        - `ls /file/path` lists the contents of the specified folder
- `mkdir folder/path` creates the specified folder in the specified location
    - `mkdir -p folder/path` creates the specified folder in the specified location, creates any parent folders along the path if needed, and does not throw an error if the folder already exists

In addition to navigating directories, there are a number of commands that are helpful for interacting with files:

- `nano filename.extension` opens the specified file if it exists or creates the specified file if it does not
- `dos2unix filename.extension` converts CRLF (Windows) line endings to LF (Linux) line endings
    - Windows and Linux use different characters to represent a line break, resulting in `\r: command not found` errors when running Windows files on Linux
    - The `dos2unix` command replaces the Windows line break character (`\r\n`) with the Linux line break character (`\n`)
    - `file filename.extension | grep CRLF` checks for the CRLF character and returns a list of instances if found or nothing if not found
    - `find ~/folder/path -name '*extension' -exec dos2unix "{}" \;` runs the `dos2unix` command on all files with the provided extension found in the provided directory
    - `sudo apt update && sudo apt install -y dos2unix` installs the `dos2unix` package (you may be prompted for your password)
- `rm filename.extension` removes the specified file
    - `rm -rf folder/path/*` removes all files/folders in the specified folder
        - Use with caution--the `f` argument means it will be deleting without confirming
- `cp fileOrFolder fileOrFolder` copies the contents of the first argument to the location of the second argument
    - `cp filename.extension newFile.extension` copies the contents of the first file to the second file
        - Use with caution--if the second file already exists, the content will be overwritten
    - `cp filename.extension folder/path/` copies the specified file into the specified folder
    - `cp -r folder/path new/folder/path` copies the first folder and all of its contents into the second folder
    - `cp filename1.extension filename2.extension new/folder/path` copies all listed files into the provided folder
- `mv fileOrFolder new/folder/path` moves the specified file or folder to the specified location
    - `mv filename.extension newFile.extension` renames the specified file to the name specified in the second argument
        - Use with caution--if the second file already exists, the content will be overwritten
    - `mv filename.extension new/folder/path` moves the specified file to the specified location
    - `mv -r folder/path new/folder/path` moves the first folder and all of its contents into the second folder
    - `mv filename1.extension filename2.extension new/folder/path` moves the specified files into the provided folder

### Accessing the Lab Drive

After accessing Argon using SSH, enter `cd /Shared/lss_itpetersen/Lab` in the terminal or emulator.

### Project Directories

Like the Lab drive, it is recommended that you store the materials for each project in its own folder:
```bash
cd /Users/itpetersen/Documents/Projects/Bayesian_IRT/
cd /Users/itpetersen/Documents/Projects/EXT_pilot/
cd /Users/itpetersen/Documents/Projects/EXT_ParentReport/
cd /Users/itpetersen/Documents/Projects/Multiple_Imputation/
cd /Users/itpetersen/Documents/Projects/SelfRegulation_IRT/
cd /Users/itpetersen/Documents/Projects/Test/
```


## Bash Scripting {#sec-bashScripting}

In order to use Argon, it is necessary to know some basic `bash` scripting.
`bash` is a shell, which is a type of scripting language used to interact with operating systems.
The commands described in the preceding sections are all `bash` commands, but there are a handful of additional commands that are helpful for setting up containers and job scripts.

### Keyboard Shortcuts

- `Ctrl` + `C` cancels/interrupts commands
- `Ctrl` + `Shift` + `C` or right clicking copies highlighted text
- `Ctrl` + `Shift` + `V` or right clicking pastes text

### Syntax & Symbols

- Folder paths and file names with spaces should be entered in quotation marks (`"`)
- When creating variables, do not include spaces around the equal sign (e.g., use `varName=value` rather than `varName = value`)
- When referencing variables, use `${varName}`
    - If inserting variables into a string, use 
- `#` at the start of a line indicates comments
- `;` separates commands written on the same line
    - Note: Unless you have very short commands, it is recommended to just put the commands on separate lines
- `\` continues a command onto the next line
- `|` passes the results of one command into the next command
- `${varName}` indicates a variable
    - `"${varName}"` (in double quotes) will insert the value of the variable
    - `'${varName}'` (in single quotes) will keep variable name
    - As an example, run the following commands in Argon:
    
      ```bash
      test_var="I love Argon!"
      echo "${test_var}"
      echo '${test_var}'
      ```
    
    - `${varName[n]}` selects the nth value from the `varName` list
- `$(function)` inserts the result of the specified function
    - If the function has multiple results (and you want all of them), use `($(function))`
- `./execFile` runs the specified executable file/program

### Advanced Package Tool Commands

The Advanced Package Tool, or `apt`, is used to install, update, and otherwise manage packages in Debian-based Linux systems, including Argon.
Although you likely will not use these commands in a job script, they are common in definition files.
Outside of definition files, these commands should be run with `sudo` (see @sec-usefulCommands).

- `apt update` retrieves a list of installed packages that need upgrades
- `apt install packageName` installs the specified package(s)
    - `apt install -y packageName` installs the package(s) non-interactively (i.e., assuming 'yes' responses to any prompts)
    - `apt install --no-install-recommends packageName` installs the package(s) with required dependencies only (default installation installs required and recommended packages)
- `apt list --installed` lists all installed packages
- `apt-get clean` clears temporary files from the package installation folder
    - `apt-get clean -y` clears files non-interactively
- `apt-get autoremove` removes unnecessary packages
    - `apt-get autoremove -y` removes unnecessary packages non-interactively

### Other Useful Commands {#sec-usefulCommands}

- `man insertCommandName` opens instructions for the provided command
- `sudo insertCommandName` runs the provided command with administrative privileges
    - If you are not logged in as an administrator, you may be prompted for an admin login
- `echo "text and/or variable(s)"` prints (or "echoes") the specified information, either directly into the terminal or (more commonly) through a pipe operator (`|`) into another command
    - `echo "${input_variable}" | program programScript.ext` passes the value of `input_variable` into the specified program to use when running the specified script
- `cat items to combine` prints the specified items sequentially
    - `cat items to combine > fileName.extension` creates a file containing the combined contents of the specified items
        - Use with caution--if the specified file already exists, the contents will be overwritten
    - `cat items to add >> fileName.extension` adds the specified items to the end of the specified file
    - `cat > fileName.extension <<EOF text to add EOF` adds the specified text to the end of the specified file
        - Since this command adds anything between the two `EOF` indicators, it is typically used for adding multiple lines
- `chmod mode fileOrFolder` updates permissions for the specified file or folder to the specified mode
    - The `mode` parameter is very flexible, but for simplicity, only the octal notation is described here
    - Octal permissions are specified by three numbers, representing owner, group, and general permissions, respectively
    - Octal Permission Numbers:
        - 0 - no access
        - 1 - execute only
        - 2 - write only
        - 3 - write and execute
        - 4 - read only
        - 5 - read and execute
        - 6 - read and write
        - 7 - read, write, and execute
    - Enter `man chmod` for more detailed `mode` information, or see [this](https://askubuntu.com/questions/518259/understanding-chmod-symbolic-notation-and-use-of-octal) AskUbuntu thread for an explanation of how different modes are used
- `wget https://link.to.file` downloads the contents of the specified link
    - `wget -O fileName.exteions https://link.to.file` saves the contents of the link to the provided file name (rather than the linked file's original name)
    - `wget --user="siteUsername" --ask-password https://link.to.file` downloads the contents of the specified link from a password-protected source (e.g., GitHub) and prompts for your password when needed

# Software

To run scripts in Argon, you will need to use either a container or a module.
Modules and containers have similar functions: they include software packages and dependencies and allow users to run scripts with that software.

As an example of why this might be helpful, consider a script that runs in `R`.
Two different users might have different operating systems (Windows, MacOS, or Linux), different versions of `R`, different packages installed, or different versions of the required packages installed.
One user might not have `R` installed at all.
With a set of modules or a container, the user does not have to figure out which software and software versions are necessary to run a certain script--all they need is the script and the module(s)/container.
These tools are also helpful for reproducibility.
Because the they is stable (i.e., the software and operating system do not update unless the creator modifies it), the results of a given script will be the same each time it is run with that module/container.

The key difference for practical use is that the Argon modules are created and maintained by ITS, while containers are created and maintained by users.
Because modules are "built in", they are easier to access.
However, they have a specific set of software available to use, and they cannot be modified.

## Modules {#sec-modules}

Argon opens with the default module stack (the 2021.1 stack) already loaded.
However, there is a more recent stack (the 2022.2 stack) that has more current versions of the software in it (see the [software list](https://uiowa.atlassian.net/wiki/spaces/hpcdocs/pages/76513440/Argon+Software+List), archived at <https://perma.cc/WJ4Q-GDUS>).
There are several commands for interacting with environment modules:

- `module avail` lists all of the installed modules in the environment, regardless of whether you have them loaded
    - `module avail searchString` lists all of the installed modules that include the search string in the name
    - `module spider searchString` lists all of the available modules (even incompatible ones or ones that can't be loaded) that include the search string in the name
- `module list` lists all of the modules loaded in the current environment
- `module load moduleName` loads the specified module (must include version number)
    - `module load stack/2022.2` loads the current software stack
    - `module load r/4.2.2_gcc-9.5.0` loads `R`
    - Note that `R` packages must be loaded in addition to base `R`
- `module unload moduleName` "unloads" the specified module
    - This command is helpful when multiple versions of a given software are loaded into the environment--any extra versions should be unloaded
- `module save environmentName` saves the current set of loaded modules under the provided name so you can load them later
- `module restore environmentName` loads the set of modules saved under the provided name

### R in Argon

If you need `R` packages that aren't available in the software stack, or if you need to use the current `R` version, using a container is recommended.
Alternatively, see below for instructions on installing different `R` versions without a container.

#### Compile Other Versions

If you want to compile a more recent version of `R` than is available in the software stacks, see [here](https://brandonlebeau.org/2022/06/02/compile-hpc-iowa/) (archived at <https://perma.cc/C6EX-EZL4>).
However, we now compile the most recent version of `R` (and packages) using a [container](#sec-containers).

#### Install Packages

<https://uiowa.atlassian.net/wiki/spaces/hpcdocs/pages/76514803/Adding+R+programs+to+a+personal+library> (archived at <https://perma.cc/3SRR-2JE7>)

1. In Argon, load the `R` environment module:

    ```bash
    module load r/4.2.2_gcc-9.5.0
    ```
    
1. Load compiler modules (not strictly necessary, since not all packages require them, but many do):

    ```bash
    module load cmake/3.25.0_intel-2021.7.1
    module load ncurses/6.3_gcc-9.5.0
    module load openssl/1.1.1s_gcc-9.5.0
    ```
    
1. Launch R:

    ```bash
    R
    ```
    
1. Install the package(s)

    - From CRAN:

        ```r
        install.packages(
          c("package1", "package2"),
          repos = "https://cloud.r-project.org",
          dependencies = TRUE,
          type = "source",
          Ncpus = 40
        )
        ```

    - From local source:
    
        ```r
        install.packages(
          "path/to/source/file",
          repos = NULL,
          dependencies = TRUE,
          type = "source",
          Ncpus = 40
        )
        ```
        
    - If you have not installed other packages before, you will see the following message:

        ```r
        Warning in install.packages("package_name", repos = "http://cran.r-project.org") :
        'lib = "/opt/R/3.0.2/lib64/R/library"' is not writable
    
        Would you like to use a personal library instead?  (yes/No/cancel)
        ```

        1. Enter `y` to use a personal library
        1. Enter `y` to create a folder for the personal library
        1. Installation will take several minutes, because all dependencies will be installed
1. After installing packages, you should periodically update them using `update.packages(ask = FALSE)`
1. For instructions on using `renv` to manage packages, see the documentation [here](R.qmd#sec-renv)

## Containers {#sec-containers}

Argon does not include any containers by default, so users must [create them](#sec-create-container).
After creating your container, there are several commands you can use to interact with it:

- `apptainer build container.sif sourceInfo` creates a container called 'container.sif' using the provided source
    - The `sourceInfo` argument can be a definition file (e.g., `defFile.def`), a link to a docker image (e.g., `docker://mathworks/matlab-deps:latest`), or one of the other options from the Apptainer documentation
    - `apptainer build --sandbox container/ sourceInfo` creates a "sandbox" container that you can edit (useful for troubleshooting builds)
        - This creates a directory rather than a `.sif` file
    - `apptainer build container.sif sandboxContainer` converts the specified sandboxContainer directory into a `.sif` file
- `apptainer exec container.sif commandName commandArguments` will run the specified command (with arguments, if included) using the specified container file
- `apptainer run container.sif` runs the default command(s) in the specified container
    - `apptainer run --app appName container.sif` runs the default command(s) for the specified app in the specified container
- `apptainer shell container.sif` opens a shell within the specified container, allowing you to run commands inside the container like you would within the terminal
    - Since container files are not editable, you will not be able to install or update any software
- `apptainer help commandName` displays information about the provided Apptainer sub-command (syntax, options, etc.)

# Building Containers {#sec-create-container}

Link to ITS documentation: <https://uiowa.atlassian.net/wiki/spaces/hpcdocs/pages/76513442/Singularity+Containers>

Link to Apptainer documentation: <https://apptainer.org/documentation/>

## Overview

Because the default Argon workspace only has the modules provided by ITS (the "stack"), many users prefer to use a container.
A container essentially operates like its own computer, with its own operating system and any files or programs the creator wants to include.
However, unlike an actual computer, a container is stored as a file that can be shared (and used) across several different computers.

The University recommends using Apptainer, formerly known as Singularity, to create and manage containers.

To create a container, you must have admin privileges on the computer you are using, which means you cannot create a container in Argon.
However, you can create a container on your own computer and use it in Argon.
Because none of the lab computers run on Linux, you will need to install a Linux subsystem, which allows users to run Linux from another operating system (such as Windows or MacOS).

## Creating a Container

### Installing Apptainer

1. Install a Linux subsystem with Ubuntu
    - Note: If a Linux subsystem is already installed, open the subsystem as an administrator and proceed to the next step

    <details>
    <summary>Windows</summary>

    1. Open PowerShell as an administrator
    1. Enter `bash wsl --install`
    1. When the installation is complete, restart your computer
    1. Your computer will install Ubuntu (a flavor/type of Linux) during the restart
    1. Open Ubuntu as an administrator

    </details>
    
    <details>
    <summary>MacOS (Not Recommended)</summary>

    1. Download the latest [Ubuntu server](https://ubuntu.com/download/server/arm)
    1. Download [UTM](https://docs.getutm.app/installation/macos/)
        1. Use the 'Download from GitHub' option--the App Store version is identical, but costs $10
    1. Open UTM and click '+' to create a new virtual machine (VM)
    1. Select 'Virtualize'
    1. Select 'Linux'
    1. Select the amount of RAM and CPU cores to allocate to the VM (or use defaults)
        - To see how much RAM/CPU cores your computer has:
            1. Click the Apple icon in the top left corner of the screen
            1. Select 'About this Mac'
            1. The RAM is listed next to 'Memory'
            1. Click 'More Info...'
            1. Scroll all the way to the bottom, then click 'System Report...'
            1. The number of CPU cores is listed next to 'Total Number of Cores'
        - In general, you should allocate no more than 75% of your total CPU cores and no more than 50% of your total RAM
    1. When prompted for the Boot Image, make sure 'Boot from ISO image' is selected
    1. Click 'Browse' and select the Ubuntu installation file
    1. Specify the maximum amount of storage space to allocate to the subsystem
        - To see how much storage space your computer has:
            1. Open System Settings
            1. Click 'Storage'
        - In general, you should overestimate the amount of storage, but you cannot exceed the available storage space on your computer
    1. Do not add a Shared Directory
    1. Click 'Save'
    1. Click the Play icon to enter the subsystem
    1. Select the 'Try or Install Ubuntu' option
    1. Complete the installation prompts, using the up and down arrow keys to move between options and the enter key to select the highlighted option

        <details>
        <summary>Installation Prompts</summary>

        - Language: English (or whichever you prefer)
        - Keyboard Layout: English (US)
        - Keyboard Variant: English (US)
        - Installation Type: Ubuntu Server
        - Network Configuration: name varies, but select the option with the 'eth' type
        - Proxy Configuration: (leave blank)
        - Mirror Configuration: (leave blank; wait for 'This mirror location passed tests.' to appear)
        - Guided Storage Configuration: (use default options)
        - Storage Configuration: (review summary and continue)
        - Confirm Destructive Action: Continue
            - Note: This pop-up is alarming, but (assuming you set an appropriate storage limit) nothing will be destroyed
        - Profile Configuration:
            - Your name: hawkid
            - Your server's name: hawkid_linux_sub
            - Pick a username: hawkid
            - Choose a password/Confirm your password: (HawkID password recommended)
        - Upgrade to Ubuntu Pro: Skip for now
        - SSH Configuration: (do not enable)
        - Featured server snaps: (select none)

        </details>

    1. When the installation is complete, select 'Reboot Now'
    1. After rebooting (on the screen with the 'Please remove the installation medium...' prompt), manually quit the VM by clicking the Power icon in the top left corner
    1. Click the Play icon to restart the VM
    1. Select the 'Boot from next volume' option
    1. Log in with the username and password created during installation

    </details>

1. Update Ubuntu
    1. Enter `sudo apt update` to check for updates
    1. Enter `sudo apt upgrade -y` to install the updates identified in the previous step
1. Install Apptainer
    1. Run the following code to install dependencies:

        ```bash
        sudo apt update
        sudo apt install -y \
        software-properties-common \
        squashfuse \
        fuse2fs \
        gocryptfs
        ```

    1. Run the following code to download and install Apptainer:

        ```bash
        sudo add-apt-repository -y ppa:apptainer/ppa
        sudo apt update
        sudo apt install -y apptainer
        ```

        - If Apptainer is not available via apt (`E: Unable to locate package apptainer`), install it from source

            <details>
            <summary>Source Installation</summary>

            1. Get the most current version number from [GitHub](https://github.com/apptainer/apptainer/releases) (do not need to download--just need version number)
            1. Define a variable to hold version info (insert version number in place of #s):

                ```bash
                export VERSION=#.#.#
                ```

            1. Download the source code:

                ```bash
                wget https://github.com/apptainer/apptainer/releases/download/v${VERSION}/apptainer-${VERSION}.tar.gz
                tar -xzf apptainer-${VERSION}.tar.gz
                cd apptainer-${VERSION}
                ```

            1. Build Apptainer:

                ```bash
                ./mconfig
                make -C builddir
                sudo make -C builddir install
                ```

            </details>

    1. Verify installation:

        ```bash
        apptainer --version
        ```

    1. If Apptainer was already installed (i.e., you did not just install it) and you want to upgrade the Apptainer installation, run the following code to upgrade Apptainer:

        ```bash
        sudo add-apt-repository -y ppa:apptainer/ppa
        sudo apt update
        sudo apt install --only-upgrade -y apptainer
        apptainer --version
        ```

        - If you had previously installed the old Apptainer version from source, you may have to remove it:

            <details>
            <summary>Remove Source Installation</summary>

            ```bash
            sudo rm /usr/local/bin/apptainer
            ```

            </details>

1. Go to your home directory (`cd ~`)
1. Create a `containers` folder if one does not exist:

    ```bash
    mkdir containers
    ```
    
### Definition Files

Definition files contain the necessary information for installing the container software (e.g., versions, packages).
Because the user (i.e., you) specifies the content of the definition file, there is a lot of flexibility when it comes to what to include and how to structure them.
However, there are a handful of necessary components for a definition file, which you can read about [here](https://apptainer.org/docs/user/latest/definition_files.html#definition-files).

There are a number of pre-written definition files for `R` (with or without `brms` and `CmdStan`), `MPlus`, `MATLAB`, and `Python`/`MNE`, which you can view or download [here](https://research-git.uiowa.edu/PetersenLab/HPC/-/tree/master/Argon/Containers/DefinitionFiles).
There are also instructions below for creating a definition file from scratch.
To build a container with a program that is not included here, see the Apptainer documentation for [building a container](https://apptainer.org/docs/user/latest/build_a_container.html) and for [including multiple apps in one container](https://apptainer.org/docs/user/latest/definition_files.html#scif-apps).

1. Go to the `containers` folder (`cd containers`)
1. If installing `MPlus`:
    1. Copy the `MPlus` installer into your `containers` folder:

        ```bash
        cp path/to/installer/MplusLinux64.bin ~/containers/
        ```
        
        Note: `/mnt/c` is equivalent to the `C:` drive of your main computer
        
    1. Run the installer to generate a properties file:

        ```bash
        sudo ./MplusLinux64.bin -r ~/containers/mplus_install.properties
        ```
        
        1. Agree to the terms of the license agreement
        1. Choose a minimal install (option `2`)
        1. Enter `/opt/mplus` as the installation path
    1. Uninstall `MPlus` from your VM (optional):
    
        ```bash
        cd /opt/mplus/uninstall
        sudo ./uninstall_mplus
        ```
        
1. If installing `MATLAB`:
    1. Copy the dependencies file into your `containers` folder (get the dependencies path for the latest released version from here: <https://github.com/mathworks-ref-arch/container-images/tree/main/matlab-deps>; make sure to specify the correct architecture—i.e., AMD or ARM):

        ```bash
        wget -q -O base-dependencies.txt https://raw.githubusercontent.com/mathworks-ref-arch/container-images/refs/heads/main/matlab-deps/r2026a/ubuntu24.04/base-dependencies.txt
        ```

1. Create or download the definition file
    1. To download an existing definition file (see [GitLab](https://research-git.uiowa.edu/PetersenLab/HPC/-/tree/master/Argon/Containers/DefinitionFiles) for options—make sure to update the version numbers for MATLAB, Python, and the Ubuntu OS as necessary):
        1. Copy the definition file into your containers folder:

            ```bash
            cp /mnt/path/to/def/file.def ~/containers
            # For example:
            # cp /mnt/c/Users/USERNAME/Downloads/file.def ~/containers
            ```
            
            Note: You must have the HPC repository cloned to your local computer for this to work.
    1. To create a definition file from scratch:
        1. Create a new file:

            ```bash
            nano argon_container.def
            ```

        1. Add the definition file header:

            ```bash
            Bootstrap: docker
            From: ubuntu:24.04
            ```

            Other common base images:
              - `anaconda/miniconda:latest` for a container with `miniconda` (helpful for `Python` environments)
              - `mathworks/matlab` for `MATLAB` only
              - `rocker/r-ver:latest` for `R`, `CmdStan`, and/or `MPlus`
                  - Note: The definition file below uses r2u, which automatically installs Linux dependencies for R packages.
                  The base `R` image at `rocker/r-ver` does not include r2u, so you will have to add it in the definition file or manually install the Linux dependencies.
        1. Add remaining parameters

            The script below includes parameters for `R` (with `brms`), `CmdStan`, `MPlus`, `MATLAB`, and `Python`/`MNE`.
            Paste the entire script into the definition file (right click to paste into the terminal).
            To install only a subset of the software, simply remove the section(s) associated with the unwanted software.
            Note that `MPlus` has a subsection in the `R` section in addition to its standalone section.
            Also note `CmdStan` does not have a standalone app section--it is installed as a subsection of the `R` installation.
              
            <details>
            <summary>All Software</summary>
            
              ```bash
              %environment
                
                  # MPlus paths
                  export PATH=/opt/mplus:$PATH
                  
                  # R paths
                  export R_LIBS_USER=/usr/local/lib/R/site-library
                  
                  # CmdStan paths
                  export CMDSTAN=/opt/cmdstan
                  export STAN_THREADS=true
                  
              %files
                  
                  # MPlus files
                  MplusLinux64.bin /opt/MplusLinux64.bin
                  mplus_install.properties /opt/mplus_install.properties
                  
                  # MATLAB files
                  base-dependencies.txt /opt/base-dependencies.txt
              
              %post
              
                  # install dependencies for all apps (do not delete)
                  # includes any dependencies required for 2+ software packages
                  apt-get update && apt-get install -y --no-install-recommends \
                    bzip2 \
                    ca-certificates \
                    cmake \
                    curl \
                    g++ \
                    git \
                    make \
                    nano \
                    python3 \
                    vim \
                    wget

                  # clean up package manager
                  apt-get clean -y
                  apt-get autoremove -y
                  rm -rf /var/lib/apt/lists/*
                  
              #############################################
              # Python/MNE
              #############################################
              
              %apprun python
              
                  exec python3 "$@"

              %appinstall python
              
                  # install pip
                  apt-get update && apt-get install -y --no-install-recommends \
                    python3-pip
                  
                  # install mne, overriding env warning
                  pip install --break-system-packages "mne[full]"
                  
                  # install hdf5 (large file handling) support packages, overriding env warning
                  pip install --break-system-packages h5io pymatreader
              
              #############################################
              # R
              #############################################
              
              %apprun R
              
                  exec R "$@"
              
              %appinstall R
              
                  # install additional r2u dependencies
                  # if installing R without r2u (from rocker base), add Linux deps here
                  # gnupg required for r2u, all others only needed for eegUtils package
                  apt-get update && apt-get install -y --no-install-recommends \
                    gnupg \
                    liblapack-dev \
                    libblas-dev \
                    gfortran
                  
                  # add r2u keys to apt folder so they are recognized as safe
                  gpg --homedir /tmp --no-default-keyring \
                    --keyring /usr/share/keyrings/r2u.gpg \
                    --keyserver keyserver.ubuntu.com \
                    --recv-keys A1489FE2AB99A21A 67C2D66C4B1D4339 51716619E084DAB9
                  
                  # add r2u mirror to apt source list to pull files
                  # do not change indentation - I know it's ugly but it will throw an error
                  cat > /etc/apt/sources.list.d/r2u.sources <<EOF
              Types: deb
              URIs: https://r2u.stat.illinois.edu/ubuntu
              Suites: noble
              Components: main
              Arch: amd64, arm64
              Signed-By: /usr/share/keyrings/r2u.gpg
              EOF
              
                  # add base r to apt source list just to be safe
                  cat > /etc/apt/sources.list.d/cran.sources <<EOF
              Types: deb
              URIs: https://cloud.r-project.org/bin/linux/ubuntu
              Suites: noble-cran40/
              Components:
              Arch: amd64, arm64
              Signed-By: /usr/share/keyrings/r2u.gpg
              EOF
                  
                  # install base r
                  apt-get update && apt-get install -y --no-install-recommends r-base-core
                  
                  # pin r2u to top of package priority list so it gets picked for installs
                  # don't change indentation here either
                  cat > /etc/apt/preferences.d/99cranapt <<EOF
              Package: *
              Pin: release o=CRAN-Apt Project
              Pin: release l=CRAN-Apt Packages
              Pin-Priority: 700
              EOF
                  
                  # install python dependencies for bspm
                  apt-get update && apt-get install -y --no-install-recommends \
                    python3-dbus \
                    python3-gi \
                    python3-apt
                    
                  # install bspm
                  Rscript -e "install.packages(
                    'bspm',
                    repos = 'https://cloud.r-project.org'
                  )"
                  
                  # update bspm settings
                  # don't change indentation
                  mkdir -p /etc/R
                  cat >> /etc/R/Rprofile.site <<EOF
              suppressMessages(bspm::enable())
              options(bspm.version.check = FALSE)
              EOF
                  
                  # install packages (add/remove as needed)
                  R -e "install.packages(
                    c(
                      'brms',
                      'data.table',
                      'devtools',
                      'flowchart',
                      'fs',
                      'ggh4x',
                      'gt',
                      'gtsummary',
                      'janitor',
                      'lavaan',
                      'lme4',
                      'parallely',
                      'performance',
                      'psych',
                      'R.matlab',
                      'remotes',
                      'renv',
                      'semTools',
                      'tidyverse'
                    ),
                    repos = 'https://cloud.r-project.org'
                  )"
                  
                  # install packages via remotes
                  R -e "remotes::install_github('DevPsyLab/petersenlab')"
                  R -e "remotes::install_github('craddm/eegUtils')"
                  
                  # clean up package manager
                  apt-get clean -y
                  apt-get autoremove -y
                  rm -rf /var/lib/apt/lists/*
                  
                  #######
                  # Mplus
                  #######

                  # install R package
                  R -e "install.packages(
                    'MplusAutomation',
                    repos = 'https://cloud.r-project.org'
                  )"
                  
                  #############
                  # CmdStan
                  #############
                  
                  # install R package
                  R -e "install.packages(
                    'cmdstanr',
                    repos = c(
                      'https://stan-dev.r-universe.dev',
                      'https://cloud.r-project.org'
                    )
                  )"
                  
                  # create folder to install into
                  mkdir -p /opt/cmdstan
                  
                  # install CmdStan using R
                  R -e "cmdstanr::install_cmdstan(
                    cores = 2,
                    dir = '/opt/cmdstan'
                  )"
                  
                  # create an R environment file
                  mkdir -p /root/.R
                  
                  # add threading vars to environment and startup
                  echo 'STAN_THREADS=true' >> /root/.Renviron
                  echo 'STAN_THREADS=true' >> /root/.R/Makevars

                  # clean up package manager
                  apt-get clean -y
                  apt-get autoremove -y
                  rm -rf /var/lib/apt/lists/*
                  
              #############################################
              # MPlus
              #############################################
              
              %apprun mplus
              
                  exec mplus "$@"
                  
              %appinstall mplus
              
                  chmod +x /opt/MplusLinux64.bin
                  /opt/MplusLinux64.bin -i silent -f /opt/mplus_install.properties
                  chmod -R 755 /opt/mplus
                  
              #############################################
              # MATLAB
              #############################################
              
              %apprun matlab
              
                  exec matlab -nodesktop -nojvm -nosplash "$@"
                  
              %appinstall matlab
              
                  # define var for noninteractive install
                  DEBIAN_FRONTEND=noninteractive \
                  
                  # install additional dependencies
                  apt-get update && apt-get install -y --no-install-recommends \
                    `cat /opt/base-dependencies.txt`
                    
                  # clean up package manager
                  apt-get clean -y
                  apt-get autoremove -y
                  rm -rf /var/lib/apt/lists/*
              
                  # define variables needed for setup
                  export MATLAB_RELEASE=R2026a
                  export MATLAB_PRODUCT_LIST="MATLAB Wavelet_Toolbox Signal_Processing_Toolbox Statistics_and_Machine_Learning_Toolbox Optimization_Toolbox Parallel_Computing_Toolbox"
                  export MATLAB_INSTALL_LOCATION=/opt/matlab/${MATLAB_RELEASE}
                  
                  # add those variables to apptainer environment
                  # do not change the line indentation - it will cause an error
                  cat << EOF > $APPTAINER_ENVIRONMENT
              export MATLAB_RELEASE=$MATLAB_RELEASE
              export MATLAB_PRODUCT_LIST=$MATLAB_PRODUCT_LIST
              export MATLAB_INSTALL_LOCATION=$MATLAB_INSTALL_LOCATION
              EOF
                  
                  # make sure apptainer has full permissions in tmp
                  chmod 777 /tmp
                  
                  # get matlab package manager (mpm) and update permissions
                  wget -q https://www.mathworks.com/mpm/glnxa64/mpm
                  chmod +x mpm
                  
                  # run mpm to install matlab
                  ./mpm install \
                    --release=${MATLAB_RELEASE} \
                    --destination=${MATLAB_INSTALL_LOCATION} \
                    --products ${MATLAB_PRODUCT_LIST}
                  
                  # clean up after installer
                  rm -f mpm /tmp/mathworks_root.log
                  ln -s ${MATLAB_INSTALL_LOCATION}/bin/matlab /usr/local/bin/matlab
                  
                  # add license info
                  mkdir -p ${MATLAB_INSTALL_LOCATION}/licenses
                  cat << EOF > ${MATLAB_INSTALL_LOCATION}/licenses/network.lic
              SERVER matlab-lm.iowa.uiowa.edu 27000
              USE_SERVER
              EOF
              ```

            </details>

        1. Press `Ctrl+O` then `Enter` to save the file
        1. Press 'Ctrl+X' to close the file
1. Build the container:

    ```bash
    sudo apptainer build argon_container.sif argon_container.def
    ```

    You may be prompted to enter your (admin login) password.
    It will take a few minutes to build, and a lot of text will appear in the terminal.
1. Test the container

    If you used a definition file without an app structure (i.e., no `%appinstall` sections in the definition file), use `apptainer exec argon_container.sif path/to/software/exec_file` instead of `apptainer run --app app_name argon_container.sif` to start the program.
    1. `Python`/`MNE`
        1. Start the `Python` app within the container:

            ```bash
            apptainer run --app python argon_container.sif
            ```

            If the app is working, you should see the `Python` start-up message, followed by `>>>` in the final line.
        1. Verify the `MNE` installation:

            ```python
            import mne; mne.sys_info()
            ```

            You should see a printout of the system information, including a list of installed libraries.
        1. Enter `exit()` to leave the container
    1. `R`
        1. Start the `R` app within the container:

            ```bash
            apptainer run --app R argon_container.sif
            ```

            If the app is working, you should see the `R` start-up message, followed by `>` in the final line.
            Containers with `r2u` may show a warning about D-Bus services--this will not affect functionality.
            Containers run without `sudo` or admin access may show a warning about the `brms` package needing admin permissions--this will not affect functionality.
        1. Check `R` version:

            ```r
            version
            ```

            You should see a printout of the version information.
        1. Check package installations:

            ```r
            installed.packages()
            ```

            Ensure that all desired packages have been installed.
        1. Enter `q()` to leave the container
    1. `MPlus`
        1. Check the `MPlus` installation:

            ```bash
            apptainer exec argon_container.sif mplus "-version"
            ```
            
            If the app is working, you should see the `MPlus` version and installation type printed in the terminal.
    1. `MATLAB`
        1. Start the `MATLAB` app within the container:

            ```bash
            apptainer run --app matlab argon_container.sif
            ```

            If the app is working, you should see the `MATLAB` start-up message, followed by `>` in the final line.
            Because `MATLAB` verifies its license via network connection, you must be connected to UIowa Wi-Fi (or be using the VPN) for it to work.
        1. Check `MATLAB` version:

            ```Matlab
            ver
            ```

            You should see a printout of the version information, including installed products.
        1. Enter `exit` to leave the container
1. Copy the container to your computer:

    ```bash
    cp ~/containers/r_argon.sif "/mnt/c/Users/hawkid/path/to/folder/"
    ```

    The `/mnt/c` path is equivalent to your computer's `C:` drive, so the rest of the file path should be defined from there.
1. If your admin login is different from your HawkID, restart the terminal (or emulator) to run it as a regular user
1. Copy the container to Argon, either by using the terminal or by manually moving it in File Explorer/Finder
1. Test the container on Argon, using the same steps described above
1. If you installed `CmdStan`, copy the executable files to a writable location:

    ```bash
    mkdir -p $HOME/cmdstan_cache
    apptainer exec /Users/hawkid/path/to/container/argon_container.sif \
      cp -r /opt/cmdstan/. $HOME/cmdstan_cache
    ```

1. To submit a job using the container, use the `apptainer exec` command:

    ```bash
    # run an R script using a container built with %app sections
    /usr/bin/apptainer exec --app R /Users/hawkid/container_path/argon_container.sif my_script.R
    
    # run a MATLAB script using a container built without %app sections
    /usr/bin/apptainer exec /Users/hawkid/container_path/argon_container.sif \
      matlab -nodesktop -nojvm -nosplash my_script.mat
    ```

    For an example job script using a container, see @sec-jobScript.
    
# Jobs {#sec-jobs}

In order to "run" scripts from a log-in node, you will need to create a job script.
A job script contains all of the commands that you would ordinarily run individually in the terminal to run a script, along with some parameters that tell Argon where and how to run the commands.
At minimum, a job script should contain a shell specification and at least one program, script, or command to execute.
However, there are several additional parameters you can include to control how the script is run.

Basic Job Submission: <https://uiowa.atlassian.net/wiki/spaces/hpcdocs/pages/76513450/Basic+Job+Submission>

Advanced Job Submission: <https://uiowa.atlassian.net/wiki/spaces/hpcdocs/pages/76513452/Advanced+Job+Submission>

## Job Commands

Below are a variety of common commands for submitting and tracking jobs.

- `qsub jobScript.job` submits a job script to run
    - `qsub -hold_jid JOB_ID jobScript.job` submits `jobScript.job` to run *after* job `JOB_ID` is finished
- `qstat` shows the status of available queues and jobs
    - `qstat -u hawkid` shows jobs submitted by the specified HawkID
    - `qstat -g c -q QUEUE_NAME` shows the status of the specified queue
    - `qstat -j JOB_ID` shows detailed satus info for the specified job (if still running)
    - Status Abbreviations:
        - `d` - deleted
        - `E` - error
        - `h` - held by user
        - `r`, `t` - running or transferring to run status
        - `R` - restarted
        - `s`  `S` - job (s) or queue (S) suspended
        - `T` - queue threshold exceeded (suspended)
        - `w` - waiting in queue
- `qacct` summarizes job and user info that meet the specified parameters
    - `qacct -j JOB_ID` shows info for the specified job (after running)
    - `qacct -o hawkid -j -d n` shows info on the specified user's jobs for the past `n` days
- `qdel JOB_ID` deletes the specified job
    - `qdel -u hawkid` deletes all jobs submitted by the specified user

## Writing Job Scripts {#sec-writingJobScripts}

1. Log in to Argon
1. Create the job script:

    ```bash
    nano jobScript.job
    ```

1. Specify the shell:
    
    ```bash
    #!/bin/sh
    ```
    
    This line indicates which shell (CLI language) to use when running the rest of the job script.
    In this case, `sh` sets the shell to the system's default shell.
    Regardless of which shell you specify, this line must start with `#!`.

1. Add `qsub` options (optional)
    
    The next few lines contain settings for the `qsub` command, which is the command used for submitting job scripts.
    You can also specify these options in the command line, but that can lead to lengthy inputs.
    When specifying `qsub` options in your job script, each one must have `#$` at the start of the line.
    Below are options commonly used in the DevPsy job scripts.
    
    | Option  | Description |
    | ------- | ----------------------------------------  |
    | -cwd  | execute the job from the current working directory |
    | -e [path] | file name or path  | file or folder for saving error messages |
    | -j [y\|n]  | whether the error messages should be saved in the same file as outputs (y) or not (n) |
    | -l mf=[nG\|nM] | number (n) of gigabytes (G) or megabytes (M) of free memory needed at startup  |
    | -l h_vmem=[nG\|nM] | number (n) of gigabytes (G) or megabytes (M) of virtual memory needed across entire job |
    | -M [email]  | email to send job notifications to  |
    | -m [b\|e\|a\|s\|n]  | when to send job notifications (b = beginning of job; e = end of job; a = job aborted; s = job suspended; n = no notifications); multiple values allowed (e.g., -m eas)  |
    | -N [name] | job name  |
    | -o [path] | file or folder for saving non-error output messages |
    | -pe smp [cores] | number of cores to use for parallel computing[^1]  |
    | -q [UI\|UI-HM\|UI-MPI\|UI-GPU\|UI-DEVELOP\|all.q] | which job queue to use[^2]  |
    | -t n-m[:s]  | indicates an array job; includes the index of the first (n) and last (m) task with an optional parameter for step size (s; defaults to 1)  |
    | -tc [n] | number of jobs in an array job to run simultaneously  |
    | -wd [path]  | execute the job from the specified folder |
    
    [^1]: We typically use the `smp` parallel computing environment, but other options are listed on the [Advanced Job Submission](https://uiowa.atlassian.net/wiki/spaces/hpcdocs/pages/76513452/Advanced+Job+Submission) documentation page and can be substituted for `smp`.
    
    [^2]: The `UI` (default) and `UI-HM` (high-memory) queues are the most used by DevPsy.
    `UI-MPI` is for jobs that can use multiple nodes simultaneously, and `UI-GPU` is for jobs that can use a GPU accelerator.
    `UI-DEVELOP` is recommend for test runs and debugging.
    The `all.q` queue is the lowest priority queue, so jobs running on it will "give up" resources if needed for jobs in higher-priority queues.
    
1. Specify the working directory (optional)

    Argon defaults to running the job script from your home directory (`/Users/hawkid`).
    However, if you would like to run the job from a different directory, you can either use the `-wd` option described in the previous step or manually specify the working directory in the body of the job script:
    
    ```bash
    # set working directory
    cd /Users/hawkid/path/to/working/directory/
    ```
    
1. Write the job script

    The remainder of the job script is extremely flexible, because it is whatever set of commands you would use to run your script "live" in the terminal.
    Typically, job scripts start by defining any variables needed for the rest of the script and loading necessary modules.
    After that, the commands specify which scripts/programs to run and when.
    See @sec-jobScript for examples of a variety of job scripts, and see @sec-bashScripting for common `bash` commands used in job scripts.
    
1. Press `Ctrl+O` then `Enter` to save the file
1. Press 'Ctrl+X' to close the file

## Example Job Scripts {#sec-jobScript}

### Using A Container

```bash
#!/bin/sh

# Set working directory
cd /Users/itpetersen/Documents/Projects/Bayesian_IRT/

# Specify qsub options
#$ -pe smp 4
#$ -M isaac-t-petersen@uiowa.edu
#$ -m eas
#$ -l mf=512G
#$ -l h_vmem=512G
#$ -cwd
#$ -q UI-HM
#$ -e /Users/itpetersen/Documents/Projects/Bayesian_IRT/Output/
#$ -o /Users/itpetersen/Documents/Projects/Bayesian_IRT/Output/

# Load the environment modules
module purge

# Run the R script inside container
apptainer exec \
    --bind $HOME/cmdstan_cache:/opt/cmdstan \
    --cleanenv \
    /Users/itpetersen/Documents/Containers/r_argon.sif \
    Rscript ./Analyses/factorScores.R
```

### Using Software Stacks

```bash
#!/bin/sh

# Set working directory
cd /Users/itpetersen/Documents/Projects/Bayesian_IRT/

# Specify qsub options
#$ -pe smp 4
#$ -M isaac-t-petersen@uiowa.edu
#$ -m eas
#$ -l mf=512G
#$ -l h_vmem=512G
#$ -cwd
#$ -q UI-HM
#$ -e /Users/itpetersen/Documents/Projects/Bayesian_IRT/Output/
#$ -o /Users/itpetersen/Documents/Projects/Bayesian_IRT/Output/

# Load the environment modules
module purge
module load stack/2022.2
module load r/4.2.2_gcc-9.5.0

# Run the R script
Rscript ./Analyses/factorScores.R
```

### Array Job with Variables

```bash
#!/bin/bash

# set job parameters
#$ -N tf_visualization
#$ -o /Users/mschumer/logs/$JOB_NAME_output.txt
#$ -e /Users/mschumer/logs/$JOB_NAME_errors.txt
#$ -q UI
#$ -l mf=36G
#$ -l h_vmem=36G
#$ -t 1-535
#$ -tc 16
#$ -pe smp 16

# activate mne environment
conda activate /Users/mschumer/anaconda3/envs/mne

# get list of subjects
cd /Shared/lss_PetersenLabEEGVideo/EEG/Oddball/2_TFData/1_preprocessed
subjects=($(ls))
cd /Users/mschumer

# select subject using current idx
sub=${subjects[${SGE_TASK_ID}-1]}

# set values for input and output paths
input_path=/Shared/lss_PetersenLabEEGVideo/EEG/Oddball/2_TFData/1_preprocessed
output_path=/Shared/lss_PetersenLabEEGVideo/EEG/Oddball/2_TFData/3_tf_processed

# run script
echo "${input_path} ${output_path} ${sub}" | python /Users/mschumer/tf_analysis.py
```

# Stan

For instructions on how to install `Stan`, see the documentation: <https://github.com/stan-dev/rstan/wiki/Installing-RStan-on-Linux> (archived at <https://perma.cc/89H9-L8S6>)

If your version of `Stan` requires `Makevars`, save it under `\\data.hpc.uiowa.edu\argon_home\.R\Makevars`.

# Troubleshooting

## Running out of Memory {#sec-outOfMemory}

May need to request more memory, either by requesting more slots (to ensure more memory; i.e., `-pe smp <SLOTS>`), by increasing how much memory is available for the node at the moment your job starts (i.e., `mf=<SIZE>`), and/or by increasing the hard limit for virtual memory (i.e., `h_vmem=<SIZE>`); see @sec-writingJobScripts.

To see how much memory was used by a killed job, run `qacct -j JOBID` and look at: `maxvmem`.

This error can generate the following error messages:

- `job JOB_ID died through signal TERM (15)`
- `job [NUMBER] died through signal KILL (9)`
- `Exit Status: 143`
- `Exit Status: 137`

## Having trouble installing `R` packages?

### Containers {#sec-installPackagesContainers}

Unless you built a sandbox container, you will not be able to install packages, because container files are not editable.
To add new packages, you need to create a new container using an updated definition file with all desired packages.
Alternatively, you can create a sandbox container based on your current container, add the desired packages, and convert the sandbox container to a static build:

```bash
apptainer build --sandbox newContainer/ oldContainer.sif
apptainer exec --writable newContainer R -e "install.packages(c('package1', 'package2'), repos = 'https://cloud.r-project.org')"
apptainer build newContainer.sif newContainer/
```

For transparency, it is preferable to create a new container from an updated definition file, which ensures all container build info is available.

### Modules

Sometimes Argon will fail to install packages in an R workspace.
We advise the following steps:

1. Sometimes R provides an error stating something like "fatal error: modulename: No such file or directory".
In this instance, you may wish to exist R with q() and load all the modules that begin with the listed module name, as listed above in "Installing Linux Packages to Install R Packages".
Then reopen R and try to install the package again.
1. You can also try installing the package from binary using the RStudio Package Manager (RSPM):

    ```r
    remotes::install_github("cran4linux/rspm")
    rspm::enable()
    install.packages("PACKAGE_NAME")
    ```

1. If that does not work, you can try downloading the .tar file directly from the CRAN repository. Then copy the file into your Argon folder and type 'install.packages("/Users/path/to/directory/package_name", repos = NULL, type = "source")'.
Note that you may need to download an older version of the package (e.g., from the [CRAN Archive](https://cran.r-project.org/src/contrib/Archive/)), such that it is compatible with the version of R you are running on Argon (which it typically not the most recent R version):

    ```r
    install.packages("http://cran.r-project.org/src/contrib/Archive/MASS/MASS_7.3-60.0.1.tar.gz", repos = NULL, type = "source")
    ```

1. Check if you are installing the necessary modules within your job script: 

    ```r
    module load stack/2022.2
    module load r/4.2.2_gcc-9.5.0
    ```
1. Add a line to your R script to make sure it is accessing the same repository where you installed the packages manually:

    ```r
    .libPaths(c("/old_Users/HAWKID/Rlibs", 
            "/cvmfs/argon.hpc.uiowa.edu/2022.2/apps/linux-centos7-broadwell/gcc-9.5.0/r-4.2.2-lv7yirk/rlib/R/library"))
    ```

1. If the prior steps fail, email research-computing@uiowa.edu with the error, asking for help to figure out how to install the packages.

## Error When Installing Packages from Source

You may have to set environment variables in each module file to help the compiler find headers and libraries.
Note that if you run into any C++ code, you will need to set the `CPLUS_INCLUDE_PATH` variable.

### `nloptr`

1. Run the following commands

```bash
module load stack/2022.1
module load r/4.1.3_gcc-9.4.0
module load nlopt
```

2. Set `LIBRARY_PATH` so linker can find library while launching `R` session (single line below):

```
LIBRARY_PATH=$ROOT_NLOPT/lib64:$LIBRARY_PATH R
```

3. In `R` console, install `nloptr` (two lines below)

```{r}
#| eval: false

install.packages(verbose = 1, "nloptr")
```

### `tkrplot`

1. Run the following commands

```bash
module load stack/2022.1
module load r/4.1.3_gcc-9.4.0
```

2. Set LIBRARY_PATH so linker can find library while launching `R` session (single line below):

```
C_INCLUDE_PATH=$ROOT_XPROTO/include LIBRARY_PATH=$ROOT_LIBXEXT/lib:$ROOT_LIBXSCRNSAVER/lib R
```

3. In `R` console, install `tkrplot` (two lines below)

```{r}
#| eval: false

install.packages(verbose = 1, "tkrplot")
```

## Error Reading from Connection

```r
Error in unserialize(node$con) : error reading from connection
Calls: parlmice ... FUN -> recvData -> recvData.SOCK0node -> unserialize
```

There likely wasn't sufficient memory for a given core.
Try increasing the max memory available and decreasing the number of cores and/or slots, so there is more memory available per core:

<https://stackoverflow.com/questions/46186375/r-parallel-error-in-unserializenodecon-in-hpc> (archived at <https://perma.cc/MF6V-NAVS>)

<https://stackoverflow.com/questions/17015598/error-calling-serialize-r-function> (archived at <https://perma.cc/3Q75-DA2D>)

<https://gforge.se/2015/02/how-to-go-parallel-in-r-basics-tips/#Memory_load> (archived at <https://perma.cc/2JRF-8Y5F>)

## Error: `$'\r': command not found`

```bash
/opt/sge/default/spool/argon-itf-bx47-36/job_scripts/5685660: line 2: $'\r': command not found
```

That often means your job script has Windows (CRLF) line endings instead of Unix (LF).
You can fix the line endings with the following command:

```bash
dos2unix your_job_file.job
```

# Appendix {#sec-appendix}
    
## Old Stacks

### Stack 2022.1

```bash
module load stack/2022.1
module load r/4.1.3_gcc-9.4.0
cd path
```

### Stack 2021.1

```bash
module load stack/2021.1
module load r/4.0.5_gcc-9.3.0
cd path
```

### Stack 2020.2

```bash
module load stack/2020.2
module load r/4.0.2_gcc-8.4.0
cd path
```

### Stack 2020.1

```bash
module load stack/2020.1
module load r/3.6.2_gcc-9.2.0
cd path
```

## Managing packages for a project using `packrat`

Please note: we now use [`renv`](#sec-renv) rather than `packrat` for package management

<https://uiowa.atlassian.net/wiki/spaces/hpcdocs/pages/76514803/Adding+R+programs+to+a+personal+library> (archived at <https://perma.cc/3SRR-2JE7>)

1. Create a directory where you want you want to store the project.
1. `cd` to the above directory
1. Start R
1. Load the `packrat` package: `library("packrat")`
1. At the `R` prompt, initialize the `packrat` project on the local repository of `R` packages with: `packrat::init(project = "/Users/itpetersen/Documents/Projects/INSERT_PROJECT_NAME/", options = list(local.repos = "/Users/itpetersen/R/x86_64-pc-linux-gnu-library/4.0"))`
1. You must restart your `R` session in the given project directory after running init in order for the changes to take effect!
1. From this point on you are working in a `packrat` project.
Installed packages will go into a library within this project.
After initializing the `packrat` project on the local repository of `R` packages, packages from the local repository can be installed with `packrat::install_local()`:
    - `packrat::install_local("package_name")`
1. To restart a packrat project simply start `R` from the directory created in step (1). The project will initialize automatically.

To save the current state of your library: `packrat::snapshot()`; if that command fails due to an error when fetching sources, try `packrat::snapshot(snapshot.sources = FALSE)` 

To disable `packrat` on a project: `disable(restart = FALSE)`

### To install a new package for a project after `packrat` has been initialized

1. First, load the `R` environment module (see `Install R packages` section above)
1. cd to the directory of the `R` project
1. launch R
    - `R`
1. then install the package, (the code below uses a generic `package_name`):
    - `install.packages("package_name", repos = "http://cran.r-project.org", type = "source", dependencies = TRUE, Ncpus = 40)`
    - `install.packages(c("packrat","psych","tidyverse","data.table","nlme","lme4","mirt","TeachingDemos","Amelia","mice","miceadds","abind","future","lavaan","blavaan","Rcpp","igraph","shinystan","StanHeaders","brms","rstan","rjags","renv"), repos = "http://cran.r-project.org", type = "source", dependencies = TRUE, Ncpus = 40)`
1. `packrat::snapshot()`; if that command fails due to an error when fetching sources, try `packrat::snapshot(snapshot.sources = FALSE)`

### Build `packrat` environment on a compute node (only if necessary if package load fails due to issues building packages)

1. `qlogin` -- for more info, see here: <https://uiowa.atlassian.net/wiki/spaces/hpcdocs/pages/76513454/Qlogin+for+Interactive+Sessions> (archived at <https://perma.cc/Y3A9-WQ3W>)
1. `module load stack/2020.2-base_arch` -- this will ensure that the modules point to the lowest common multiarchitecture on Argon and will run on all nodes
1. `module load r`
1. cd to the directory of the `R` project
1. `R`
1. Create `packrat` project (see above)
1. Install packages
1. Take a package snapshot using `packrat`
1. `exit` -- to exit the qlogin session

Note: Your `packrat` environment will then be linked to the proper `glpk` library and will run on any Argon node.
You do not need to use the `2020.2-base_arch` module at run time, only build time.