Key Concepts

This section defines some key concepts that users should acknowledge before submitting jobs in the Magellano infrastructure.

Batch vs. Interactive Jobs

Slurm jobs can be run in interactive mode or batch mode (i.e. non-interactive).

Interactive Jobs (srun)

You can submit interactive Slurm jobs using the srun command, as in the following example:

srun --partition=levante --nodes=1 --tasks-per-node=4 --cpus-per-task=1 [your executable] ...

The executable will be run as soon as the resources are available and the output will be printed on your terminal.

With the same command, you can start an interactive shell on a node of the cluster by specifying the --pty option:

srun --partition=levante --nodes=1 --tasks-per-node=4 --cpus-per-task=1 --pty /bin/bash -l

Batch Jobs (sbatch)

A batch job is a script that is submitted using the sbatch command and is run non-interactively.

You write a batch script and submit it. Slurm will run it as soon as the required resources become available. You can specify sbatch command line options at the top of the script, one per line, using comments. For example:

#!/bin/bash
#SBATCH --partition levante
#SBATCH --ntasks 1
#SBATCH --cpus-per-task 8

[your batch workflow here]

If your workflow includes MPI steps, you can use sbatch and require multiple parallel tasks distributed on multiple nodes. For example:

#!/bin/bash
#SBATCH --partition levante
#SBATCH --nodes 4
#SBATCH --tasks-per-node 2
#SBATCH --cpus-per-task 1

[some serial workload]

srun [your MPI executable]

Note that srun is called inside the batch script: this is the recommended way of running non-interactive parallel jobs in Slurm.

Slurm Users and Accounts

Slurm keeps information about Magellano users and resource accounts in its database.

A Slurm user is not necessarily identified with a Unix user. However, in Magellano we keep a 1:1 relationship between Slurm users and INFN AAI identities. When you log into the Magellano shell for the first time, a Slurm user matching your INFN username is automatically created. All the jobs submitted from your shell environment will belong to your personal user.

Slurm accounts must be intended as wallets that keep information about your resource share and usage. In Magellano, we assign accounts to specific experiments or research groups that own a share of the resources. All users that belong to an eligible group are automatically associated to the corresponding account.

All users are also associates to the default account. The default account allows you to charge resource usage to yourself, that is, without consuming the resource shares of your research group. When specifying the default account, users can submit to any public node in the infrastructure. Jobs submitted with the default account have lower priority compared to group-specific accounts (see also Job Priority).

You can check your associations with the following command:

sacctmgr list assoc where user=$USER

Walltime

The walltime is the maximum real-world time limit your job is allowed to run.

All your jobs are scheduled with a default walltime, which may depend on both the target partition and the Slurm account. You can specify a walltime for your job as follows:

#SBATCH --time HH:MM:SS

If your job runs longer than the walltime you request, Slurm kills the job.

For the sake of fairness, you should always explicitly set a walltime for your jobs. This way, the scheduler will allocate resources more efficiently by filling short gaps with short jobs (backfilling). Specifying the walltime also prevents unnecessary killing of jobs, in case the default walltime is too short.

Job Priority

Whenever a job is queued, Slurm calculates a priority, which determines in which order the pending jobs will start. The higher the priority (compared to other pending jobs), the sooner the job will start.

In Magellano, we use the Slurm-native multi-factor priority. In particular, priority is the sum of these factors:

  • Account: accounts have different priority weights, and the relative weight between accounts determines the initial job priority;

  • Fairshare: it depends on your share usage relative to siblings in the same account, and for relative usage between accounts;

  • Age: the priority of a job increases linearly in time.

You can check the priority of your pending jobs with the sprio command (see the man page).