Key Concepts ============ This section defines some key concepts that users should acknowledge **before submitting jobs** in the Magellano infrastructure. Batch vs. Interactive Jobs -------------------------- Slurm jobs can be run in interactive mode or batch mode (i.e. non-interactive). Interactive Jobs (``srun``) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ You can submit interactive Slurm jobs using the ``srun`` command, as in the following example:: srun --partition=levante --nodes=1 --tasks-per-node=4 --cpus-per-task=1 [your executable] ... The executable will be run as soon as the resources are available and the output will be printed on your terminal. With the same command, you can start an **interactive shell** on a node of the cluster by specifying the ``--pty`` option:: srun --partition=levante --nodes=1 --tasks-per-node=4 --cpus-per-task=1 --pty /bin/bash -l Batch Jobs (``sbatch``) ^^^^^^^^^^^^^^^^^^^^^^^ A batch job is a script that is submitted using the ``sbatch`` command and is run non-interactively. You write a **batch script** and submit it. Slurm will run it as soon as the required resources become available. You can specify ``sbatch`` command line options at the top of the script, one per line, using comments. For example:: #!/bin/bash #SBATCH --partition levante #SBATCH --ntasks 1 #SBATCH --cpus-per-task 8 [your batch workflow here] If your workflow includes MPI steps, you can use ``sbatch`` and require multiple parallel tasks distributed on multiple nodes. For example:: #!/bin/bash #SBATCH --partition levante #SBATCH --nodes 4 #SBATCH --tasks-per-node 2 #SBATCH --cpus-per-task 1 [some serial workload] srun [your MPI executable] Note that ``srun`` is called inside the batch script: this is the recommended way of running non-interactive parallel jobs in Slurm. Slurm Users and Accounts ------------------------ Slurm keeps information about Magellano users and resource accounts in its **database**. A **Slurm user** is not necessarily identified with a Unix user. However, in Magellano we keep a 1:1 relationship between Slurm users and **INFN AAI identities**. When you log into the Magellano shell for the first time, a Slurm user matching your INFN username is automatically created. All the jobs submitted from your shell environment will belong to your personal user. **Slurm accounts** must be intended as *wallets* that keep information about your resource share and usage. In Magellano, we assign accounts to specific **experiments or research groups** that own a share of the resources. All users that belong to an eligible group are automatically associated to the corresponding account. All users are also associates to the ``default`` account. The ``default`` account allows you to charge resource usage `to yourself`, that is, without consuming the resource shares of your research group. When specifying the ``default`` account, users can submit to any public node in the infrastructure. Jobs submitted with the ``default`` account have **lower priority** compared to group-specific accounts (see also `Job Priority `__). You can check your associations with the following command:: sacctmgr list assoc where user=$USER Walltime -------- The walltime is the maximum **real-world time limit** your job is allowed to run. All your jobs are scheduled with a default walltime, which may depend on both the target partition and the Slurm account. You can specify a walltime for your job as follows:: #SBATCH --time HH:MM:SS If your job runs longer than the walltime you request, Slurm kills the job. For the sake of **fairness**, you should always explicitly set a walltime for your jobs. This way, the scheduler will allocate resources more efficiently by filling short gaps with short jobs (`backfilling`). Specifying the walltime also prevents unnecessary killing of jobs, in case the default walltime is too short. Job Priority ------------ Whenever a job is queued, Slurm calculates a priority, which determines **in which order** the pending jobs will start. The higher the priority (compared to other pending jobs), the sooner the job will start. In Magellano, we use the Slurm-native **multi-factor priority**. In particular, priority is the sum of these factors: * **Account:** accounts have different priority weights, and the relative weight between accounts determines the initial job priority; * **Fairshare:** it depends on your share usage relative to siblings in the same account, and for relative usage between accounts; * **Age:** the priority of a job increases linearly in time. You can check the priority of your pending jobs with the ``sprio`` command (see the `man page `__).