configuration information, the nodes to be managed, information about
how those nodes are grouped into partitions, and various scheduling
parameters associated with those partitions. This file should be
consistent across all nodes in the cluster.
.LP
The file location can be modified at execution time by setting the SLURM_CONF
environment variable. The Slurm daemons also allow you to override
both the built\-in and environment\-provided location using the "\-f"
option on the command line.
.LP
The contents of the file are case insensitive except for the names of nodes
and partitions. Any text following a "#" in the configuration file is treated
as a comment through the end of that line.
Changes to the configuration file take effect upon restart of
Slurm daemons, daemon receipt of the SIGHUP signal, or execution
of the command "scontrol reconfigure" unless otherwise noted.
.LP
If a line begins with the word "Include" followed by whitespace
and then a file name, that file will be included inline with the current
configuration file. For large or complex systems, multiple configuration files
may prove easier to manage and enable reuse of some files (See INCLUDE
MODIFIERS for more details).
.LP
Note on file permissions:
.LP
The \fIslurm.conf\fR file must be readable by all users of Slurm, since it
is used by many of the Slurm commands.  Other files that are defined
in the \fIslurm.conf\fR file, such as log files and job accounting files,
may need to be created/owned by the user "SlurmUser" to be successfully
accessed.  Use the "chown" and "chmod" commands to set the ownership
and permissions appropriately.
See the section \fBFILE AND DIRECTORY PERMISSIONS\fR for information
about the various files and directories used by Slurm.

.SH "PARAMETERS"
.LP
The overall configuration parameters available include:

.TP
\fBAccountingStorageBackupHost\fR
The name of the backup machine hosting the accounting storage database.
If used with the accounting_storage/slurmdbd plugin, this is where the backup
slurmdbd would be running.
Only used with systems using SlurmDBD, ignored otherwise.
.IP

.TP
\fBAccountingStorageEnforce\fR
This controls what level of association\-based enforcement to impose
on job submissions.  Valid options are any combination of
\fIassociations\fR, \fIlimits\fR, \fInojobs\fR, \fInosteps\fR, \fIqos\fR,
\fIsafe\fR, and \fIwckeys\fR, or \fIall\fR for all things (except nojobs
enforced, users can be limited by association to whatever job size or run
time limits are defined.

If \fInojobs\fR is set, Slurm will not account for any jobs or steps on the
system. Likewise, if \fInosteps\fR is set, Slurm will not account for any
steps that have run.

If \fIsafe\fR is enforced, a job will only be launched against an association
or qos that has a TRES\-minutes limit set, if the job will be able to
run to completion. Without this option set, jobs will be launched as long as
their usage hasn't reached the TRES\-minutes limit. This can lead to jobs being
launched but then killed when the limit is reached.
With the 'safe' option set, a job won't be killed due to limits,
even if the limits are changed after the job was started and the
association or qos violates the updated limits.

With \fIqos\fR and/or \fIwckeys\fR enforced jobs will not be scheduled unless
a valid qos and/or workload characterization key is specified.

A restart of slurmctld is required for changes to this parameter to take effect.
.IP

.TP
\fBAccountingStorageExternalHost\fR
A comma\-separated list of external slurmdbds (<host/ip>[:port][,...]) to
register with. If no port is given, the \fBAccountingStoragePort\fR will be
used.

This allows clusters registered with the external slurmdbd to communicate with
each other using the \fI\-\-cluster/\-M\fR client command options.

The cluster will add itself to the external slurmdbd if it doesn't exist. If a
non\-external cluster already exists on the external slurmdbd, the slurmctld
will ignore registering to the external slurmdbd.
.IP

.TP
\fBAccountingStorageHost\fR
The name of the machine hosting the accounting storage database.
Only used with systems using SlurmDBD, ignored otherwise.
.IP

.TP
\fBAccountingStorageParameters\fR
Comma\-separated list of key\-value pair parameters. Currently
supported values include options to establish a secure connection to the
database:
.IP
.RS
.TP 2
\fBSSL_CERT\fR
The path name of the client public key certificate file.
The path name of the client private key file.
.IP

.TP
\fBSSL_CIPHER\fR
The list of permissible ciphers for SSL encryption.
.RE
.IP

.TP
\fBAccountingStoragePass\fR
The password used to gain access to the database to store the
accounting data.  Only used for database type storage plugins, ignored
otherwise.  In the case of Slurm DBD (Database Daemon) with MUNGE
authentication this can be configured to use a MUNGE daemon
specifically configured to provide authentication between clusters
while the default MUNGE daemon provides authentication within a
cluster.  In that case, \fBAccountingStoragePass\fR should specify the
named port to be used for communications with the alternate MUNGE
daemon (e.g.  "/var/run/munge/global.socket.2"). The default value is
NULL.
.IP

.TP
\fBAccountingStoragePort\fR
The listening port of the accounting storage database server.
Only used for database type storage plugins, ignored otherwise.
The default value is SLURMDBD_PORT as established at system
build time. If no value is explicitly specified, it will be set to 6819.
This value must be equal to the \fBDbdPort\fR parameter in the
slurmdbd.conf file.
.IP

.TP
\fBAccountingStorageTRES\fR
Comma\-separated list of resources you wish to track on the cluster.
These are the resources requested by the sbatch/srun job when it
is submitted. Currently this consists of any GRES, BB (burst buffer) or
license along with CPU, Memory, Node, Energy, FS/[Disk|Lustre], IC/OFED, Pages,
and VMem. By default Billing, CPU, Energy, Memory, Node, FS/Disk, Pages and VMem
are tracked. These default TRES cannot be disabled, but only appended to.
AccountingStorageTRES=gres/craynetwork,license/iop1
will track billing, cpu, energy, memory, nodes, fs/disk, pages and vmem along
with a gres called craynetwork as well as a license called iop1. Whenever these
resources are used on the cluster they are recorded. The TRES are automatically
set up in the database on the start of the slurmctld.

If multiple GRES of different types are tracked (e.g. GPUs of different types),
then job requests with matching type specifications will be recorded.
Given a configuration of
"AccountingStorageTRES=gres/gpu,gres/gpu:tesla,gres/gpu:volta"
Then "gres/gpu:tesla" and "gres/gpu:volta" will track only jobs that explicitly
\fBNOTE\fR: Setting gres/gpu will also set gres/gpumem and gres/gpuutil.
gres/gpumem and gres/gpuutil can be set individually when gres/gpu is not set.
.IP

.TP
\fBAccountingStorageType\fR
The accounting storage mechanism type.  Acceptable values at
present "accounting_storage/slurmdbd".
The "accounting_storage/slurmdbd" value indicates that accounting records
will be written to the Slurm DBD, which manages an underlying MySQL
database. See "man slurmdbd" for more information.  When this is not set it
indicates that account records are not maintained.
.IP

.TP
\fBAccountingStorageUser\fR
The user account for accessing the accounting storage database.
Only used for database type storage plugins, ignored otherwise.
.IP

.TP
\fBAccountingStoreFlags\fR
Comma separated list used to tell the slurmctld to store extra fields that may
be more heavy weight than the normal job information.
.IP
.RS
.TP
Current options are:
.IP

.TP
\fBjob_comment\fR
Include the job's comment field in the job complete message sent to the Accounting Storage database.
Note the AdminComment and SystemComment are always recorded in the database.
.IP

.TP
\fBjob_env\fR
Include a batch job's environment variables used at job submission in the job
start message sent to the Accounting Storage database.
.IP

.TP
\fBjob_extra\fR
Include the job's extra field in the job complete message sent to the Accounting
Storage database.
.IP

.TP
\fBjob_script\fR
Include the job's batch script in the job start message sent to the Accounting Storage database.
.RE

.TP
\fBAcctGatherEnergyType\fR
Identifies the plugin to be used for energy consumption accounting.
The jobacct_gather plugin and slurmd daemon call this plugin to collect
energy consumption data for jobs and nodes. The collection of energy
consumption data takes place on the node level, hence only in case of exclusive
job allocation the energy consumption measurements will reflect the job's
real consumption. In case of node sharing between jobs the reported consumed
energy per job (through sstat or sacct) will not reflect the real energy
consumed by the jobs. Default is nothing is collected.

Configurable values at present are:
.IP
.RS
.TP 20
\fBacct_gather_energy/ipmi\fR
Energy consumption data is collected from the Baseboard Management Controller
(BMC) using the Intelligent Platform Management Interface (IPMI).
.IP

.TP
\fBacct_gather_energy/pm_counters\fR
Energy consumption data is collected from the Baseboard Management
Controller (BMC) for HPE Cray systems.
.IP

.TP
\fBacct_gather_energy/rapl\fR
Energy consumption data is collected from hardware sensors using the Running
Average Power Limit (RAPL) mechanism. Note that enabling RAPL may require the
execution of the command "sudo modprobe msr".
.IP

.TP
\fBacct_gather_energy/xcc\fR
Energy consumption data is collected from the Lenovo SD650 XClarity Controller
(XCC) using IPMI OEM raw commands.
.RE
.IP

.TP
\fBAcctGatherInterconnectType\fR
Identifies the plugin to be used for interconnect network traffic accounting.
The jobacct_gather plugin and slurmd daemon call this plugin to collect
network traffic data for jobs and nodes.
The collection of network traffic data takes place on the node level,
hence only in case of exclusive job allocation the collected values will
reflect the job's real traffic. In case of node sharing between jobs the reported
network traffic per job (through sstat or sacct) will not reflect the real
network traffic by the jobs.

pseudo\-filesystem for specific interfaces defined in
\fBacct_gather.conf\fR(5).
In order to account for per job network traffic, add the "ic/sysfs" TRES to
\fIAccountingStorageTRES\fR.
.RE
.IP

.TP
\fBAcctGatherFilesystemType\fR
Identifies the plugin to be used for filesystem traffic accounting.
The jobacct_gather plugin and slurmd daemon call this plugin to collect
filesystem traffic data for jobs and nodes.
The collection of filesystem traffic data takes place on the node level,
hence only in case of exclusive job allocation the collected values will
reflect the job's real traffic. In case of node sharing between jobs the reported
filesystem traffic per job (through sstat or sacct) will not reflect the real
filesystem traffic by the jobs.


Configurable values at present are:
.IP
.RS
.TP 20
\fBacct_gather_filesystem/lustre\fR
Lustre filesystem traffic data are collected from the counters found in
/proc/fs/lustre/.
In order to account for per job lustre traffic, add the "fs/lustre" TRES to
\fIAccountingStorageTRES\fR.
.RE
.IP

.TP
\fBAcctGatherProfileType\fR
Identifies the plugin to be used for detailed job profiling.
The jobacct_gather plugin and slurmd daemon call this plugin to collect
detailed data such as I/O counts, memory usage, or energy consumption for jobs
and nodes. There are interfaces in this plugin to collect data as step start
and completion, task start and completion, and at the account gather
frequency. The data collected at the node level is related to jobs only in
case of exclusive job allocation.

Configurable values at present are:
.IP
.RS
.TP 20
\fBacct_gather_profile/hdf5\fR
This enables the HDF5 plugin. The directory where the profile files
are stored and which values are collected are configured in the
acct_gather.conf file.
.IP

.TP

.TP
\fBAuthAltTypes\fR
Comma\-separated list of alternative authentication plugins that the slurmctld
will permit for communication. Acceptable values at present include
\fIauth/jwt\fR.

\fBNOTE\fR: \fIauth/jwt\fR requires a jwt_hs256.key to be populated in the
\fBStateSaveLocation\fR directory for \fBslurmctld\fR only. The jwt_hs256.key
should only be visible to the SlurmUser and root. It is not suggested to place
the jwt_hs256.key on any nodes but the controller running \fBslurmctld\fR.
\fIauth/jwt\fR can be activated by the presence of the \fISLURM_JWT\fR
environment variable.  When activated, it will override the default
\fBAuthType\fR.
.IP

.TP
\fBAuthAltParameters\fR
Used to define alternative authentication plugins options. Multiple options may
be comma separated.
.IP
.RS
.TP 15
\fBdisable_token_creation\fR
Disable "scontrol token" use by non\-SlurmUser accounts.
.TP
\fBmax_token_lifespan\fR=<seconds>
Set max lifespan (in seconds) for any token generated for user accounts.
(This limit does not apply to SlurmUser.)
.IP

.TP
\fBjwks=\fR
Absolute path to JWKS file. Key should be owned by SlurmUser or root, must be
readable by SlurmUser, with suggested permissions of 0400. It must not be
writable by 'other'.
Only RS256 keys are supported, although other key types may be listed in the
file. If set, no HS256 key will be loaded by default (and token generation is
disabled), although the jwt_key setting may be used to explicitly re\-enable
HS256 key use (and token generation).
.IP

.TP
\fBjwt_key=\fR
Absolute path to JWT key file. Key must be HS256. Key should be owned by
SlurmUser or root, must be readable by SlurmUser, with suggested permissions of
0400. It must not be accessible by 'other'.
If not set, the default key file is jwt_hs256.key in \fIStateSaveLocation\fR.
.IP

.TP
\fBuserclaimfield=\fR
Multiple options may be specified in a comma\-delimited list.
If not specified, the default authentication information will be used.
.IP
.RS
.TP 14
\fBcred_expire\fR
Default job step credential lifetime, in seconds (e.g. "cred_expire=1200").
It must be sufficiently long enough to load user environment, run prolog,
deal with the slurmd getting paged out of memory, etc.
This also controls how long a requeued job must wait before starting again.
The default value is 120 seconds.
.IP

.TP
\fBsocket\fR
Path name to a MUNGE daemon socket to use
(e.g. "socket=/var/run/munge/munge.socket.2").
The default value is "/var/run/munge/munge.socket.2".
Used by \fIauth/munge\fR and \fIcred/munge\fR.
.IP

.TP
\fBttl\fR
Credential lifetime, in seconds (e.g. "ttl=300").
The default value is dependent upon the MUNGE installation, but is typically
300 seconds.
.RE
.IP

.TP
\fBAuthType\fR
The authentication method for communications between Slurm
components.
Acceptable values at present include "auth/munge", which is the default.
"auth/munge" indicates that MUNGE is to be used.
(See "https://dun.github.io/munge/" for more information).
All Slurm daemons and commands must be terminated prior to changing
the value of \fBAuthType\fR and later restarted.
.IP

.TP
\fBBackupAddr\fR
Deprecated option, see \fBSlurmctldHost\fR.
.IP

.TP
\fBBackupController\fR
Deprecated option, see \fBSlurmctldHost\fR.

The backup controller recovers state information from the
\fBStateSaveLocation\fR directory, which must be readable and writable from both
the primary and backup controllers.
the Slurm daemon on the compute node registers state with the slurmctld daemon
on the head node, which happens fairly rarely.
Therefore a job will not necessarily be terminated if its start time exceeds
\fBBatchStartTimeout\fR.
This configuration parameter is also applied to launch tasks and avoid aborting
\fBsrun\fR commands due to long running \fBProlog\fR scripts.
.IP

.TP
\fBBcastExclude\fR
Comma\-separated list of absolute directory paths to be excluded when
autodetecting and broadcasting executable shared object dependencies through
\fBsbcast\fR or \fBsrun \-\-bcast\fR. The keyword "\fInone\fR" can be used to
indicate that no directory paths should be excluded. The default value is
"\fI/lib,/usr/lib,/lib64,/usr/lib64\fR". This option can be overridden by
\fBsbcast \-\-exclude\fR and \fBsrun \-\-bcast\-exclude\fR.
.IP

.TP
\fBBcastParameters\fR
Controls sbcast and srun \-\-bcast behavior. Multiple options can be specified
in a comma separated list.
Supported values include:
.IP
.RS
.TP 15
\fBDestDir=\fR
Destination directory for file being broadcast to allocated compute nodes.
Default value is current working directory, or \-\-chdir for srun if set.
.IP

.TP
\fBCompression=\fR
Specify default file compression library to be used.
Supported values are "lz4" and "none".
The default value with the sbcast \-\-compress option is "lz4" and "none" otherwise.
Some compression libraries may be unavailable on some systems.
.IP

.TP
\fBsend_libs\fR
If set, attempt to autodetect and broadcast the executable's shared object
dependencies to allocated compute nodes. The files are placed in a directory
alongside the executable. For \fBsrun\fR only, the \fBLD_LIBRARY_PATH\fR is
automatically updated to include this cache directory as well.
This can be overridden with either \fBsbcast\fR or \fBsrun\fR
\fB\-\-send\-libs\fR option. By default this is disabled.
.RE
.IP

.TP
\fBBurstBufferType\fR

.TP
\fBburst_buffer/none\fR
.RE
.IP

.TP
\fBCliFilterPlugins\fR
A comma\-delimited list of command line interface option filter/modification
plugins. The specified plugins will be executed in the order listed.
No cli_filter plugins are used by default. Acceptable values at present are:
.IP
.RS
.TP
\fBcli_filter/lua\fR
This plugin allows you to write your own implementation of a cli_filter
using lua.
.IP

.TP
\fBcli_filter/syslog\fR
This plugin enables logging of job submission activities performed. All the
salloc/sbatch/srun options are logged to syslog together with environment
variables in JSON format. If the plugin is not the last one in the list it may
log values different than what was actually sent to slurmctld.
.IP

.TP
\fBcli_filter/user_defaults\fR
This plugin looks for the file $HOME/.slurm/defaults and reads every line of it
as a \fIkey\fR=\fIvalue\fR pair, where \fIkey\fR is any of the job submission
options available to salloc/sbatch/srun and \fIvalue\fR is a default value
defined by the user. For instance:
.nf
time=1:30
mem=2048
.fi
The above will result in a user defined default for each of their jobs of
"\-t 1:30" and "\-\-mem=2048".
.RE
.IP

.TP
\fBClusterName\fR
The name by which this Slurm managed cluster is known in the
accounting database.  This is needed distinguish accounting records
when multiple clusters report to the same database. Because of limitations
in some databases, any upper case letters in the name will be silently mapped
to lower case. In order to avoid confusion, it is recommended that the name
be lower case. The cluster name must be 40 characters or less in order to
comply with the limit on the maximum length for table names in MySQL/MariaDB.
.IP
\fBCheckGhalQuiesce\fR
Used specifically on a Cray using an Aries Ghal interconnect.  This will check
to see if the system is quiescing when sending a message, and if so, we wait
until it is done before sending.
.IP

.TP
\fBDisableIPv4\fR
Disable IPv4 only operation for all slurm daemons (except slurmdbd). This
should also be set in your \fBslurmdbd.conf\fR file.
.IP

.TP
\fBEnableIPv6\fR
Enable using IPv6 addresses for all slurm daemons (except slurmdbd). When
using both IPv4 and IPv6, address family preferences will be based on your
/etc/gai.conf file. This should also be set in your \fBslurmdbd.conf\fR file.
.IP

.TP
\fBgetnameinfo_cache_timeout\fR
When munge is used as AuthType slurmctld makes use of getnameinfo to obtain
the hostname from IP address stored in munge credential. This parameter controls
the number of seconds slurmctld should keep the IP to hostname resolution. When
set to 0 cache is disabled. The default value is 60.
.IP

.TP
\fBkeepaliveinterval=#\fR
Specifies the interval between keepalive probes on the socket communications
between srun and its slurmstepd process.
.IP

.TP
\fBkeepaliveprobes=#\fR
Specifies the number of keepalive probes sent on the socket communications
between srun command and its slurmstepd process before the connection is
considered broken.
.IP

.TP
\fBkeepalivetime=#\fR
Specifies how long sockets communications used between the srun command and its
slurmstepd process are kept alive after disconnect. Longer values can be used
to improve reliability of communications in the event of network failures.
.IP

.TP
\fBNoCtldInAddrAny\fR
Used to directly bind to the address of what the node resolves to running
the slurmctld instead of binding messages to any address on the node,
which is the default.
nodes that were recently in use, with the goal of preventing fragmentation.
If set to zero, pending jobs will be started as soon as possible.
Since a COMPLETING job's resources are released for use by other
jobs as soon as the \fBEpilog\fR completes on each individual node,
this can result in very fragmented resource allocations.
To provide jobs with the minimum response time, a value of zero is
recommended (no waiting).
To minimize fragmentation of resources, a value equal to \fBKillWait\fR
plus two is recommended.
In that case, setting \fBKillWait\fR to a small value may be beneficial.
The default value of \fBCompleteWait\fR is zero seconds.
The value may not exceed 65533.

\fBNOTE\fR: Setting \fBreduce_completing_frag\fR affects the behavior
of \fBCompleteWait\fR.
.IP

.TP
\fBControlAddr\fR
Deprecated option, see \fBSlurmctldHost\fR.
.IP

.TP
\fBControlMachine\fR
Deprecated option, see \fBSlurmctldHost\fR.
.IP

.TP
\fBCoreSpecPlugin\fR
Identifies the plugins to be used for enforcement of core specialization.
A restart of the slurmd daemons is required for changes to this parameter
to take effect.
Acceptable values at present include:
.IP
.RS
.TP 20
\fBcore_spec/cray_aries\fR
used only for Cray systems
.RE
.IP

.TP
\fBCpuFreqDef\fR
Default CPU governor to use when running a job step if it has not been
explicitly set with the \-\-cpu\-freq option. Acceptable values at present
include one of the following governors:
.IP
.RS
.TP 14
\fBConservative\fR
attempts to use the Conservative CPU governor
.IP
.TP
Default: Use system default. No attempt to set the governor is made if
\-\-cpu\-freq option has not been specified.
.RE
.IP

.TP
\fBCpuFreqGovernors\fR
List of CPU frequency governors allowed to be set with the salloc, sbatch, or
srun option \-\-cpu\-freq.
Acceptable values at present include:
.IP
.RS
.TP 14
\fBConservative\fR
attempts to use the Conservative CPU governor
.IP

.TP
\fBOnDemand\fR
attempts to use the OnDemand CPU governor (a default value)
.IP

.TP
\fBPerformance\fR
attempts to use the Performance CPU governor (a default value)
.IP

.TP
\fBPowerSave\fR
attempts to use the PowerSave CPU governor
.IP

.TP
\fBSchedUtil\fR
attempts to use the SchedUtil CPU governor
.IP

.TP
\fBUserSpace\fR
attempts to use the UserSpace CPU governor (a default value)
.TP
Default: OnDemand, Performance and UserSpace.
.RE
.IP

.TP
\fBCredType\fR
The cryptographic signature tool to be used in the creation of
job step credentials.
A restart of slurmctld is required for changes to this parameter to take effect.
The default (and recommended) value is "cred/munge".
Valid subsystems available include:
.IP
.RS
.TP 17
\fBAccrue\fR
Accrue counters accounting details
.IP

.TP
\fBAgent\fR
RPC agents (outgoing RPCs from Slurm daemons)
.IP

.TP
\fBBackfill\fR
Backfill scheduler details
.IP

.TP
\fBBackfillMap\fR
Backfill scheduler to log a very verbose map of reserved resources through
time. Combine with \fBBackfill\fR for a verbose and complete view of the
backfill scheduler's work.
.IP

.TP
\fBBurstBuffer\fR
Burst Buffer plugin
.IP

.TP
\fBCgroup\fR
Cgroup details
.IP

.TP
\fBCPU_Bind\fR
CPU binding details for jobs and steps
.IP

.TP
\fBCpuFrequency\fR
Cpu frequency details for jobs and steps using the \-\-cpu\-freq option.
.IP

.TP
\fBData\fR
Generic data structure details.
.IP

.TP
\fBDependency\fR
\fBFederation\fR
Federation scheduling debug info
.IP

.TP
\fBFrontEnd\fR
Front end node details
.IP

.TP
\fBGres\fR
Generic resource details
.IP

.TP
\fBHetjob\fR
Heterogeneous job details
.IP

.TP
\fBGang\fR
Gang scheduling details
.IP

.TP
\fBJobAccountGather\fR
Common job account gathering details (not plugin specific).
.IP

.TP
\fBJobComp\fR
Job Completion plugin details
.IP

.TP
\fBJobContainer\fR
Job container plugin details
.IP

.TP
\fBLicense\fR
License management details
.IP

.TP
\fBNetwork\fR
Network details. \fBWarning\fR: activating this flag may cause logging of
passwords, tokens or other authentication credentials.
.IP

.TP
\fBNetworkRaw\fR

.TP
\fBPower\fR
Power management plugin and power save (suspend/resume programs) details
.IP

.TP
\fBPriority\fR
Job prioritization
.IP

.TP
\fBProfile\fR
AcctGatherProfile plugins details
.IP

.TP
\fBProtocol\fR
Communication protocol details
.IP

.TP
\fBReservation\fR
Advanced reservations
.IP

.TP
\fBRoute\fR
Message forwarding debug info
.IP

.TP
\fBScript\fR
Debug info regarding the process that runs slurmctld scripts such as
PrologSlurmctld and EpilogSlurmctld
.IP

.TP
\fBSelectType\fR
Resource selection plugin
.IP

.TP
\fBSteps\fR
Slurmctld resource allocation for job steps
.IP

.TP
\fBSwitch\fR
Switch plugin
.IP

.IP

.TP
\fBWorkQueue\fR
Work Queue details
.RE
.IP

.TP
\fBDefCpuPerGPU\fR
Default count of CPUs allocated per allocated GPU. This value is used only if
the job didn't specify \-\-cpus\-per\-task and \-\-cpus\-per\-gpu.
.IP

.TP
\fBDefMemPerCPU\fR
Default real memory size available per usable allocated CPU in megabytes.
Used to avoid over\-subscribing memory and causing paging.
\fBDefMemPerCPU\fR would generally be used if individual processors
are allocated to jobs (\fBSelectType=select/cons_tres\fR).
The default value is 0 (unlimited).
Also see \fBDefMemPerGPU\fR, \fBDefMemPerNode\fR and \fBMaxMemPerCPU\fR.
\fBDefMemPerCPU\fR, \fBDefMemPerGPU\fR and \fBDefMemPerNode\fR are
mutually exclusive.


\fBNOTE\fR: This applies to \fBusable\fR allocated CPUs in a job allocation.
This is important when more than one thread per core is configured.
If a job requests \-\-threads\-per\-core with fewer threads on a core than
exist on the core (or \-\-hint=nomultithread which implies
\-\-threads\-per\-core=1), the job will be unable to use those extra threads on
the core and those threads will not be included in the memory per CPU
calculation. But if the job has access to all threads on the core, those threads
will be included in the memory per CPU calculation even if the job did not
explicitly request those threads.

In the following examples, each core has two threads.

In this first example, two tasks can run on separate hyperthreads
in the same core because \-\-threads\-per\-core is not used. The
third task uses both threads of the second core. The allocated
memory per cpu includes all threads:

.nf
.ft B
$ salloc \-n3 \-\-mem\-per\-cpu=100
salloc: Granted job allocation 17199
$ sacct \-j $SLURM_JOB_ID \-X \-o jobid%7,reqtres%35,alloctres%35
  JobID                             ReqTRES                           AllocTRES
\-\-\-\-\-\-\- \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
  17199     billing=3,cpu=3,mem=300M,node=1     billing=4,cpu=4,mem=400M,node=1
.ft
\-\-\-\-\-\-\- \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
  17200     billing=3,cpu=3,mem=300M,node=1     billing=6,cpu=6,mem=300M,node=1
.ft
.fi
.IP

.TP
\fBDefMemPerGPU\fR
Default real memory size available per allocated GPU in megabytes.
The default value is 0 (unlimited).
Also see \fBDefMemPerCPU\fR and \fBDefMemPerNode\fR.
\fBDefMemPerCPU\fR, \fBDefMemPerGPU\fR and \fBDefMemPerNode\fR are
mutually exclusive.
.IP

.TP
\fBDefMemPerNode\fR
Default real memory size available per allocated node in megabytes.
Used to avoid over\-subscribing memory and causing paging.
\fBDefMemPerNode\fR would generally be used if whole nodes
are allocated to jobs (\fBSelectType=select/linear\fR) and
resources are over\-subscribed (\fBOverSubscribe=yes\fR or
\fBOverSubscribe=force\fR).
The default value is 0 (unlimited).
Also see \fBDefMemPerCPU\fR, \fBDefMemPerGPU\fR and \fBMaxMemPerCPU\fR.
\fBDefMemPerCPU\fR, \fBDefMemPerGPU\fR and \fBDefMemPerNode\fR are
mutually exclusive.
.IP

.TP
\fBDependencyParameters\fR
Multiple options may be comma separated.
.IP
.RS
.TP
\fBdisable_remote_singleton\fR
By default, when a federated job has a singleton dependency, each cluster in the
federation must clear the singleton dependency before the job's singleton
dependency is considered satisfied. Enabling this option means that only the
origin cluster must clear the singleton dependency. This option must be set
in every cluster in the federation.
.IP

.TP
\fBkill_invalid_depend\fR
If a job has an invalid dependency and it can never run terminate it
and set its state to be JOB_CANCELLED. By default the job stays pending
with reason DependencyNeverSatisfied.
.IP

.TP
\fBmax_depend_depth=#\fR
\fBEioTimeout\fR
The number of seconds srun waits for slurmstepd to close the TCP/IP
connection used to relay data between the user application and srun
when the user application terminates. The default value is 60 seconds.
May not exceed 65533.
.IP

.TP
\fBEnforcePartLimits\fR
If set to "ALL" then jobs which exceed a partition's size and/or
time limits will be rejected at submission time. If job is submitted to
multiple partitions, the job must satisfy the limits on all the requested
partitions. If set to "NO" then the job will be accepted and remain queued
until the partition limits are altered(Time and Node Limits).
If set to "ANY" a job must satisfy any of the requested partitions
to be submitted. The default value is "NO".
\fBNOTE\fR: If set, then a job's QOS can not be used to exceed partition limits.
\fBNOTE\fR: The partition limits being considered are its configured
MaxMemPerCPU, MaxMemPerNode, MinNodes, MaxNodes, MaxTime, AllocNodes,
AllowAccounts, AllowGroups, AllowQOS, and QOS usage threshold.
.IP

.TP
\fBEpilog\fR
Pathname of a script to execute as user root on every node when a user's job
completes (e.g. "/usr/local/slurm/epilog"). If it is not an absolute path name
(i.e. it does not start with a slash), it will be searched for in the same
directory as the slurm.conf file. A glob pattern (See \fBglob\fR (7)) may also
be used to run more than one epilog script (e.g. "/etc/slurm/epilog.d/*").
When more than one epilog script is configured, they are executed in reverse
order. The Epilog script or scripts may be used to purge files, disable user
login, etc.
By default there is no epilog.
See \fBProlog and Epilog Scripts\fR for more information.
.IP

.TP
\fBEpilogMsgTime\fR
The number of microseconds that the slurmctld daemon requires to process
an epilog completion message from the slurmd daemons. This parameter can
be used to prevent a burst of epilog completion messages from being sent
at the same time which should help prevent lost messages and improve
throughput for large jobs.
The default value is 2000 microseconds.
For a 1000 node job, this spreads the epilog completion messages out over
two seconds.
.IP

.TP
\fBEpilogSlurmctld\fR
Fully qualified pathname of a program for the slurmctld to execute
upon termination of a job allocation (e.g.
between exceeding the fair share at high levels (e.g. a value of 1 results
in almost no difference between overconsumption by a factor of 10 and 100,
while a value of 5 will result in a significant difference in priority).
The default value is 1.
.IP

.TP
\fBFederationParameters\fR
Used to define federation options. Multiple options may be comma separated.
.IP
.RS
.TP
\fBfed_display\fR
If set, then the client status commands (e.g. squeue, sinfo, sprio, etc.) will
display information in a federated view by default. This option is functionally
equivalent to using the \-\-federation options on each command. Use the client's
\-\-local option to override the federated view and get a local view of the
given cluster.
.RE
.IP

.TP
\fBFirstJobId\fR
The job id to be used for the first job submitted to Slurm.
Job id values generated will incremented by 1 for each subsequent job.
Value must be larger than 0. The default value is 1.
Also see \fBMaxJobId\fR
.IP

.TP
\fBGetEnvTimeout\fR
Controls how long the job should wait (in seconds) to load the user's
environment before attempting to load it from a cache file.
Applies when the salloc or sbatch \fI\-\-get\-user\-env\fR option is used.
If set to 0 then always load the user's environment from the cache file.
The default value is 2 seconds.
.IP

.TP
\fBGresTypes\fR
A comma\-delimited list of generic resources to be managed (e.g.
\fIGresTypes=gpu,mps\fR).
These resources may have an associated GRES plugin of the same name providing
additional functionality.
No generic resources are managed by default.
Ensure this parameter is consistent across all nodes in the cluster for
proper operation.
A restart of slurmctld and the slurmd daemons is required for this to take
effect.
.IP

.TP
group membership lists will be cached.
The time interval is given in seconds with a default value of 600 seconds.
A value of zero will prevent periodic updating of group membership information.
Also see the \fBGroupUpdateForce\fR parameter.
.IP

.TP
\fBGpuFreqDef\fR=[<\fItype\fR]=\fIvalue\fR>[,<\fItype\fR=\fIvalue\fR>]
Default GPU frequency to use when running a job step if it
has not been explicitly set using the \-\-gpu\-freq option.
This option can be used to independently configure the GPU and its memory
frequencies.
There is no default value. If unset, no attempt to change the GPU frequency
is made if the \-\-gpu\-freq option has not been set.
After the job is completed, the frequencies of all affected GPUs will be reset
to the highest possible values.
In some cases, system power caps may override the requested values.
The field \fItype\fR can be "memory".
If \fItype\fR is not specified, the GPU frequency is implied.
The \fIvalue\fR field can either be "low", "medium", "high", "highm1" or
a numeric value in megahertz (MHz).
If the specified numeric value is not possible, a value as close as
possible will be used.
See below for definition of the values.
Examples of use include "GpuFreqDef=medium,memory=high and "GpuFreqDef=450".

Supported \fIvalue\fR definitions:
.IP
.RS
.TP 10
\fBlow\fR
the lowest available frequency.
.IP

.TP
\fBmedium\fR
attempts to set a frequency in the middle of the available range.
.IP

.TP
\fBhigh\fR
the highest available frequency.
.IP

.TP
\fBhighm1\fR
(high minus one) will select the next highest available frequency.
.RE
.IP

.TP
\fBHealthCheckInterval\fR
.IP

.TP
\fBANY\fR
Run on nodes in any state.
.IP

.TP
\fBCYCLE\fR
Rather than running the health check program on all nodes at the same time,
cycle through running on all compute nodes through the course of the
\fBHealthCheckInterval\fR. May be combined with the various node state
options.
.IP

.TP
\fBIDLE\fR
Run on nodes in the IDLE state.
.IP

.TP
\fBNONDRAINED_IDLE\fR
Run on nodes that are in the IDLE state and not DRAINED.
.IP

.TP
\fBMIXED\fR
Run on nodes in the MIXED state (some CPUs idle and other CPUs allocated).
.RE
.IP

.TP
\fBHealthCheckProgram\fR
Fully qualified pathname of a script to execute as user root periodically
on all compute nodes that are \fBnot\fR in the NOT_RESPONDING state. This
program may be used to verify the node is fully operational and DRAIN the node
or send email if a problem is detected.
Any action to be taken must be explicitly performed by the program
(e.g. execute
"scontrol update NodeName=foo State=drain Reason=tmp_file_system_full"
to drain a node).
The execution interval is controlled using the \fBHealthCheckInterval\fR
parameter.
Note that the \fBHealthCheckProgram\fR will be executed at the same time
on all nodes to minimize its impact upon parallel programs.
This program will be killed if it does not terminate normally within
60 seconds.
This program will also be executed when the slurmd daemon is first started and
before it registers with the slurmctld daemon.
By default, no program will be executed.
.IP

.IP

.TP
\fBInteractiveStepOptions\fR
When LaunchParameters=use_interactive_step is enabled, launching salloc will
automatically start an srun process with InteractiveStepOptions to launch
a terminal on a node in the job allocation.
The default value is "\-\-interactive \-\-preserve\-env \-\-pty $SHELL".
The "\-\-interactive" option is intentionally not documented in the srun man
page. It is meant only to be used in \fBInteractiveStepOptions\fR in order to
create an "interactive step" that will not consume resources so that other
steps may run in parallel with the interactive step.
.IP

.TP
\fBJobAcctGatherType\fR
The JobAcctGather plugin collects memory, cpu, io, interconnect, energy and gpu
usage information at the task level, depending on which plugins are configured
in Slurm. This parameter will control how some of these metrics will be
collected.

Configurable values at present are:
.IP
.RS
.TP 20
\fBjobacct_gather/cgroup\fR (recommended)
Collect cpu and memory statistics by reading the task's cgroup directory
interfaces (e.g. memory.stat, cpu.stat) by issuing a call to the configured
CgroupPlugin (see "man cgroup.conf").
This mechanism ignores JobAcctGatherParams=UsePSS or NoShared since these are
used only when reading memory usage from the proc filesystem.
.IP

.TP
\fBjobacct_gather/linux\fR
Collect cpu and memory statistics by reading procfs. The plugin will take all
the pids of the task and for each of them will read /proc/<pid>/stats. If UsePSS
is set it will also read /proc/<pid>/smaps, and if NoShare is set it will also
read /proc/<pid>/statm (see \fBJobAcctGatherParams\fR for more information).

This plugin carries a performance penalty on jobs with a large number of spawned
processes since it needs to iterate over all the task pids and aggregate the
stats into one single metric for the ppid, and then these values need to be
aggregated to the task stats.
.IP

.TP
\fBjobacct_gather/none\fR
This is the default value. No accounting data is collected. \fBsstat\fR will
not work.
.RE
.IP
where \fI<datatype>\fR=\fI<interval>\fR specifies the task sampling
interval for the jobacct_gather plugin or a
sampling interval for a profiling type by the
acct_gather_profile plugin. Multiple,
comma\-separated \fI<datatype>\fR=\fI<interval>\fR intervals
may be specified. Supported datatypes are as follows:
.IP
.RS
.TP
\fBtask=\fI<interval>\fR
where \fI<interval>\fR is the task sampling interval in seconds
for the jobacct_gather plugins and for task
profiling by the acct_gather_profile plugin.
.IP

.TP
\fBenergy=\fI<interval>\fR
where \fI<interval>\fR is the sampling interval in seconds
for energy profiling using the acct_gather_energy plugin
.IP

.TP
\fBnetwork=\fI<interval>\fR
where \fI<interval>\fR is the sampling interval in seconds
for infiniband profiling using the acct_gather_interconnect
plugin.
.IP

.TP
\fBfilesystem=\fI<interval>\fR
where \fI<interval>\fR is the sampling interval in seconds
for filesystem profiling using the acct_gather_filesystem
plugin.

.RE
The default value for task sampling interval
is 30 seconds. The default value for all other intervals is 0.
An interval of 0 disables sampling of the specified type.
If the task sampling interval is 0, accounting
information is collected only at job termination, which reduces Slurm
interference with the job, but also means that the statistics about a job
don't reflect the average or maximum of several samples throughout the life
of the job, but just show the information collected in the single sample.
.br
.br
Smaller (non\-zero) values have a greater impact upon job performance,
but a value of 30 seconds is not likely to be noticeable for
applications having less than 10,000 tasks.
.br
.br
Users can independently override each interval on a per job basis using the
\fB\-\-acctg\-freq\fR option when submitting the job.
.TP
\fBUsePss\fR
Use PSS value instead of RSS to calculate real usage of memory. The PSS value
will be saved as RSS. This option cannot be used with NoShared.
.IP

.TP
\fBOverMemoryKill\fR
Kill processes that are being detected to use more memory than requested by
steps every time accounting information is gathered by the JobAcctGather plugin.
This parameter should be used with caution because a job exceeding its memory
allocation may affect other processes and/or machine health.

\fBNOTE\fR: If available, it is recommended to limit memory by enabling
task/cgroup as a TaskPlugin and making use of ConstrainRAMSpace=yes in the
cgroup.conf instead of using this JobAcctGather mechanism for memory
enforcement. Using JobAcctGather is polling based and there is a
delay before a job is killed, which could lead to system Out of Memory events.

\fBNOTE\fR: When using \fBOverMemoryKill\fR, if the combined memory used by
all the processes in a step exceeds the memory limit, the entire step will be
killed/cancelled by the JobAcctGather plugin.
This differs from the behavior when using \fBConstrainRAMSpace\fR, where
processes in the step will be killed, but the step will be left active,
possibly with other processes left running.
.IP

.TP
\fBDisableGPUAcct\fR
Do not do accounting of GPU usage and skip any gpu driver library call. This
parameter can help to improve performance if the GPU driver response is slow.
.RE
.IP

.TP
\fBJobCompHost\fR
The name of the machine hosting the job completion database.
Only used for database type storage plugins, ignored otherwise.
.IP

.TP
\fBJobCompLoc\fR
This option sets a string which has different meanings depending on
\fBJobCompType\fR:
.IP
.RS
.TP
If \fBjobcomp/elasticsearch\fR:
Instructs this plugin to send the finished job records information to the
Elasticsearch server URL endpoint (including the port number and the target
index) configured in this option. This string should typically take the form
of \fI<host>:<port>/<target>/_doc\fR. There is no default value for
If \fBjobcomp/kafka\fR:
When this plugin is configured, finished job records information is sent to a
Kafka server. The plugin makes use of \fBlibrdkafka\fR. This string represents
an absolute path to a file containing 'key=value' pairs configuring the library
behavior. For the plugin to work properly, this file needs to exist and least
the \fIbootstrap.servers\fR \fBlibrdkafka\fR property needs to be configured
in it. There is no default value for JobCompLoc when this plugin is enabled.

\fBNOTE\fR: For a full list of \fBlibrdkafka\fR properties, please refer to
the library documentation. You can also view the jobcomp_kafka page for more
information: <https://slurm.schedmd.com/jobcomp_kafka.html>

\fBNOTE\fR: The target Kafka topic and other plugin parameters can be configured
via \fBJobCompParams\fR.

\fBNOTE\fR: The \fBlibrdkafka\fR parameters configured in the file referenced by
this option take effect upon slurmctld restart.
.IP

.TP
If \fBjobcomp/lua\fR:
This option is ignored in this plugin. The finished job record is processed
by a hardcoded \fIjobcomp.lua\fR script expected to be located in the same
location of slurm.conf. There is no default value for JobCompLoc when this
plugin is enabled.
.IP

.TP
If \fBjobcomp/mysql\fR:
Instructs this plugin to send the finished job records information to a database
name configured in this option. This string should represent a database name.
The default value for this plugin is \fIslurm_jobcomp_db\fR.
.IP

.TP
If \fBjobcomp/script\fR:
The finished job record information is made available via environment variables
and processed by a script with name configured by this option. This string
should represent a path to a script. There is no default value for JobCompLoc
when this plugin is enabled. It needs to be explicitly configured or the
plugin will fail to initialize.
.RE
.IP

.TP
\fBJobCompParams\fR
Pass arbitrary text string to job completion plugin.
Also see \fBJobCompType\fR.
.RS
.IP

.TP
Accepted values are [-1,2147483647].
Defaults to 500 (milliseconds).
.IP

.TP
\fBpoll_interval\fR=<seconds>
Seconds between calls to \fBlibrdkafka\fR API poll function, which polls the
provided Kafka handle for events. The plugin spawns a separate thread to perform
this call at the configured interval.
Accepted values are [0,4294967295].
Defaults to 2 (seconds).
.IP

.TP
\fBrequeue_on_msg_timeout\fR
Instruct the delivery report callback to requeue messages that failed delivery
because their time waiting for successful delivery reached the \fBlibrdkafka\fR
property \fBmessage.timeout.ms\fR.
Defaults to not set (don't requeue and thus discard these messages).
.IP

.TP
\fBtopic\fR=<string>
Target Kafka topic to send messages to.
Defaults to \fBClusterName\fR.
.RE
.IP

.RE
.IP

.TP
\fBJobCompPass\fR
The password used to gain access to the database to store the job
completion data.
Only used for database type storage plugins, ignored otherwise.
.IP

.TP
\fBJobCompPort\fR
The listening port of the job completion database server.
Only used for database type storage plugins, ignored otherwise.
.IP

.TP
\fBJobCompType\fR
The job completion logging mechanism type.
Acceptable values at present include:
.IP
.RS
.TP
\fBjobcomp/none\fR
.TP
\fBjobcomp/filetxt\fR
Upon job completion, a record of the job should be written to a text file,
specified by the \fBJobCompLoc\fR parameter.
.IP

.TP
\fBjobcomp/kafka\fR
Upon job completion, a record of the job should be sent to a Kafka server,
specified by the file path referenced in \fBJobCompLoc\fR and/or using other
\fBJobCompParams\fR.
.IP

.TP
\fBjobcomp/lua\fR
Upon job completion, a record of the job should be processed by the
\fIjobcomp.lua\fR script, located in the default script directory
(typically the subdirectory \fIetc\fR of the installation directory.
.IP

.TP
\fBjobcomp/mysql\fR
Upon job completion, a record of the job should be written to a MySQL
or MariaDB database, specified by the \fBJobCompLoc\fR parameter.
.IP

.TP
\fBjobcomp/script\fR
Upon job completion, a script specified by the \fBJobCompLoc\fR parameter is
to be executed with environment variables providing the job information.
.RE
.IP

.TP
\fBJobCompUser\fR
The user account for accessing the job completion database.
Only used for database type storage plugins, ignored otherwise.
.IP

.TP
\fBJobContainerType\fR
Identifies the plugin to be used for job tracking.
A restart of slurmctld is required for changes to this parameter to take effect.
\fBNOTE\fR: The \fBJobContainerType\fR applies to a job allocation, while
\fBProctrackType\fR applies to job steps.
Acceptable values at present include:
.IP
.RS
.TP 20
\fBjob_container/cncu\fR
Used only for Cray systems (CNCU = Compute Node Clean Up)
.IP
the existing file.
By default, any existing file is truncated.
.IP

.TP
\fBJobRequeue\fR
This option controls the default ability for batch jobs to be requeued.
Jobs may be requeued explicitly by a system administrator, after node
failure, or upon preemption by a higher priority job.
If \fBJobRequeue\fR is set to a value of 1, then batch jobs may be requeued
unless explicitly disabled by the user.
If \fBJobRequeue\fR is set to a value of 0, then batch jobs will not be requeued
unless explicitly enabled by the user.
Use the \fBsbatch\fR \fI\-\-no\-requeue\fR or \fI\-\-requeue\fR
option to change the default behavior for individual jobs.
The default value is 1.
.IP

.TP
\fBJobSubmitPlugins\fR
These are intended to be site\-specific plugins which can be used to set
default job parameters and/or logging events. Slurm can be configured to use
multiple job_submit plugins if desired, which must be specified as a
comma\-delimited list and will be executed in the order listed.
.nf
e.g. for multiple job_submit plugin configuration:
JobSubmitPlugins=lua,require_timelimit
.fi
Take a look at <https://slurm.schedmd.com/job_submit_plugins.html> for further
plugin implementation details. No job submission plugins are used by default.
Currently available plugins are:
.IP
.RS
.TP 24
\fBall_partitions\fR
Set default partition to all partitions on the cluster.
.IP

.TP
\fBdefaults\fR
Set default values for job submission or modify requests.
.IP

.TP
\fBlogging\fR
Log select job submission and modification parameters.
.IP

.TP
\fBlua\fR
Execute a Lua script implementing site's own job_submit logic. Only one Lua
script will be executed. It must be named "job_submit.lua" and must be located
Set a job's default partition based upon job submission parameters and
available partitions.
.IP

.TP
\fBpbs\fR
Translate PBS job submission options to Slurm equivalent (if possible).
.IP

.TP
\fBrequire_timelimit\fR
Force job submissions to specify a timelimit.
.RE
.IP

\fBNOTE\fR: For examples of use see the Slurm code in "src/plugins/job_submit"
and "contribs/lua/job_submit*.lua" then modify the code to satisfy your needs.
.IP

.TP
\fBKillOnBadExit\fR
If set to 1, a step will be terminated immediately if any task is
crashed or aborted, as indicated by a non\-zero exit code.
With the default value of 0, if one of the processes is crashed or aborted
the other processes will continue to run while the crashed or aborted process
waits. The user can override this configuration parameter by using srun's
\fB\-K\fR, \fB\-\-kill\-on\-bad\-exit\fR.
.IP

.TP
\fBKillWait\fR
The interval, in seconds, given to a job's processes between the
SIGTERM and SIGKILL signals upon reaching its time limit.
If the job fails to terminate gracefully in the interval specified,
it will be forcibly terminated.
The default value is 30 seconds.
The value may not exceed 65533.
.IP

.TP
\fBMaxBatchRequeue\fR
Maximum number of times a batch job may be automatically requeued before
being marked as JobHeldAdmin. (Mainly useful when the \fBSchedulerParameters\fR
option \fBnohold_on_prolog_fail\fR is enabled.)
The default value is 5.
.IP

.TP
\fBNodeFeaturesPlugins\fR
Identifies the plugins to be used for support of node features which can
change through time. For example, a node which might be booted with various
BIOS setting. This is supported through the use of a node's active_features
.IP

.TP
\fBnode_features/helpers\fR
Used to report and modify features on nodes using arbitrary scripts or
programs.
See helpers.conf man page for more information:
https://slurm.schedmd.com/helpers.conf.html
.RE
.IP

.TP
\fBLaunchParameters\fR
Identifies options to the job launch plugin.
Acceptable values include:
.IP
.RS
.TP 24
\fBbatch_step_set_cpu_freq\fR
Set the cpu frequency for the batch step from given \-\-cpu\-freq, or
slurm.conf CpuFreqDef, option.  By default only steps started with srun will
utilize the cpu freq setting options.

\fBNOTE\fR: If you are using srun to launch your steps inside a batch script
(advised) this option will create a situation where you may have multiple
agents setting the cpu_freq as the batch step usually runs on the same
resources one or more steps the sruns in the script will create.
.IP

.TP 24
\fBcray_net_exclusive\fR
Allow jobs on a Cray XC cluster exclusive access to network resources.
This should only be set on clusters providing exclusive access to each
node to a single job at once, and not using parallel steps within the job,
otherwise resources on the node can be oversubscribed.
.IP

.TP 24
\fBenable_nss_slurm\fR
Permits passwd and group resolution for a job to be serviced by slurmstepd rather
than requiring a lookup from a network based service. See
https://slurm.schedmd.com/nss_slurm.html for more information.
.IP

.TP 24
\fBlustre_no_flush\fR
If set on a Cray XC cluster, then do not flush the Lustre cache on job step
completion. This setting will only take effect after reconfiguring, and will
only take effect for newly launched jobs.
.IP

.TP 24
.TP
\fBdisable_send_gids\fR
By default, the slurmctld will look up and send the user_name and extended gids
for a job, rather than independently on each node as part of each task launch.
This helps mitigate issues around name service scalability when launching jobs
involving many nodes. Using this option will disable this functionality. This
option is ignored if enable_nss_slurm is specified.
.IP

.TP 24
\fBslurmstepd_memlock\fR
Lock the slurmstepd process's current memory in RAM.
.IP

.TP
\fBslurmstepd_memlock_all\fR
Lock the slurmstepd process's current and future memory in RAM.
.IP

.TP
\fBtest_exec\fR
Have srun verify existence of the executable program along with user
execute permission on the node where srun was called before attempting to
launch it on nodes in the step.
.IP

.TP
\fBuse_interactive_step\fR
Have salloc use the Interactive Step to launch a shell on an allocated compute
node rather than locally to wherever salloc was invoked. This is accomplished
by launching the srun command with InteractiveStepOptions as options.

This does not affect salloc called with a command as an argument. These jobs
will continue to be executed as the calling user on the calling host.
.IP

.TP
\fBulimit_pam_adopt\fR
When pam_slurm_adopt is used to join an external process into a job cgroup,
RLIMIT_RSS is set, as is done for tasks running in regular steps.
.RE
.IP

.TP
\fBLicenses\fR
Specification of licenses (or other resources available on all
nodes of the cluster) which can be allocated to jobs.
License names can optionally be followed by a colon
and count with a default count of one.
Multiple license names should be comma separated (e.g.
"Licenses=foo:4,bar").
Note that Slurm prevents jobs from being scheduled if their
with the C standard clock() function. The "short" format is a short
date and time format. The "thread_id" format shows the timestamp
in the C standard ctime() function form without the year but
including the microseconds, the daemon's process ID and the current thread name
and ID.
A special option "format_stderr" can be added to the format as a comma separated
value (e.g. "LogTimeFormat=iso8601_ms,format_stderr"). It will change the
default format of the logs on stderr stream by prepending the timestamp as
specified by \fBLogTimeFormat\fR.
.IP

.TP
\fBMailDomain\fR
Domain name to qualify usernames if email address is not explicitly given
with the "\-\-mail\-user" option. If unset, the local MTA will need to qualify
local address itself. Changes to MailDomain will only affect new jobs.
.IP

.TP
\fBMailProg\fR
Fully qualified pathname to the program used to send email per user request.
The default value is "/bin/mail" (or "/usr/bin/mail" if "/bin/mail" does not
exist but "/usr/bin/mail" does exist).
The program is called with arguments suitable for the default mail command,
however additional information about the job is passed in the form of
environment variables.

Additional variables are the same as those passed to \fIPrologSlurmctld\fR and
\fIEpilogSlurmctld\fR with additional variables in the following contexts:
.IP
.RS
.TP
\fBALL\fR
.IP
.RS
.TP
\fBSLURM_JOB_STATE\fR
The base state of the job when the MailProg is called.
.RE
.IP
.RS
.TP
\fBSLURM_JOB_MAIL_TYPE\fR
The mail type triggering the mail.
.RE
.RE
.IP
.RS
.TP
\fBBEGIN\fR
.IP
.RS
.RE
.RE
.IP
.RS
.TP
\fBEND, FAIL\fR
.IP
.RS
.TP
\fBSLURM_JOB_EXIT_CODE_MAX\fR
Job's exit code or highest exit code for an array job.
.RE
.IP
.RS
.TP
\fBSLURM_JOB_EXIT_CODE_MIN\fR
Job's minimum exit code for an array job.
.RE
.IP
.RS
.TP
\fBSLURM_JOB_TERM_SIGNAL_MAX\fR
Job's highest signal for an array job.
.RE
.RE
.IP
.RS
.TP
\fBSTAGE_OUT\fR
.IP
.RS
.TP
\fBSLURM_JOB_STAGE_OUT_TIME\fR
Job's staging out time.
.RE
.RE
.IP

.TP
\fBMaxArraySize\fR
The maximum job array task index value will be one less than MaxArraySize
to allow for an index value of zero.
Configure MaxArraySize to 0 in order to disable job array use.
The value may not exceed 4000001.
The value of \fBMaxJobCount\fR should be much larger than \fBMaxArraySize\fR.
The default value is 1001.
See also \fBmax_array_tasks\fR in SchedulerParameters.
.IP

.TP
\fBMaxDBDMsgs\fR
When communication to the SlurmDBD is not possible the slurmctld will queue
Performance can suffer with more than a few hundred thousand jobs.
Setting per MaxSubmitJobs per user is generally valuable to prevent a single
user from filling the system with jobs.
This is accomplished using Slurm's database and configuring enforcement of
resource limits.
A restart of slurmctld is required for changes to this parameter to take effect.
.IP

.TP
\fBMaxJobId\fR
The maximum job id to be used for jobs submitted to Slurm without a specific
requested value. Job ids are unsigned 32bit integers with the first 26 bits
reserved for local job ids and the remaining 6 bits reserved for a cluster id
to identify a federated job's origin. The maximum allowed local job id is
67,108,863 (0x3FFFFFF). The default value is 67,043,328 (0x03ff0000).
\fBMaxJobId\fR only applies to the local job id and not the federated job id.
Job id values generated will be incremented by 1 for each subsequent job. Once
\fBMaxJobId\fR is reached, the next job will be assigned \fBFirstJobId\fR.
Federated jobs will always have a job ID of 67,108,865 or higher.
Also see \fBFirstJobId\fR.
.IP

.TP
\fBMaxMemPerCPU\fR
Maximum real memory size available per allocated CPU in megabytes.
Used to avoid over\-subscribing memory and causing paging.
\fBMaxMemPerCPU\fR would generally be used if individual processors
are allocated to jobs (\fBSelectType=select/cons_tres\fR).
The default value is 0 (unlimited).
Also see \fBDefMemPerCPU\fR, \fBDefMemPerGPU\fR and \fBMaxMemPerNode\fR.
\fBMaxMemPerCPU\fR and \fBMaxMemPerNode\fR are mutually exclusive.

\fBNOTE\fR: If a job specifies a memory per CPU limit that exceeds this system
limit, that job's count of CPUs per task will try to automatically increase.
This may result in the job failing due to CPU count limits. This
auto\-adjustment feature is a best\-effort one and optimal assignment is not
guaranteed due to the possibility of having heterogeneous configurations and
multi\-partition/qos jobs. If this is a concern it is advised to use a job
submit LUA plugin instead to enforce auto\-adjustments to your specific needs.
.IP

.TP
\fBMaxMemPerNode\fR
Maximum real memory size available per allocated node in megabytes.
Used to avoid over\-subscribing memory and causing paging.
\fBMaxMemPerNode\fR would generally be used if whole nodes
are allocated to jobs (\fBSelectType=select/linear\fR) and
resources are over\-subscribed (\fBOverSubscribe=yes\fR or
\fBOverSubscribe=force\fR).
The default value is 0 (unlimited).
Also see \fBDefMemPerNode\fR and \fBMaxMemPerCPU\fR.
\fBMaxMemPerCPU\fR and \fBMaxMemPerNode\fR are mutually exclusive.
The maximum number of steps that any job can initiate. This parameter
is intended to limit the effect of bad batch scripts.
The default value is 40000 steps.
.IP

.TP
\fBMaxTasksPerNode\fR
Maximum number of tasks Slurm will allow a job step to spawn
on a single node. The default \fBMaxTasksPerNode\fR is 512.
May not exceed 65533.
.IP

.TP
\fBMCSParameters\fR
MCS = Multi\-Category Security
MCS Plugin Parameters.
The supported parameters are specific to the \fBMCSPlugin\fR.
Changes to this value take effect when the Slurm daemons are reconfigured.
More information about MCS is available here