Allocating resources and placing jobs

Resource types and chunk

PBS resources represent things such as CPUs (cores), memory, switches, hosts, Operating System, chassis, time.... They can also represent whether or not something is true or not, for example whether a node is dedicated to a particular project or group.

A chunk is a set of resources that are allocated as a unit to a job. All parts of a chunk come from the same host/node.

A chunk-level resource or a host-level resource is a resource available at host level. The resources of a chunk are to be applied to the portion of the job running in that chunk. Chunk resources are requested inside a select statement.

Chunk level resources that can be requested are: ncpus, mpiprocs, ompthreads, mem, chassis, host,...
Check chunk resources available with the pbsnodes command:
pbsnodes <node_name>

Job-wide resource, also called queue-level or server-level resource, is a resource that is available to the entire job at the server or the queue.
Job-wide resources that can be requested are: walltime, pmem, place

Format for requesting job-wide resources:
qsub … (non-resource portion of the job) -l resource=value

Format for requesting chunk resources:
qsub … (non-resource portion of the job) -l select=[N:][chunk specification][+[N:]...]
where N is the number of identical chunks, + adds a new set of identical chunks.

PBS assigns chunks to job processes in the order in which the chunks appear in the select statement.

Memory resource

The memory lower bound is 256MB. Jobs requesting less than 256MB per chunk will be put on hold (system) by the PBS enforcement tools with the message  CgroupLimitError.

Placement

Users can specify how the job should be placed on nodes. Users can choose to place each chunk on a different host, or to share a specific value for some resource.

The place statement must be used to specify how the job's chunk are placed. The place statement has the form:
qsub … -l place=[arrangement][:sharing][:grouping]
where

• arrangement is one of free | pack | scatter| vscatter
• sharing is excl | shared | exclhost
• grouping can have only one instance group=resource

Examples:

Academic users (and others) can find a useful script generation wizard on the CÉCI web site.

Below are some other specific examples:

1. You want five identical chunks requesting twelve CPUS and 43000MB of memory each, placed on different nodes:
   qsub … -l select=5:ncpus=12:mem=43000mb -l place=scatter
2. You want four chunks, where the first has two CPUs and 20GB of memory, the second has 4 CPUs and 4GB of memory and the two last ones, one CPU and 40GB of memory with a free placement:
   qsub … -l select=1:ncpus=2:mem=20GB+1:ncpus=4:mem=4GB+2:ncpus=1:mem=40GB …

Some resources or placement can be requested by users, other ones are read-only and cannot be modified by users (queues limits).