...
Each line of the task file must be incorporated into an array task of the job. The processing of the task file will look very similar for all jobs and so having a tool to handle it is useful, and we provide one, called qbatch-argon, which is a wrapper around the qbatch program with settings specific to the Argon HPC cluster. However, generating the task file is something that is dependent on the jobs, and up to each user, but would generally be done with a script.
...
The job script in this case will not specify the command for the computation, as that is in the task file. Instead, the task file will use the value of $SGE_TASK_ID and correlate that with the line numbers of the task file. The command line is captured for that particular array task and will be run when the queue system launches the array task. The details of generating the job script will not be covered here as we provide the qbatch-argon tool to handle the details.
qbatch
...
The qbatch-argon program is a tool to submit a list of commands in a file (task file) to a queue. The official documentation can be found at qbatch/README.md at master · pipitone/qbatch.
Some of the options of qbatch-argon need a bit of explanation to make the best use of it on University of Iowa HPC clusters. There are defaults set for qbatch-argon such that in many cases you could just execute the following:
| No Format |
|---|
qbatch-argon my_taskfile |
That will create a job script file in the .qbatch folder of the working directory and submit it as an array job. The job will request the all.q queue using the name of the task file as the job name. It will set the current working directory and output stdout/stderr to the logs directory. Options for the queue, the parallel environment, the number of slots, the job name, and some other options can be specified with arguments. It is also possible to specify all qsub options and pass those on to the eventual call to qsub.
Default settings for qbatch
...
The relevant default settings for qbatch-argon on Argon are:
| No Format |
|---|
Processors per core: QBATCH_PPJ=1 Chunksize: QBATCH_CHUNKSIZE=1 Cores: QBATCH_CORES=1 System: QBATCH_SYSTEM=sge SGE_PE: QBATCH_SGE_PE=smp Queue: QBATCH_QUEUE=all.q |
Those, and other settings, can be changed either via variables or the command line. You can not change the setting for system.
Using qbatch
...
The first option to know about is the help option, either -h or --help. That is a good place to start but note that some options and some text are not relevant to the Argon HPC cluster. One of the most important arguments for qbatch-argon is the 'dryrun' option, requested with either -n or --dryrun. This will generate the array job script but not submit it, giving you an opportunity to examine the contents. You could either submit the resultant script manually, or rerun qbatch-argon with the same parameters but without the --dryrun flag to submit it.
...
Every line of the task file expresses a command to run. By default, each line will be incorporated into a separate job array. If we run the program on the previous example,
| No Format |
|---|
qbatch-argon --dryrun my_taskfile |
the following batch script is produced.
...
The contents of the task file are copied into the script. Notice the variables of CHUNK_SIZE and CORES. These correspond to arguments that can be passed to qbatch-argon. The CHUNK_SIZE (-c, --chunksize) controls how many lines of the task file go into each array task. The CORES (-j, --cores) determines how many of those can be run in parallel. In the current example, each line represents an independent computation, so some could be run in parallel. Say that you want to maximize use of a compute node by running many jobs on it. You could alter the command to
| No Format |
|---|
qbatch-argon --dryrun --chunksize 50 --cores 50 --ppj 50 my_taskfile |
...
The computations are no longer independent and every three lines should be incorporated into each array task. Specifying --chunksize=3 will be used for that, but since those commands are not independent, and should be run serially, --cores=1 will be set. Running the following:
| No Format |
|---|
qbatch-argon --dryrun --chunksize=3 my_taskfile |
...
The value of --ppj should be set to the number of slots needed for the command that needs the most resources. So, if the main computation needs 4 slots, then
| No Format |
|---|
qbatch-argon --dryrun --chunksize=3 --ppj=4 my_taskfile |
...
Several of the important options for qsub are set with corresponding flags to qbatch-argon.
...
...
All other options can be passed with the --options flag. For example
| No Format |
|---|
qbatch-argon --dryrun --chunksize=3 --ppj=4 --options='-l ngpus=2' |
...
–header and --footer flags, respectively. For example, to set up modules
| No Format |
|---|
qbatch-argon --dryrun --chunksize=3 --ppj=4 --header='module reset' --header='module load stack/2021.1' --header='module load r-champ' my_taskfile |
...
The order is important and if there are spaces the line must be quoted. You could also use this to add comments.
| No Format |
|---|
qbatch-argon --dryrun --chunksize=3 --ppj=4 --header='# load environment modules' --header='module reset' --header='module load stack/2021.1' --header='module load r-champ' my_taskfile |
...
It is possible to generate the entire submission script with qbatch-argon but if the command line seems too long, you can generate just the important features, use the --dryrun flag, and then copy the resultant script to edit and submit manually.
...
Using natural array jobs are fairly straight forward. Using a task file is a bit more complicated but using qbatch-argon makes much of the difficult work automatic. You just have to keep in mind the number of lines of commands per array task, and make sure that each task has the same resource requirements. As long as necessary files for a job are in the same directory, it is possible to combine many jobs into a single array job submission. Not all jobs that might seem like a good fit for creating an array job can be converted with a task file however. For instance, job dependencies are very coarse with array jobs so if you have job dependencies, using a task file may not be possible, unless the dependencies can be managed within the array task, in sequential order. However, SGE does have some array task dependency capability and it may be possible to craft a set of multiple array tasks that can make use of the
--depend flag of qbatch-argon.