| Table of Contents |
|---|
...
That would run the 4 MPI ranks on physical cores and not HT cores. That works because, unless overridden, we have To help insure the binding you can set the default SGE core binding strategy to linear. Unless overridden by the user this This strategy will bind the next launched process to the next available core. Since HT cores are mapped after all physical cores this will fill the actual cores first. Once the slots are used, as they will be because the number of slots is 2x the number of cores, the HT cores would be effectively blocked. Note that this will work for non-MPI jobs as well. If you have a non-threaded process that you want to ensure runs on an actual core, you could use the same 2x slot request.
...
Note that core binding is a soft request in SGE. If the binding can not be done the job will still run, if it otherwise has the resources. This is particularly true on machines where jobs are being shared as the actual cores can be bound while still leaving slots available. The only way to assure binding is with dedicated nodes. However, core binding in and of itself may not really boost performance much. Generally speaking, if you want to minimize contention with hardware threads then simply request twice the number of slots than cores your job will use. Even if the processes are not bound to cores, the OS scheduler will do a good job of minimizing contention.
The core binding will be handled differently depending on whether jobs can span multiple nodes, as in MPI, or not. For serial jobs, SGE will bind a process to the next available core. For parallel jobs within a node, ie, the smp PE, SGE will bind to a number of cores equal to half the number of slots requested.
| Info |
|---|
You can modify binding attributes if you wish with the qsub -binding flag or clear all default requested parameters with the qsub -clear flag. |
SGE is not currently able to properly set core binding for multi node jobs. For jobs that span nodes with MPI, the system provided openmpi will bind processes to core by default.
| Expand | ||
|---|---|---|
| ||
| Technically, the binding is to a hardware thread but the mapping of processes is by core and is distributed to span the allocation. Thus, using all 56 HT cores will have all processes bound to HT cores. Running a job with half the number of processes as HT cores will distribute processes on 1 hardware thread per core. That is not quite the same as binding to a core but will ensure that jobs that use all of the HT cores and jobs that use half of the HT cores will work by default. |
...
| Info |
|---|
You can modify binding attributes if you wish with the qsub -binding flag. |
For MPI jobs, the system provided openmpi will not bind processes to cores by default, as would be the normal default for openmpi. In addition, the system openmpi settings will treat the HT cores as processors. This may be important if you wish to run hybrid MPI/OpenMP threaded jobs.
The binding parameters can be overridden with parameters to mpirun. Openmpi provides fine grained control of process layout. The options that are set by default should be good in most cases but can be overridden with the openmpi options for
...
If your job does not use the system openmpi, or does not use MPI, then any desired core binding will need to be set up with whatever mechanism the software uses. Otherwise, there will be no core binding. Again, that may not be a major issue. If your job does not work well with HT then run on a number of cores equal to half of the number of slots requested and the OS scheduler will minimize contention.
new SGE utilities
While SoGE is very similar to previous versions of SGE there are some new utilities that people may find of interest. There are manual pages for each of these.
...