MPI Job Submission on Briareo

Table of Contents

  1. 1. The JDL
  2. 2. mpi-start-wrapper script
  3. 3. mpi-hooks script
  4. 4. Job submission
    1. 4.1 Direct job submission
    2. 4.2 WMS job submission

1.  The JDL

[
JobType = "Normal";
CERequirements = "smpgranularity==<NUMBER_OF_SMP_CORES>";
CpuNumber = <TOTALE_NUMBER_OF_CPUS>;
Executable = "mpi-start-wrapper.sh";
Arguments  = "<EXECUTABLE_PROGRAM> <MPI_ENVIRONMENT>";
StdOutput  = "standard.out";
StdError   = "standard.err";
InputSandbox  = {"mpi-start-wrapper.sh","mpi-hooks.sh","<PROGRAM_CODE>"};
OutputSandbox = {"standard.err","standard.out","mpi-hello"};
OutputSandboxBaseDestUri = <DESTINATION_DIR>;
]
  • CERequirements
    In this option you can define the smpgranularity parameter which are the number of cores you want on the same machines. In our environment (on briareo.grid.elettra.trieste.it) we have 2 dual core CPU, so smpgranularity should be 4. eg.:
    CERequirements = "smpgranularity==4";
  • CpuNumber
    The total number of CPUs on which you want to run. eg.:
    CpuNumber = 12;

If you submit a JDL with the cpu attributes showed above (smpgranularity=4 and cpunumber=12) the batch server will assign you nodes=3,ppn=4. So for smpgranularity=4 cpunumber=32 will be nodes=8,ppn=4 and so on...

  • OutputSandboxBaseDestUri
    This attribute specifies the location where the job will upload the output files. If we omitted this attribute, then the output files should be retrieved manually with glite-wms-job-output. eg.: 
    OutputSandboxBaseDestUri = "gsiftp://briareo.grid.elettra.trieste.it/tmp";
  • mpi-start-wrapper.sh
    As described here this script exports some mandatory variables to perpare the environment. The script accepts 2 arguments, the executable and the environment In our case we have only one environment on briareo.grid.elettra.trieste.it and it is openmpi. So the mpi-start-wrapper script has to be call with OPENMPI option.
    Executable = "mpi-start-wrapper.sh";
Arguments  = "<EXECUTABLE_PROGRAM> OPENMPI";

2.  mpi-start-wrapper script

The script defines the pre and post processing functions by colling a other script (in the example mpi-hooks.sh)
mpi-start-wrapper.sh 
....
export I2G_MPI_PRE_RUN_HOOK=mpi-hooks.sh
export I2G_MPI_POST_RUN_HOOK=mpi-hooks.sh
....
Be sure, when you write you JDL, to include both mpi-start-wrapper.sh and mpi-hooks.sh in your input sandbox!
InputSandbox  = {"mpi-start-wrapper.sh","mpi-hooks.sh"};

3.  mpi-hooks script

The mpi-hooks.sh file provides some pre and post processing operations to perform before and after the code execution. The script can be used to compile the code, to download data, or to analyze results and to save them on a storage element in the grid. An example of this script can be find here.

4.  Job submission

Both direct and wms job submission require a valid user proxy, so you have to obtain it with the voms-proxy-init command.

4.1  Direct job submission

To submit a job directly to the CE you have to: 

glite-ce-job-submit  -a -r briareo.grid.elettra.trieste.it:8443/cream-pbs-iblade <YOUR_JDL>

4.2  WMS job submission

To submit via WMS you have to: 

glite-wms-job-submit  -a -r briareo.grid.elettra.trieste.it:8443/cream-pbs-iblade <YOUR_JDL>