Building and Running on Summit

Setting up the environment

I have a little script that I source after logging in:

[kaig1@login3 ~]$ cat setup-psc-env-summit.sh
module load cmake/3.14.2
module load gcc/7.4.0
module load hdf5/1.10.3
module load cuda/10.1.168

[kaig1@login3 ~]$ source setup-psc-env-summit.sh

It might be even better to load those modules from your .bashrc, so they’ll be available automatically every time you log in.

Getting the code from github

I keep my codes in the ~/src directory, so I’m using this in the following example.

PSC is available from github at https:://github.com/psc-code/psc.git. To use it, make a local clone using git:

[kaig1@login3 ~]$ cd src
[kaig1@login3 src]$ git clone https://github.com/psc-code/psc.git
Cloning into 'psc'...
remote: Enumerating objects: 1245, done.
remote: Counting objects: 100% (1245/1245), done.
remote: Compressing objects: 100% (387/387), done.
remote: Total 102767 (delta 885), reused 1111 (delta 838), pack-reused 101522
Receiving objects: 100% (102767/102767), 39.34 MiB | 24.27 MiB/s, done.
Resolving deltas: 100% (84389/84389), done.
Checking out files: 100% (1467/1467), done.

Building

PSC uses cmake as a build system. The build happens in a separate build directory, e.g. called build-summit:

[kaig1@login3 src]$ cd psc
[kaig1@login3 psc]$ mkdir build-summit
[kaig1@login3 psc]$ cd build-summit

I create another little script file cmake.sh in the build-summit/ directory, so that I know how I invoked cmake if I need to do it again in the future:

#! /bin/bash

cmake \
   -DCMAKE_BUILD_TYPE=Release \
   -DCMAKE_C_COMPILER=mpicc \
   -DCMAKE_CXX_COMPILER=mpicxx \
   -DADIOS2_ROOT=/ccs/home/kaig1/build/ADIOS2/build-summit/install \
   -DUSE_CUDA=ON \
..

Now, invoke cmake:

[kaig1@login3 build-summit]$ ./cmake.sh
-- The C compiler identification is GNU 7.4.0
-- The CXX compiler identification is GNU 7.4.0
[...]

Hopefully, the cmake step will succeed without error, at which point we’re ready to actually compile the code:

[kaig1@login3 build-summit]$ make
[...]

From now on, after making changes, one should only ever need to rebuild the code using the above make command.

Note

On Summit, building the tests creates a bunch of annoying warnings since cmake runs those executables to discover the tests, and on Summit, running those executables gives warnings. The following helps to quiet those down somewhat (but don’t use this when actually running the code with mpirun).

[kaig1@login3 build-summit]$ export OMPI_MCA_btl=tcp,self

Running the tests

The PSC code base includes a bunch of unit tests, though coverage is still far from complete. To run these tests, use ctest:

[kaig1@login3 build-summit]$ ctest .
[...]

100% tests passed, 0 tests failed out of 233

Total Test time (real) = 120.68 sec

Running a job

Here is a job script flatfoil.sh to run the small sample 2-d flatfoil case on Summit:

#! /bin/bash
#BSUB -P AST147
#BSUB -W 00:10
#BSUB -nnodes 1
#BSUB -J flatfoil_summit004

DIR=$PROJWORK/ast147/kaig1/flatfoil-summit004
mkdir -p $DIR
cd $DIR

jsrun -n 4 -a 1 -c 1 -g 1 ~/src/psc/build-summit-gpu/src/psc_flatfoil_yz

Submit as usual:

[kaig1@login3 build-summit-gpu]$ bsub flatfoil.sh
Job <523811> is submitted to default queue <batch>.
[kaig1@login3 build-summit-gpu]$ bjobs
JOBID   USER       STAT   SLOTS    QUEUE       START_TIME    FINISH_TIME   JOB_NAME
523811  kaig1      PEND      -     batch             -             -       flatfoil_summit004