Basic information for OpenFOAM containers at Pawsey

Overview

Teaching: 10 min
Exercises: 5 min

Questions

Basic information for OpenFOAM containers at Pawsey

Objectives

Explain about the OpenFOAM containers maintained by Pawsey

Explain containers MPI requirements to run at Pawsey

Meaning of icons and colours (just for the context of this tutorial)

Series of steps (commands) to be typed in users terminal (guided by instructor)

Series of steps (commands) to be typed in users terminal by themselves (in breakout room)

Collapsed blocks, with additional information, important parts of the scripts or pre-executed steps

No action required

The rest are just blocks …

… with general information

A. OpenFOAM containers maintained by Pawsey

OpenFOAM Singularity images maintained by Pawsey are in the following directory:
/group/singularity/pawseyRepository/OpenFOAM/

A.I List the content of our repository

To check which versions we currently maintain:

 zeus-1:~> ls -lat /group/singularity/pawseyRepository/OpenFOAM

 total 5855948
 drwxrwsr-x  2 maali    pawsey0001       4096 May 13 13:59 .
 -rwxr-x---+ 1 espinosa pawsey0001 1005383680 May  7 10:24 openfoam-2.4.x-pawsey.sif
 -rwxr-xr-x+ 1 espinosa pawsey0001 1330900992 May  6 12:38 openfoam-v1912-pawsey.sif
 -rwxr-xr-x+ 1 espinosa pawsey0001 1199386624 May  5 17:14 openfoam-7-pawsey.sif
 -rwxr-xr-x  1 espinosa pawsey0001 1308553216 Apr  6 12:23 openfoam-v1812-pawsey.sif
 drwxrwsr-x  3 maali    pawsey0001       4096 Mar  3 09:52 ..
 -rwxr-xr-x+ 1 espinosa pawsey0001 1152237568 Feb 11 13:27 openfoam-5.x-pawsey.sif

A.II Perform a basic mini-test

Load the Singularity module:
```
zeus-1:~> module load singularity
```

Choose one image (we like to save its name in a variable called theImage):

zeus-1:~> theImage="/group/singularity/pawseyRepository/OpenFOAM/openfoam-v1912-pawsey.sif"

Use singularity to execute the icoFoam solver inside the chose container and call its help message:

zeus-1:~> singularity exec $theImage icoFoam -help

Usage: icoFoam [OPTIONS]
Options:
  -case <dir>       Specify case directory to use (instead of the cwd)
  -decomposeParDict <file>
                    Use specified file for decomposePar dictionary
  -dry-run          Check case set-up only using a single time step
  -dry-run-write    Check case set-up and write only using a single time step
  -parallel         Run in parallel
  -postProcess      Execute functionObjects only
  -doc              Display documentation in browser
  -help             Display short help and exit
  -help-full        Display full help and exit
   
Transient solver for incompressible, laminar flow of Newtonian fluids.
   
Using: OpenFOAM-v1912 (1912) - visit www.openfoam.com
Build: _f3950763fe-20191219
Arch:  LSB;label=32;scalar=64

Yes, you can also use your own containers

For example, I keep my personal containers in:
zeus-1:~> ls /group/pawsey0001/espinosa/singularity/myRepository/OpenFOAM/
openfoam-7-mickey.sif      openfoam-v1912-esi.sif
openfoam-7-foundation.sif
And our group containers in:
zeus-1:~> ls /group/pawsey0001/singularity/groupRepository/OpenFOAM/
openfoam-5.x_CFDEM-pawsey.sif

B. OpenFOAM containers need MPICH to run on Crays

Singularity Hybrid Mode for MPI applications

“Hybrid mode” execution is a way in which Singularity allows a containerised MPI application to use the host MPI libraries for better performance.

The only restriction for the hybrid mode to work is that host-MPI installation and container-MPI installation to be ABI compatible

(ABI=Application Binary Interface)

More information about the “hybrid mode” can be found in the Singularity documentation

Why do we prefer Hybrid-mode?

Because it gives better performance:

Running containerised MPI applications with the internal MPI is not the best approach.

For example, we have tested the solution of the channel395 tutorial (10 executions each) on a desktop computer with 4 cores:

(All cases use the fileHandler collated; purgeWrite 10; writeFormat binary; runtimeModifiable false;)

Tutorial Container Mode Avg.ClockTime

channel395 openfoam/openfoam7-paraview56:latest Docker-internalOpenMPI 1064.8 s

channel395 openfoam-7-foundation.sif Singularity-internalOpenMPI 806.4 s

channel395 openfoam-7-foundation.sif Singularity-hybrid-HostOpenMPI 787.2 s

channel395 pawsey/openfoam:7 Docker-internalMPICH 975.4 s

channel395 openfoam-7-pawsey.sif Singularity-internalMPICH 783.6 s

channel395 openfoam-7-pawsey.sif Singularity-hybrid-HostMPICH 779.2 s

It is also the only way to run multi-node applications

Tutorial	Container	Mode	Avg.ClockTime
channel395	openfoam/openfoam7-paraview56:latest	Docker-internalOpenMPI	1064.8 s
channel395	openfoam-7-foundation.sif	Singularity-internalOpenMPI	806.4 s
channel395	openfoam-7-foundation.sif	Singularity-hybrid-HostOpenMPI	787.2 s
channel395	pawsey/openfoam:7	Docker-internalMPICH	975.4 s
channel395	openfoam-7-pawsey.sif	Singularity-internalMPICH	783.6 s
channel395	openfoam-7-pawsey.sif	Singularity-hybrid-HostMPICH	779.2 s

Why MPICH?

To achieve the best performance in a supercomputer, we want to run in hybrid mode with the optimised host MPI

In the case of Magnus, Cray-MPI (ABI compatible to MPICH) is the only supported flavour

Therefore, we decided to support containerised MPI applications compiled with MPICH

MPICH containers also run properly on Zeus

Main drawback:

Developers’ OpenFOAM containers equipped with OpenMPI would not run properly on Magnus (even after conversion to Singularity)

OpenFOAM containers need to be built from scratch with MPICH

B.I Briefly check the settings at Pawsey

You can check the definition of key variables for hybrid-mode execution (SINGULARITY_BINDPATH and SINGULARITYENV_LD_LIBRARY_PATH) with:

zeus-1:~> module show singularity

---------------------------------------------------------------------------------------------------------------
   /pawsey/sles12sp3/modulefiles/devel/singularity/3.5.2.lua:
---------------------------------------------------------------------------------------------------------------
help([[Sets up the paths you need to use singularity version 3.5.2]])
whatis("Singularity enables users to have full control of their environment. Singularity 
containers can be used to package entire scientific workflows, software and 
libraries, and even data.
   
For further information see https://sylabs.io/singularity")
whatis("Compiled with gcc/4.8.5")
setenv("MAALI_SINGULARITY_HOME","/pawsey/sles12sp3/devel/gcc/4.8.5/singularity/3.5.2")
prepend_path("MANPATH","/pawsey/sles12sp3/devel/gcc/4.8.5/singularity/3.5.2/share/man")
prepend_path("PATH","/pawsey/sles12sp3/devel/gcc/4.8.5/singularity/3.5.2/bin")
setenv("SINGULARITYENV_LD_LIBRARY_PATH","/usr/lib64:/pawsey/intel/17.0.5/compilers_and_libraries/linux/mpi/intel64/lib")
setenv("SINGULARITY_BINDPATH","/astro,/group,/scratch,/pawsey,/etc/dat.conf,/etc/libibverbs.d,/usr/lib64/libdaplofa.so.2,/usr/lib64/libdaplofa.so.2.0.0,/usr/lib64/libdat2.so.2,/usr/lib64/libdat2.so.2.0.0,/usr/lib64/libibverbs,/usr/lib64/libibverbs.so,/usr/lib64/libibverbs.so.1,/usr/lib64/libibverbs.so.1.1.14,/usr/lib64/libmlx5.so,/usr/lib64/libmlx5.so.1,/usr/lib64/libmlx5.so.1.1.14,/usr/lib64/libnl-3.so.200,/usr/lib64/libnl-3.so.200.18.0,/usr/lib64/libnl-cli-3.so.200,/usr/lib64/libnl-cli-3.so.200.18.0,/usr/lib64/libnl-genl-3.so.200,/usr/lib64/libnl-genl-3.so.200.18.0,/usr/lib64/libnl-idiag-3.so.200,/usr/lib64/libnl-idiag-3.so.200.18.0,/usr/lib64/libnl-nf-3.so.200,/usr/lib64/libnl-nf-3.so.200.18.0,/usr/lib64/libnl-route-3.so.200,/usr/lib64/libnl-route-3.so.200.18.0,/usr/lib64/librdmacm.so,/usr/lib64/librdmacm.so.1,/usr/lib64/librdmacm.so.1.0.14")
se   tenv("SINGULARITY_CACHEDIR","/group/pawsey0001/espinosa/.singularity")

And you can check the MPI version installed inside the container with:

zeus-1:~> theImage="/group/singularity/pawseyRepository/OpenFOAM/openfoam-v1912-pawsey.sif"
zeus-1:~> singularity exec $theImage mpiexec --version

HYDRA build details:
    Version:                                 3.1.4
    Release Date:                            Fri Feb 20 15:02:56 CST 2015
    CC:                              gcc    
    CXX:                             g++    
    F77:                             gfortran   
    F90:                             gfortran   
    Configure options:                       '--disable-option-checking' '--prefix=/usr' '--enable-fast=all,O3' '--cache-file=/dev/null' '--srcdir=.' 'CC=gcc' 'CFLAGS= -DNDEBUG -DNVALGRIND -O3' 'LDFLAGS= ' 'LIBS=-lpthread ' 'CPPFLAGS= -I/tmp/mpich-build/mpich-3.1.4/src/mpl/include -I/tmp/mpich-build/mpich-3.1.4/src/mpl/include -I/tmp/mpich-build/mpich-3.1.4/src/openpa/src -I/tmp/mpich-build/mpich-3.1.4/src/openpa/src -D_REENTRANT -I/tmp/mpich-build/mpich-3.1.4/src/mpi/romio/include'
    Process Manager:                         pmi
    Launchers available:                     ssh rsh fork slurm ll lsf sge manual persist
    Topology libraries available:            hwloc
    Resource management kernels available:   user slurm ll lsf sge pbs cobalt
    Checkpointing libraries available:       
    Demux engines available:                 poll select

Key Points

A singularity image is a file that can be stored anywhere, but we reccommend to use some defined “policy” within your group

OpenFOAM images maintained by pawsey are stored at /group/singularity/pawseyRepository/OpenFOAM/

Containers with MPI applications need to be equipped with MPICH for running on Crays

Executing the full workflow of a case

Overview

Teaching: 30 min
Exercises: 30 min

Questions

How can I use OpenFOAM containers at Pawsey supercomputers?

Objectives

Explain how to use typical pre-processing, solving and post-processing OpenFOAM tools

0. Introduction

Typical workflow

Typical steps for analysing a problem with OpenFOAM include:

The setup of initial conditions, mesh definition and solver parameters

Use of pre-processing tools (typically decompose the case into several subdomains)

Execute a solver to obtain a flow field solution

Use post-processing tools (typically reconstruct the decomposed results into a single domain result)

0.I Accessing the scripts for this episode

Here we make use of a series of scripts to cover a typical workflow in full.
cd into the directory where the provided scripts are. In this case we’ll use OpenFOAM-v1912.
 zeus-1:~> cd $MYSCRATCH/pawseyTraining/containers-openfoam-workshop-scripts
 zeus-1:*-scripts> cd 02_executingFullWorkflow/example_OpenFOAM-v1912
 zeus-1:*-v1912> ls
 A.extractTutorial.sh  B.adaptCase.sh  C.decomposeFoam.sh  D.runFoam.sh  E.reconstructFoam.sh  run
Quickly read one of the scripts, for example C.decomposeFoam.sh. We recommed the following text readers:

view (navigate with up and down arrows, use :q or :q! to quit)

less (navigate with up and down arrows, use q to quit) (this one does not have syntax highlight)

If you are using an editor to read the scripts, DO NOT MODIFY THEM! because your exercise could mess up

(Try your own settings after succeding with the original exercise, ideally in a copy of the script)
 zeus-1:*-v1912> view C.decomposeFoam.sh
 ~
 ~
 ~
 :q
 zeus-1*-v1912>

Sections and scripts for this episode

In the following sections, there are instructions for submitting these job scripts for execution in the supercomputer one by one:

A.extractTutorial.sh (already pre-executed) is for copying a tutorial from the interior of the container into our local file system

B.adaptCase.sh (already pre-executed) is for modifying the tutorial to comply with Pawsey’s best practices

C.decomposeFoam.sh is for executing the mesher and the decomposition of initial condition into subdomains

D.runFoam.sh is for executing the solver

E.reconstructFoam.sh is for executing the reconstruction of the last available result time

So how will this episode flow?

The first two scripts (A. and B.) have already been executed for you. So we can concentrate in the main three stages of OpenFOAM usage.

We’ll start with a detailed explanation at the final step of section “B. Adapt the case”.

And continue the explanation up to the beginning of section “C. Decomposition”.

Users will then proceed by themselves afterwards.

At the end we’ll discuss the main instructions in the scripts and the whole process.

A. Extraction of the tutorial: channel395

The `A.extractTutorial.sh` script

Main command in the script:

tutorialCase=incompressible/pimpleFoam/LES/channel395
caseDir=$SLURM_SUBMIT_DIR/run/channel395
srun -n 1 -N 1 singularity exec $theImage cp -r /opt/OpenFOAM/OpenFOAM-$theVersion/tutorials/$tutorialCase $caseDir

Other important parts of the script:

#!/bin/bash -l
#SBATCH --export=NONE
#SBATCH --time=00:05:00
#SBATCH --ntasks=1
#SBATCH --partition=copyq #Ideally, you should be using the copyq for this kind of processes

#1. Load the necessary modules
module load singularity

#2. Defining the container to be used
theRepo=/group/singularity/pawseyRepository/OpenFOAM
theContainerBaseName=openfoam
theVersion=v1912
theProvider=pawsey
theImage=$theRepo/$theContainerBaseName-$theVersion-$theProvider.sif

#3. Defining the tutorial and the case directory
tutorialAppDir=incompressible/pimpleFoam/LES
tutorialName=channel395
tutorialCase=$tutorialAppDir/$tutorialName

baseWorkingDir=$SLURM_SUBMIT_DIR/run
if ! [ -d $baseWorkingDir ]; then
    echo "Creating baseWorkingDir=$baseWorkingDir"
mkdir -p $baseWorkingDir
fi
caseName=$tutorialName
caseDir=$baseWorkingDir/$caseName

#4. Copy the tutorialCase to the workingDir
if ! [ -d $caseDir ]; then
   srun -n 1 -N 1 singularity exec $theImage cp -r /opt/OpenFOAM/OpenFOAM-$theVersion/tutorials/$tutorialCase $caseDir
else
   echo "The case=$caseDir already exists, no new copy has been performed"
fi

A.I Steps for dealing with the extraction of the “channel395” case: - [Pre-Executed]
Submit the job (no need for reservation as the script uses the copyq partition)
zeus-1:*-v1912> sbatch A.extractTutorial.sh 
Submitted batch job 4632458
zeus-1:*-v1912> squeue -u $USER
JOBID    USER     ACCOUNT     PARTITION            NAME EXEC_HOST ST     REASON   START_TIME     END_TIME  TIME_LEFT NODES   PRIORITY
4632468  espinosa pawsey0001  workq      A.extractTutor       n/a PD       None          N/A          N/A       5:00     1      75190
Check that the tutorial has been copied to our local file system
zeus-1:*-v1912> ls ./run/channel395/
0  0.orig  Allrun  constant  system

B. Adapt the case to your needs (and Pawsey best practices)

The `B.adaptCase.sh` script

Main command in the script

foam_runTimeModifiable="false"
sed -i 's,^runTimeModifiable.*,runTimeModifiable    '"$foam_runTimeModifiable"';,' ./system/controlDict

Other important parts of the script:

#5. Defining OpenFOAM controlDict settings for Pawsey Best Practices
##5.1 Replacing writeFormat, runTimeModifiable and purgeRight settings
foam_writeFormat="binary"
sed -i 's,^writeFormat.*,writeFormat    '"$foam_writeFormat"';,' ./system/controlDict
foam_runTimeModifiable="false"
sed -i 's,^runTimeModifiable.*,runTimeModifiable    '"$foam_runTimeModifiable"';,' ./system/controlDict
foam_purgeWrite=10
sed -i 's,^purgeWrite.*,purgeWrite    '"$foam_purgeWrite"';,' ./system/controlDict

##5.2 Defining the use of collated fileHandler of output results 
echo "OptimisationSwitches" >> ./system/controlDict
echo "{" >> ./system/controlDict
echo "   fileHandler collated;" >> ./system/controlDict
echo "}" >> ./system/controlDict

B.I Initial steps for dealing with the adaptation of the case - [Pre-Executed]
Submit the adaptation script
zeus-1:*-v1912> sbatch B.adaptCase.sh 
Submitted batch job 4632548

B.II Final steps. To check the adapted settings:
cd into the case directory:
zeus-1:*-v1912> cd run/channel395
zeus-1:channel395> ls
0  0.orig  Allrun  constant  system
Initial conditions are in the subdirectory 0

Mesh and fluid properties definition are under the tree of the subdirectory constant

Solver settings are in the “dictionaries” inside the system subdirectory
Read the controlDict dictionary inside the system subdirectory:
zeus-1:channel395> view system/controlDict
~
~
~
:q
The settings that were adapted in ..../run/channel395/system/controlDict
To keep only a few result directories at a time (10 maximum in this case)
 purgeWrite      10;
To use binary writing format to accelerate writing and reduce the size of the files
 writeFormat     binary;
Never use runTimeModifiable. This option creates permanent reading of dictionaries (each time step) which overloads the shared file system.
 runTimeModifiable false;
If version is higher-or-equal than OpenFOAM-6 or OpenFOAM-v1812, always use the collated option
 optimisationSwitches
 {
     fileHandler collated;
 }

C. Decomposition

The `C.decomposeFoam.sh` script

Main command in the script:

srun -n 1 -N 1 singularity exec $theImage decomposePar -cellDist -force

Other important parts of the script:

#!/bin/bash -l
#SBATCH --ntasks=1
#SBATCH --mem=4G
#SBATCH --ntasks-per-node=28
#SBATCH --clusters=zeus
#SBATCH --partition=workq
#SBATCH --time=0:10:00
#SBATCH --export=none

#1. Load the necessary modules
module load singularity

#2. Defining the container to be used
theRepo=/group/singularity/pawseyRepository/OpenFOAM
theContainerBaseName=openfoam
theVersion=v1912
theProvider=pawsey
theImage=$theRepo/$theContainerBaseName-$theVersion-$theProvider.sif

#3. Defining the case directory
baseWorkingDir=$SLURM_SUBMIT_DIR/run
caseName=channel395
caseDir=$baseWorkingDir/$caseName

#4. Going into the case and creating the logs directory
if [ -d $caseDir ]; then
   cd $caseDir
   echo "pwd=$(pwd)"
else
   echo "For some reason, the case=$caseDir, does not exist"
   echo "Exiting"; exit 1
fi
logsDir=./logs/pre
if ! [ -d $logsDir ]; then
   mkdir -p $logsDir
fi

#6. Defining the ioRanks for collating I/O
# groups of 2 for this exercise (please read our documentation for the recommendations for production runs)
export FOAM_IORANKS='(0 2 4 6)'

#7. Perform all preprocessing OpenFOAM steps up to decomposition
echo "Executing blockMesh"
srun -n 1 -N 1 singularity exec $theImage blockMesh 2>&1 | tee $logsDir/log.blockMesh.$SLURM_JOBID
echo "Executing decomposePar"
srun -n 1 -N 1 singularity exec $theImage decomposePar -cellDist -force 2>&1 | tee $logsDir/log.decomposePar.$SLURM_JOBID

C.I Steps for dealing with decomposition:
Submit the decomposition script from the scripts directory (use the reservation for the workshop if available)
zeus-1:*-v1912> myReservation=containers
zeus-1:*-v1912> sbatch --reservation=$myReservation C.decomposeFoam.sh 
If you do not have a reservation

Then, submit normally (or choose the best partition for executing the exercise, the debugq for example:)
   zeus-1:*-v1912> sbatch -p debugq C.decomposeFoam.sh
Submitted batch job 4632558
Check that the decomposition has been performed:
zeus-1:*-v1912> ls ./run/channel395/processor*
./run/channel395/processors4_0-1:
0  constant
   
./run/channel395/processors4_2-3:
0  constant
You should also check for success/errors in:

the slurm output file: slurm-<SLURM_JOBID>.out

the log files created when executing the OpenFOAM tools in: ./run/channel395/logs/pre/

D. Executing the solver

The `D.runFoam.sh` script

Main command in the script:

theSolver=myPimpleFoam
srun -n $SLURM_NTASKS -N $SLURM_JOB_NUM_NODES singularity exec $theImage $theSolver -parallel

Other important parts of the script:

#SBATCH --ntasks=4
#SBATCH --mem=16G
#SBATCH --ntasks-per-node=28
#SBATCH --cluster=zeus

#5. Reading OpenFOAM decomposeParDict settings
foam_numberOfSubdomains=$(grep "^numberOfSubdomains" ./system/decomposeParDict | tr -dc '0-9')

#7. Checking if the number of tasks coincide with the number of subdomains
if [[ $foam_numberOfSubdomains -ne $SLURM_NTASKS ]]; then
   echo "foam_numberOfSubdomains read from ./system/decomposeParDict is $foam_numberOfSubdomains"
   echo "and"
   echo "SLURM_NTASKS in this job is $SLURM_NTASKS"
   echo "These should be the same"
   echo "Therefore, exiting this job"
   echo "Exiting"; exit 1
fi

#8. Defining OpenFOAM controlDict settings for this run
foam_startFrom=startTime
#foam_startFrom=latestTime
foam_startTime=0
#foam_startTime=15
foam_endTime=10
#foam_endTime=30
foam_writeInterval=1
foam_purgeWrite=10

#9. Changing OpenFOAM controlDict settings
sed -i 's,^startFrom.*,startFrom    '"$foam_startFrom"';,' system/controlDict
sed -i 's,^startTime.*,startTime    '"$foam_startTime"';,' system/controlDict
sed -i 's,^endTime.*,endTime    '"$foam_endTime"';,' system/controlDict
sed -i 's,^writeInterval.*,writeInterval    '"$foam_writeInterval"';,' system/controlDict
sed -i 's,^purgeWrite.*,purgeWrite    '"$foam_purgeWrite"';,' system/controlDict

#10. Defining the solver
theSolver=pimpleFoam

#11. Execute the case 
echo "About to execute the case"
srun -n $SLURM_NTASKS -N $SLURM_JOB_NUM_NODES singularity exec $theImage $theSolver -parallel 2>&1 | tee $logsDir/log.$theSolver.$SLURM_JOBID
echo "Execution finished"

D.I Steps for dealing with the solver

Submit the solver script (from the scripts directory)
```
zeus-1:*-v1912> sbatch --reservation=$myReservation D.runFoam.sh 
```
- If a reservation is not available, do not use the option. (Or you can use the debugq: --partition=debugq instead.)
```
Submitted batch job 4632685 on cluster zeus
```

Check that the status of your job:

zeus-1:*-v1912> squeue -u $USER

JOBID    USER     ACCOUNT     PARTITION            NAME EXEC_HOST ST     REASON   START_TIME     END_TIME  TIME_LEFT NODES   PRIORITY
4632685  espinosa pawsey0001  workq        D.runFoam.sh       n/a PD  Resources     17:09:28     17:19:28      10:00     1      75190 

Observe the output of the job with tail -f at runtime (press <Ctrl-C> to exit the command):

zeus-1:*-v1912> tail -f slurm-4632685.out

.
.
.
Time = 0.2
   
PIMPLE: iteration 1
smoothSolver:  Solving for Ux, Initial residual = 0.0118746, Final residual = 1.89249e-06, No Iterations 3
smoothSolver:  Solving for Uy, Initial residual = 0.0617212, Final residual = 1.68113e-06, No Iterations 4
smoothSolver:  Solving for Uz, Initial residual = 0.0589944, Final residual = 9.70923e-06, No Iterations 3
Pressure gradient source: uncorrected Ubar = 0.13369, pressure gradient = -0.000964871
GAMG:  Solving for p, Initial residual = 0.213844, Final residual = 0.00414884, No Iterations 2
time step continuity errors : sum local = 5.82807e-06, global = -1.41211e-19, cumulative = -1.41211e-19
Pressure gradient source: uncorrected Ubar = 0.133687, pressure gradient = -0.000947989
GAMG:  Solving for p, Initial residual = 0.0222643, Final residual = 4.30412e-07, No Iterations 7
time step continuity errors : sum local = 5.63638e-10, global = -2.40486e-19, cumulative = -3.81697e-19
Pressure gradient source: uncorrected Ubar = 0.133687, pressure gradient = -0.000947874
ExecutionTime = 0.25 s  ClockTime = 0 s
.
.
.

Check that the solver gave some results:

zeus-1:*-v1912> ls ./run/channel395/processor*

./run/channel395/processors4_0-1:
0  10  8.2  8.4  8.6  8.8  9  9.2  9.4  9.6  9.8  constant
   
./run/channel395/processors4_2-3:
0  10  8.2  8.4  8.6  8.8  9  9.2  9.4  9.6  9.8  constant

You should also check for success/errors in:
- the slurm output file: slurm-<SLURM_JOBID>.out
- the log files created when executing the OpenFOAM tools in: ./run/channel395/logs/run/

E. Reconstruction

The `E.reconstructFoam.sh` script

Main command in the script:

srun -n 1 -N 1 singularity exec $theImage reconstructPar -latestTime

Other important parts of the script:

#SBATCH --ntasks=1
#SBATCH --mem=16G
#SBATCH --ntasks-per-node=28
#SBATCH --clusters=zeus

#7. Execute reconstruction
echo "Start reconstruction"
srun -n 1 -N 1 singularity exec $theImage reconstructPar -latestTime 2>&1 | tee $logsDir/log.reconstructPar.$SLURM_JOBID

E.1 Steps for dealing with reconstruction:
Submit the reconstruction script (from the scripts directory)
zeus-1:*-v1912> sbatch --reservation=$myReservation E.reconstructFoam.sh 
If a reservation is not available, do not use the option. (Or you can use the debugq: --partition=debugq instead.)
Submitted batch job 4632899 on cluster zeus
Check that the reconstruction has been performed (a directory for the last time of the solution, 10 in this case, should appear at in the case directory):
zeus-1:*-v1912> ls ./run/channel395/
0  0.orig  10  Allrun  constant  logs  processors4_0-1  processors4_2-3  system
You should also check for success/errors in:

the slurm output file: slurm-<SLURM_JOBID>.out

the log files created when executing the OpenFOAM tools in: ./run/channel395/logs/post/

Y. The collated option

Information about the fileHandler collated option

The default for common OpenFOAM installations outside Pawsey is to output results into many processor* subdirectories. Indeed, as many as decomposed subdomains. So, for example, in a common installation you would have finished with the following directories in your $caseDir:
0  0.orig  Allrun  constant  logs  processor0  processor1  processor2  processor3  system
So, the default fileHandler for common installations is uncollated. That is not a problem for a small case like this tutorial, but it is indeed a problem when the decomposition requires hundreds of subdomains. Then if results are saved very frequently, and the user needs to keep all their output for analysis (purgeWrite=0), they can end with millions of files. And the problem increases if the user needs to analyse and keep results for multiple cases.

The existence of millions of files is already a problem for the user, but is also a problem for all our user base, because the file system manager gets overloaded and the performance of the file system (which is shared) degrades.

Because of that, for versions greater-than or equal-to OpenFOAM-6 and OpenFOAM-v1812, the use “fileHandler collated” is a required policy in our systems. (Older versions do not have that option or do not perform well.) The collated option allows result files to be merged into a reduce number of directories. (We have set that in point B.adaptCase (above) within the controlDict. Pawsey containers and system-wide installations have the collated option set by default. Nevertheless, we decided to explicitly include it in the controlDict as a reminder to our users.)

Together with the setting of the collated option, the merging of results is controlled by the ioRanks option. In this case, the option is being set through the FOAM_IORANKS environment variable.

In the scripts we have:
export FOAM_IORANKS='(0 2 4 6 8)'
which indicates that decomposed results will be merged every 2 subdomains. [0-1] in the first subdirectory, and [2-3] in the second one. As seen in our case directory:
0  0.orig  Allrun  constant  logs  processors4_0-1  processors4_2-3  system
(The extra numbers in the list does not affect the settings, but strictly speaking ‘(0 2)’ would be enough.) The “export” is needed for the variable to be exported to the srun executions.

If the ioRanks option were not set, then the collated option alone would have merged all the results into a single directory, as this:
0  0.orig  Allrun  constant  logs  processors4  system
In production runs, we recommend to group results for each node into a single directory. So, for a case to be solved on Magnus, the recommended setting is:
export FOAM_IORANKS='(0 24 48 72 . . . numberOfSubdomains)'
and for a case to be solved on Zeus:
export FOAM_IORANKS='(0 28 56 84 . . . numberOfSubdomains)'
If the numberOfSubdomains fit in a single node, we recommend to use the plain collated option alone without ioRanks.

Also, we recommend to decompose your domain so that subdomains count with ~100,000 cells. This rule of thumb gives good performance in most of situations, although users need to verify the best settings for their own cases and solvers.

Z. Further notes on how to use OpenFOAM and OpenFOAM containers at Pawsey

The usage of OpenFOAM and OpenFOAM containers at Pawsey has already been described in our documentation: OpenFOAM documentation at Pawsey

and in a technical newsletter note: https://support.pawsey.org.au/documentation/display/US/Pawsey+Technical+Newsletter+2020-04

Key Points

Use singularity exec $image <OpenFOAM-Tool> <Tool-Options> for using containerised OpenFOAM tools

Pre- and Post-Processing are usually single threaded and should be executed on Zeus

Always use the recommended Pawsey Best Practices for OpenFOAM

Most recent versions of OpenFOAM are not installed system-wide at Pawsey’s Supercomputers, but are available via singularity containers

Compile and execute user's own tools

Overview

Teaching: 30 min
Exercises: 30 min

Questions

How can users execute tools that are not initially installed inside the container?

Objectives

Compile and execute user’s own tools with the compiler and OpenFOAM installation of an existing container

0. Introduction

Users own tools

One of the attractive features of OpenFOAM is the possibility of building your own tools/solvers

And execute them together with the whole OpenFOAM environment

OpenFOAM containers are usually equipped only with the standard tools/solvers

Nevertheless, users can still compile their own tools and use them with a standard container

The trick is to bind the local host directory to the path where the internal installation looks for user’s source files/tools: WM_PROJECT_USER_DIR

0.I Accessing the scripts for this episode

In this whole episode, we make use of a series of scripts to cover a typical compilation/execution workflow. Lets start by listing the scripts.
cd into the directory where the provided scripts are. In this case we’ll use OpenFOAM-v1912.
zeus-1:~> cd $MYSCRATCH/pawseyTraining/containers-openfoam-workshop-scripts
zeus-1:*-scripts> cd 03_compileAndExecuteUsersOwnTools/example_OpenFOAM-v1912
zeus-1:*-v1912> ls
A.cloneMyPimpleFoam.sh    C.extractTutorial.sh  E.decomposeFoam.sh  projectUserDir
B.compileMyPimpleFoam.sh  D.adaptCase.sh        F.runFoam.sh        run
Quickly read one of the scripts, for example B.compileMyPimpleFoam.sh. We recommed the following text readers:

view (navigate with up and down arrows, use :q or :q! to quit)

or less (navigate with up and down arrows, use q to quit) (this one does not have syntax highlight)

If you are using an editor to read the scripts, DO NOT MODIFY THEM! because your exercise could mess up

(Try your own settings after succeding with the original exercise, ideally in a copy of the script)
zeus-1:*-v1912> view B.compileMyPimpleFoam.sh
~
~
~
:q
zeus-1*-v1912>

Sections and scripts for this episode

In the following sections, there are instructions for submitting these job scripts for execution in the supercomputer one by one:

A.cloneMyPimpleFoam.sh (already pre-executed) is for copying the source files of an existing solver into our local file system

After the copy, the script above also renames the solver as a user’s own solver myPimpleFoam

B.compileMyPimpleFoam.sh is for compiling user’s own solver

C.extractTutorial.sh (already pre-executed) is for copying a tutorial from the interior of the container into our local file system

D.adaptCase.sh (already pre-executed) is for modifying the tutorial to comply with Pawsey’s best practices

E.decomposeFoam.sh is for meshing and decomposing the intial conditions of the case

F.runFoam.sh is for executing the user’s own solver

So how will this episode flow?

The script of section “A” has already been pre-executed.

We’ll start with a detailed explanation at the end of section “A. Cloning of the solver”.

And continue the explanation through all section “B. compileMyPimpleFoam”.

Users will then jump to section E. and proceed by themselves afterwards.

At the end, of each section we’ll discuss the main instructions within the scripts and the whole process.

A. Cloning a standard solver into user’s own solver `myPimpleFoam`

The `A.cloneMyPimpleFoam.sh` script

Main command in the script:
appDirInside=applications/solvers/incompressible
solverOrg=pimpleFoam
solverNew=myPimpleFoam
srun -n 1 -N 1 singularity exec $theImage bash -c 'cp -r $WM_PROJECT_DIR/'"$appDirInside/$solverOrg $projectUserDir/applications/$solverNew"
The WM_PROJECT_DIR variable only exist inside the container, that is why it is being evaluated using the bash -c command and the sigle quotes

(See the explanation of the bash -c command at the end of this section if needed)

Other important parts of the script:

#3. Define the user directory in the local host and the place where to put the solver
#projectUserDir=$MYGROUP/OpenFOAM/$USER-$theVersion/workshop/02_runningUsersOwnTools
projectUserDir=$SLURM_SUBMIT_DIR/projectUserDir
if ! [ -d $projectUserDir/applications ]; then
   mkdir -p $projectUserDir/applications
else
   echo "The directory $projectUserDir/applications already exists."
fi

#4. Copy the solver from the inside of the container to the local file system
appDirInside=applications/solvers/incompressible
solverOrg=pimpleFoam
solverNew=myPimpleFoam
if ! [ -d $projectUserDir/applications/$solverNew ]; then
   srun -n 1 -N 1 singularity exec $theImage bash -c 'cp -r $WM_PROJECT_DIR/'"$appDirInside/$solverOrg $projectUserDir/applications/$solverNew"
else
   echo "The directory $projectUserDir/applications/$solverNew already exists, no new copy has been performed"
fi

(See the explanation of the bash -c command at the end of this section if needed)

#5. Going into the new solver directory
if [ -d $projectUserDir/applications/$solverNew ]; then
   cd $projectUserDir/applications/$solverNew
   echo "pwd=$(pwd)"
else
   echo "For some reason, the directory $projectUserDir/applications/$solverNew, does not exist"
   echo "Exiting"; exit 1
fi

#6. Remove not needed stuff
echo "Removing not needed stuff"
rm -rf *DyMFoam SRFP* *.dep

#7. Rename the source files and replace words inside for the new solver to be: "myPimpleFoam"
echo "Renaming the source files"
rename pimpleFoam myPimpleFoam *
sed -i 's,pimpleFoam,myPimpleFoam,g' *.C
sed -i 's,pimpleFoam,myPimpleFoam,g' *.H

#8. Modify files inside the Make directory to create the new executable in $FOAM_USER_APPBIN
echo "Adapting files inside the Make directory"
sed -i 's,pimpleFoam,myPimpleFoam,g' ./Make/files
sed -i 's,FOAM_APPBIN,FOAM_USER_APPBIN,g' ./Make/files

A.I Initial steps for dealing with this section - [Pre-Executed]
Submit the job (no need for reservation as the script uses the copyq partition)
zeus-1:*-v1912> sbatch A.cloneMyPimpleFoam.sh 
Submitted batch job 4632458

A.II Final steps. To check the source code and important settings for the compilation:

Users stuff in the local host is saved under the ./projectUserDir.

At this point, it only has an applications folder where the new solver source files are kept:
Check what is in the ./projectUserDir
zeus-1:*-v1912> ls projectUserDir/ 
applications
cd into the myPimpleFoam folder within applications
zeus-1:*-v1912> cd projectUserDir/applications/myPimpleFoam
zeus-1:myPimpleFoam> ls
correctPhi.H  createFields.H  Make  myPimpleFoam.C  pEqn.H  UEqn.H
There is our own new solver to be compiled
Take a quick look to the source file of the solver:
zeus-1:myPimpleFoam> view myPimpleFoam.C
~
~
~
:q
Discussion of the intrinsics of the solver are out of the scope of this training

But we know it works because it is an exact replica of the standard pimpleFoam solver (but renamed)
Inside the Make directory there are two important files:
zeus-1:myPimpleFoam> cd Make
zeus-1:Make> ls
files  options
In the file options there is a list of the OpenFOAM libraries to be included in the solver:
zeus-1:Make> cat options
EXE_INC = \
    -I$(LIB_SRC)/finiteVolume/lnInclude \
    -I$(LIB_SRC)/meshTools/lnInclude \
    -I$(LIB_SRC)/sampling/lnInclude \
    -I$(LIB_SRC)/TurbulenceModels/turbulenceModels/lnInclude \
    -I$(LIB_SRC)/TurbulenceModels/incompressible/lnInclude \
    -I$(LIB_SRC)/transportModels \
    -I$(LIB_SRC)/transportModels/incompressible/singlePhaseTransportModel \
    -I$(LIB_SRC)/dynamicMesh/lnInclude \
    -I$(LIB_SRC)/dynamicFvMesh/lnInclude
   
EXE_LIBS = \
    -lfiniteVolume \
    -lfvOptions \
    -lmeshTools \
    -lsampling \
    -lturbulenceModels \
    -lincompressibleTurbulenceModels \
    -lincompressibleTransportModels \
    -ldynamicMesh \
    -ldynamicFvMesh \
    -ltopoChangerFvMesh \
    -latmosphericModels
In the file files there is a setting for the place where the final executable binary will be created:
zeus-1:Make> cat files
myPimpleFoam.C
   
EXE = $(FOAM_USER_APPBIN)/myPimpleFoam

A.III The FOAM_USER_APPBIN, the WM_PROJECT_USER_DIR and other OpenFOAM environment variables

OpenFOAM makes use of several environmet variables.

Most of those variables start with FOAM_ and WM_.

The setting of those variables is usually performed by sourcing the bashrc file

For our images, that bashrc file is in /opt/OpenFOAM/OpenFOAM-v1912/etc

For our Singularity images, this file is sourced every time the container is ran (so no additional sourcing is needed)

(If executing the Docker image elsewhere, the you will indeed need to source the bashrc file)
To check a clean list of the variables (defined inside the container) you can use:
zeus-1:myPimpleFoam> module load singularity 
zeus-1:myPimpleFoam> theImage=/group/singularity/pawseyRepository/OpenFOAM/openfoam-v1912-pawsey.sif 
zeus-1:myPimpleFoam> singularity exec $theImage bash -c 'awk "BEGIN{for(v in ENVIRON) print v}" | egrep "FOAM_|WM_" | sort'
For anything related to OpenFOAM environmental variables, we recommend the use of the bash -c command

(See the explanation of the bash -c command at the end of this section if needed)
FOAM_API
FOAM_APP
FOAM_APPBIN
FOAM_ETC
FOAM_EXT_LIBBIN
FOAM_LIBBIN
FOAM_MPI
FOAM_RUN
FOAM_SETTINGS
FOAM_SITE_APPBIN
FOAM_SITE_LIBBIN
FOAM_SOLVERS
FOAM_SRC
FOAM_TUTORIALS
FOAM_USER_APPBIN
FOAM_USER_LIBBIN
FOAM_UTILITIES
WM_ARCH
WM_COMPILER
WM_COMPILER_LIB_ARCH
WM_COMPILER_TYPE
WM_COMPILE_OPTION
WM_DIR
WM_LABEL_OPTION
WM_LABEL_SIZE
WM_MPLIB
WM_NCOMPPROCS
WM_OPTIONS
WM_PRECISION_OPTION
WM_PROJECT
WM_PROJECT_DIR
WM_PROJECT_USER_DIR
WM_PROJECT_VERSION
WM_THIRD_PARTY_DIR
The main variables of interst here are WM_PROJECT_USER_DIR, FOAM_USER_APPBIN and FOAM_USER_LIBBIN:
zeus-1:myPimpleFoam> singularity exec $theImage bash -c 'printenv | grep "_USER_" | sort -r'
WM_PROJECT_USER_DIR=/home/ofuser/OpenFOAM/ofuser-v1912
FOAM_USER_LIBBIN=/home/ofuser/OpenFOAM/ofuser-v1912/platforms/linux64GccDPInt32Opt/lib
FOAM_USER_APPBIN=/home/ofuser/OpenFOAM/ofuser-v1912/platforms/linux64GccDPInt32Opt/bin
FOAM_SETTINGS=bash -c printenv | grep "_USER_" | sort -r
WM_PROJECT_USER_DIR is the base path where user’s stuff is stored

FOAM_USER_APPBIN is the place where the binary executables of user’s own solvers are stored and looked for

FOAM_USER_LIBBIN is the place where user’s own libraries are stored and looked for

As you can see, the APPBIN and LIBBIN paths are under the WM_PROJECT_USER_DIR path

Internal directories of the container are non-writable

But we’ll bind a local directory (with -B option) to the path of WM_PROJECT_USER_DIR to make things work

The bash -c command and the single quotes

The bash command creates a new bash kernel for execution. Together with the -c option, we can define a command to be executed inside that new bash kernel. The use of bash -c is very important and useful for the execution of OpenFOAM containers. You will notice that and lear its usage along the several episodes of this workshop.

For now, we can try to echo the content of the FOAM_TUTORIALS variable from the command line (non-interactively). First we exemplify some failed attempts and then the use of bash -c.

A first (ineffective) try could be:
zeus-1:*-v1912> theImage="/group/singularity/pawseyRepository/OpenFOAM/openfoam-v1912-pawsey.sif"
zeus-1:*-v1912> singularity exec $theImage echo $FOAM_TUTORIALS
echo is ran within the container.

But the command did not work because $FOAM_TUTORIALS is being evaluated in the host kernel before passing the arguments to the containerxi

And, in the host kernel, that variable does not exist:
zeus-1:*-v1912> echo $FOAM_TUTORIALS
A second (ineffective) try could be:
zeus-1:*-v1912> singularity exec $theImage echo '$FOAM_TUTORIALS'
$FOAM_TUTORIALS
Did not work because echo now understands to display the exact string but not the content of the variable.

A third (ineffective) try could be the use of bash -c with double quotes:
zeus-1:*-v1912> singularity exec $theImage bash -c "echo $FOAM_TUTORIALS"
Almost there, but it did not work because due to the double-apostrophes.

Yes, the echo command is being evaluated inside the new bash kernel created by the bash -c command (inside the container)

But the double apostrophes allow the evaluation of the variables inside the quote in the host kernel at the command line.

Then, $FOAM_TUTORIALS is being evaluated again in the host kernel (where it has no value) before passing the arguments to the container.

Finally, this one works (correct use of bash -c and single quotes):
zeus-1:*-v1912> singularity exec $theImage bash -c 'echo $FOAM_TUTORIALS'
/opt/OpenFOAM/OpenFOAM-v1912/tutorials
Yes, the echo command is being evaluated inside the new bash kernel created by the bash -c command (inside the container)

The exact string 'echo $FOAM_TUTORIALS' is passed as an argument without being evaluated by the host kernel.

The argument is then received correctly by the new bash kernel inside the container

As the variable exists inside the container, then the value can be displayed.

Note that without bash -c things do not work either:
zeus-1:*-v1912> singularity exec $theImage 'echo $FOAM_TUTORIALS'
/.singularity.d/actions/exec: line 21: exec: echo $FOAM_TUTORIALS: not found

B. Compilation of myPimpleFoam

The binding

As mentioned above, the trick is to bind a directory in the local host to the internal path where WM_PROJECT_USER_DIR point to

Inspect the script to check specific command syntax

The `B.compileMyPimpleFoam.sh` script

Main command in the script

projectUserDir=$SLURM_SUBMIT_DIR/projectUserDir
srun -n 1 -N 1 singularity exec -B $projectUserDir:/home/ofuser/OpenFOAM/ofuser-$theVersion $theImage wmake

Other important parts of the script:

#3. Going into the new solver directory and creating the logs directory
projectUserDir=$SLURM_SUBMIT_DIR/projectUserDir
solverNew=myPimpleFoam
if [ -d $projectUserDir/applications/$solverNew ]; then
   cd $projectUserDir/applications/$solverNew
   echo "pwd=$(pwd)"
else
   echo "For some reason, the directory $projectUserDir/applications/$solverNew, does not exist"
   echo "Exiting"; exit 1
fi
logsDir=./logs/compile
if ! [ -d $logsDir ]; then
   mkdir -p $logsDir
fi

#4. Use container's "wclean" to clean previously existing compilation 
echo "Cleaning previous compilation"
srun -n 1 -N 1 singularity exec -B $projectUserDir:/home/ofuser/OpenFOAM/ofuser-$theVersion $theImage wclean 2>&1 | tee $logsDir/wclean.$SLURM_JOBID

#5. Use container's "wmake" (and compiler) to compile your own tool
echo "Compiling myPimpleFoam"
srun -n 1 -N 1 singularity exec -B $projectUserDir:/home/ofuser/OpenFOAM/ofuser-$theVersion $theImage wmake 2>&1 | tee $logsDir/wmake.$SLURM_JOBID

#6. Very simple test of the new solver
echo "Performing a basic test"
singularity exec -B $projectUserDir:/home/ofuser/OpenFOAM/ofuser-$theVersion $theImage myPimpleFoam -help | tee $logsDir/myPimpleFoam.$SLURM_JOBID

B.I Steps for dealing with the compilation:

From the scripts directory, submit the compilation script (use the reservation for the workshop if available):
```
zeus-1:*-v1912> myReservation=containers 
zeus-1:*-v1912> sbatch --reservation=$myReservation B.compileMyPimpleFoam.sh 
```
If you do not have a reservation

Then, submit normally (or choose the best partition for executing the exercise, the debugq for example:)
```
   zeus-1:*-v1912> sbatch -p debugq B.compileMyPimpleFoam.sh
```
```
Submitted batch job 4632558
```

Check that the new solver binary is now under the projectUserDir/platforms tree

zeus-1:*-v1912> ls projectUserDir

applications   platforms

zeus-1:*-v1912> ls -la projectUserDir/platforms/linux64GccDPInt32Opt/bin

total 832
drwxrws---+ 2 espinosa pawsey0001   4096 May 24 10:59 .
drwxrws---+ 3 espinosa pawsey0001   4096 May 23 20:23 ..
-rwxrwx--x+ 1 espinosa pawsey0001 840664 May 24 10:59 myPimpleFoam

You can check the slurm-4632558.out or the log file created by tee in the job-script (use the SLURM_JOBID of your job):

zeus-1:*-v1912> cat projectUserDir/applications/myPimpleFoam/logs/compile/wmake.4632558 

Making dependency list for source file myPimpleFoam.C
g++ -std=c++11 -m64 -DOPENFOAM=1912 -DWM_DP -DWM_LABEL_SIZE=32 -Wall -Wextra -Wold-style-cast -Wnon-virtual-dtor -Wno-unused-parameter -Wno-invalid-offsetof -Wno-attributes -Wno-unknown-pragmas -O3  -DNoRepository -ftemplate-depth-100 -I/opt/OpenFOAM/OpenFOAM-v1912/src/finiteVolume/lnInclude -I/opt/OpenFOAM/OpenFOAM-v1912/src/meshTools/lnInclude -I/opt/OpenFOAM/OpenFOAM-v1912/src/sampling/lnInclude -I/opt/OpenFOAM/OpenFOAM-v1912/src/TurbulenceModels/turbulenceModels/lnInclude -I/opt/OpenFOAM/OpenFOAM-v1912/src/TurbulenceModels/incompressible/lnInclude -I/opt/OpenFOAM/OpenFOAM-v1912/src/transportModels -I/opt/OpenFOAM/OpenFOAM-v1912/src/transportModels/incompressible/singlePhaseTransportModel -I/opt/OpenFOAM/OpenFOAM-v1912/src/dynamicMesh/lnInclude -I/opt/OpenFOAM/OpenFOAM-v1912/src/dynamicFvMesh/lnInclude -IlnInclude -I. -I/opt/OpenFOAM/OpenFOAM-v1912/src/OpenFOAM/lnInclude -I/opt/OpenFOAM/OpenFOAM-v1912/src/OSspecific/POSIX/lnInclude   -fPIC -c myPimpleFoam.C -o Make/linux64GccDPInt32Opt/myPimpleFoam.o
g++ -std=c++11 -m64 -DOPENFOAM=1912 -DWM_DP -DWM_LABEL_SIZE=32 -Wall -Wextra -Wold-style-cast -Wnon-virtual-dtor -Wno-unused-parameter -Wno-invalid-offsetof -Wno-attributes -Wno-unknown-pragmas -O3  -DNoRepository -ftemplate-depth-100 -I/opt/OpenFOAM/OpenFOAM-v1912/src/finiteVolume/lnInclude -I/opt/OpenFOAM/OpenFOAM-v1912/src/meshTools/lnInclude -I/opt/OpenFOAM/OpenFOAM-v1912/src/sampling/lnInclude -I/opt/OpenFOAM/OpenFOAM-v1912/src/TurbulenceModels/turbulenceModels/lnInclude -I/opt/OpenFOAM/OpenFOAM-v1912/src/TurbulenceModels/incompressible/lnInclude -I/opt/OpenFOAM/OpenFOAM-v1912/src/transportModels -I/opt/OpenFOAM/OpenFOAM-v1912/src/transportModels/incompressible/singlePhaseTransportModel -I/opt/OpenFOAM/OpenFOAM-v1912/src/dynamicMesh/lnInclude -I/opt/OpenFOAM/OpenFOAM-v1912/src/dynamicFvMesh/lnInclude -IlnInclude -I. -I/opt/OpenFOAM/OpenFOAM-v1912/src/OpenFOAM/lnInclude -I/opt/OpenFOAM/OpenFOAM-v1912/src/OSspecific/POSIX/lnInclude   -fPIC -Xlinker --add-needed -Xlinker --no-as-needed Make/linux64GccDPInt32Opt/myPimpleFoam.o -L/opt/OpenFOAM/OpenFOAM-v1912/platforms/linux64GccDPInt32Opt/lib \
    -lfiniteVolume -lfvOptions -lmeshTools -lsampling -lturbulenceModels -lincompressibleTurbulenceModels -lincompressibleTransportModels -ldynamicMesh -ldynamicFvMesh -ltopoChangerFvMesh -latmosphericModels -lOpenFOAM -ldl  \
     -lm -o /home/ofuser/OpenFOAM/ofuser-v1912/platforms/linux64GccDPInt32Opt/bin/myPimpleFoam

You can check the log for the mini-test of the solver:

zeus-1:*-v1912> cat projectUserDir/applications/myPimpleFoam/logs/compile/myPimpleFoam.4632558 

Usage: myPimpleFoam [OPTIONS]
Options:
  -case <dir>       Specify case directory to use (instead of the cwd)
  -decomposeParDict <file>
                    Use specified file for decomposePar dictionary
  -dry-run          Check case set-up only using a single time step
  -dry-run-write    Check case set-up and write only using a single time step
  -parallel         Run in parallel
  -postProcess      Execute functionObjects only
  -doc              Display documentation in browser
  -help             Display short help and exit
  -help-full        Display full help and exit
   
Transient solver for incompressible, turbulent flow of Newtonian fluids on a
moving mesh.
   
Using: OpenFOAM-v1912 (1912) - visit www.openfoam.com
Build: _f3950763fe-20191219
Arch:  LSB;label=32;scalar=64

Or you can perform that basic mini-test from the command line (binding the local projectUserDir directory):

zeus-1:*-v1912> module load singularity
zeus-1:*-v1912> theImage=/group/singularity/pawseyRepository/OpenFOAM/openfoam-v1912-pawsey.sif
zeus-1:*-v1912> singularity exec -B ./projectUserDir:/home/ofuser/OpenFOAM/ofuser-v1912 $theImage myPimpleFoam -help 

Usage: myPimpleFoam [OPTIONS]
Options:
  -case <dir>       Specify case directory to use (instead of the cwd)
  -decomposeParDict <file>
                    Use specified file for decomposePar dictionary
  -dry-run          Check case set-up only using a single time step
  -dry-run-write    Check case set-up and write only using a single time step
  -parallel         Run in parallel
  -postProcess      Execute functionObjects only
  -doc              Display documentation in browser
  -help             Display short help and exit
  -help-full        Display full help and exit
   
Transient solver for incompressible, turbulent flow of Newtonian fluids on a
moving mesh.
   
Using: OpenFOAM-v1912 (1912) - visit www.openfoam.com
Build: _f3950763fe-20191219
Arch:  LSB;label=32;scalar=64

(If you do not use the binding, the solver will not be found):

zeus-1:*-v1912> singularity exec $theImage myPimpleFoam -help

/.singularity.d/actions/exec: line 21: exec: myPimpleFoam: not found

C. Extraction of the tutorial: channel395

The `C.extractTutorial.sh` script (main parts to be discussed):

#!/bin/bash -l
#SBATCH --export=NONE
#SBATCH --time=00:05:00
#SBATCH --ntasks=1
#SBATCH --partition=copyq #Ideally, you should be using the copyq for this kind of processes

#4. Copy the tutorialCase to the workingDir
if ! [ -d $caseDir ]; then
   srun -n 1 -N 1 singularity exec $theImage bash -c 'cp -r $FOAM_TUTORIALS/'"$tutorialCase $caseDir"
else
   echo "The case=$caseDir already exists, no new copy has been performed"
fi

C.I Steps for dealing with the extraction of the “channel395” case: - [Pre-Executed]
Submit the job (no need for reservation as the script uses the copyq partition)
zeus-1:*-v1912> sbatch C.extractTutorial.sh 
Submitted batch job 4632458
Check its status in the queue:
zeus-1:*-v1912> squeue -u $USER
JOBID    USER     ACCOUNT     PARTITION            NAME EXEC_HOST ST     REASON   START_TIME     END_TIME  TIME_LEFT NODES   PRIORITY
4632468  espinosa pawsey0001  workq      C.extractTutor       n/a PD       None          N/A          N/A       5:00     1      75190
If no status is shown, it may have finished execution already.
Check that the tutorial has been copied to our host file system
zeus-1:*-v1912> ls ./run/channel395/
0  0.orig  Allrun  constant  system

D. Adapt the case to your needs (and Pawsey best practices)

The `D.adaptCase.sh` script (main parts to be discussed):

#5. Defining OpenFOAM controlDict settings for Pawsey Best Practices
##5.1 Replacing writeFormat, runTimeModifiable and purgeRight settings
foam_writeFormat="binary"
sed -i 's,^writeFormat.*,writeFormat    '"$foam_writeFormat"';,' ./system/controlDict
foam_runTimeModifiable="false"
sed -i 's,^runTimeModifiable.*,runTimeModifiable    '"$foam_runTimeModifiable"';,' ./system/controlDict
foam_purgeWrite=10
sed -i 's,^purgeWrite.*,purgeWrite    '"$foam_purgeWrite"';,' ./system/controlDict

##5.2 Defining the use of collated fileHandler of output results 
echo "OptimisationSwitches" >> ./system/controlDict
echo "{" >> ./system/controlDict
echo "   fileHandler collated;" >> ./system/controlDict
echo "}" >> ./system/controlDict

D.I Steps for dealing with the adaptation of the case: - [Pre-Executed]
Submit the adaptation script
zeus-1:*-v1912> sbatch D.adaptCase.sh 
Submitted batch job 4632548
Check the adapted settings
zeus-1:*-v1912> cd run/channel395
zeus-1:channel395> ls
0  0.orig  Allrun  constant  system
Initial conditions are in the subdirectory 0

Mesh and fluid prperties definition are under the tree of the subdirectory constant

Solver settings are in the “dictionaries” inside the system subdirectory

Read the controlDict dictionary:
zeus-1:channel395> view system/controlDict
~
~
~
:q
The settings that were adapted in ..../run/channel395/system/controlDict
To keep only a few result directories at a time (10 maximum in this case)
 purgeWrite      10;
To use binary writing format to accelerate writing and reduce the size of the files
 writeFormat     binary;
Never use runTimeModifiable. This option creates permanent reading of dictionaries (each time step) which overloads the shared file system.
 runTimeModifiable false;
If version is higher-or-equal than OpenFOAM-6 or OpenFOAM-v1812, always use the collated option
 optimisationSwitches
 {
     fileHandler collated;
 }

E. Decomposition

The `E.decomposeFoam.sh` script (main parts to be discussed):

#!/bin/bash -l
#SBATCH --ntasks=1
#SBATCH --mem=4G
#SBATCH --ntasks-per-node=28
#SBATCH --clusters=zeus
#SBATCH --partition=workq
#SBATCH --time=0:10:00
#SBATCH --export=none

#6. Defining the ioRanks for collating I/O
# groups of 2 for this exercise (please read our documentation for the recommendations for production runs)
export FOAM_IORANKS='(0 2 4 6)'

#7. Perform all preprocessing OpenFOAM steps up to decomposition
echo "Executing blockMesh"
srun -n 1 -N 1 singularity exec $theImage blockMesh 2>&1 | tee $logsDir/log.blockMesh.$SLURM_JOBID
echo "Executing decomposePar"
srun -n 1 -N 1 singularity exec $theImage decomposePar -cellDist -force 2>&1 | tee $logsDir/log.decomposePar.$SLURM_JOBID

E.I Steps for dealing with decomposition:
Submit the decomposition script from the scripts directory (use the reservation for the workshop if available)
zeus-1:*-v1912> myReservation=containers
zeus-1:*-v1912> sbatch --reservation=$myReservation E.decomposeFoam.sh 
If a reservation is not available, do not use the option. (Or you can use the debugq: --partition=debugq instead.)
Submitted batch job 4632558
Check that the decomposition has been performed:
zeus-1:*-v1912> ls ./run/channel395/processor*
./run/channel395/processors4_0-1:
0  constant
   
./run/channel395/processors4_2-3:
0  constant
You should also check for success/errors in:

the slurm output file: slurm-<SLURM_JOBID>.out

the log files created when executing the OpenFOAM tools in: ./run/channel395/logs/pre/

F. Executing the NEW solver “myPimpleFoam”

The binding

Again, the important concept here is the binding of the local folder for the container to be able to read the binary executable

And, obviously, to call the own solver: myPimpleFoam to perform the solution

Check the interior of the scripts for especific command syntax

The `F.runFoam.sh` script

Main command in the script:

theSolver=myPimpleFoam
projectUserDir=$SLURM_SUBMIT_DIR/projectUserDir
srun -n $SLURM_NTASKS -N $SLURM_JOB_NUM_NODES singularity exec -B $projectUserDir:/home/ofuser/OpenFOAM/ofuser-$theVersion $theImage $theSolver -parallel 2>&1

Other important parts of the script:

#SBATCH --ntasks=4
#SBATCH --mem=16G
#SBATCH --ntasks-per-node=28
#SBATCH --cluster=zeus

#5. Reading OpenFOAM decomposeParDict settings
foam_numberOfSubdomains=$(grep "^numberOfSubdomains" ./system/decomposeParDict | tr -dc '0-9')

#7. Checking if the number of tasks coincide with the number of subdomains
if [[ $foam_numberOfSubdomains -ne $SLURM_NTASKS ]]; then
   echo "foam_numberOfSubdomains read from ./system/decomposeParDict is $foam_numberOfSubdomains"
   echo "and"
   echo "SLURM_NTASKS in this job is $SLURM_NTASKS"
   echo "These should be the same"
   echo "Therefore, exiting this job"
   echo "Exiting"; exit 1
fi

#8. Defining OpenFOAM controlDict settings for this run
foam_startFrom=startTime
#foam_startFrom=latestTime
foam_startTime=0
#foam_startTime=15
foam_endTime=10
#foam_endTime=30
foam_writeInterval=1
foam_purgeWrite=10

#9. Changing OpenFOAM controlDict settings
sed -i 's,^startFrom.*,startFrom    '"$foam_startFrom"';,' system/controlDict
sed -i 's,^startTime.*,startTime    '"$foam_startTime"';,' system/controlDict
sed -i 's,^endTime.*,endTime    '"$foam_endTime"';,' system/controlDict
sed -i 's,^writeInterval.*,writeInterval    '"$foam_writeInterval"';,' system/controlDict
sed -i 's,^purgeWrite.*,purgeWrite    '"$foam_purgeWrite"';,' system/controlDict

#10. Defining the solver
theSolver=myPimpleFoam

#11. Defining the projectUserDir to be mounted into the path of the internal WM_PROJECT_USER_DIR
projectUserDir=$SLURM_SUBMIT_DIR/projectUserDir

#12. Execute the case 
echo "About to execute the case"
srun -n $SLURM_NTASKS -N $SLURM_JOB_NUM_NODES singularity exec -B $projectUserDir:/home/ofuser/OpenFOAM/ofuser-$theVersion $theImage $theSolver -parallel 2>&1 | tee $logsDir/log.$theSolver.$SLURM_JOBID
echo "Execution finished"

F.I Steps for dealing with the solver

Submit the solver script (from the scripts directory)
```
zeus-1:*-v1912> sbatch --reservation=$myReservation F.runFoam.sh 
```
- If a reservation is not available, do not use the option. (Or you can use the debugq: --partition=debugq instead.)
```
Submitted batch job 4632685 on cluster zeus
```

Check that the solver is running:

zeus-1:*-v1912> squeue -u $USER

JOBID    USER     ACCOUNT     PARTITION            NAME EXEC_HOST ST     REASON   START_TIME     END_TIME  TIME_LEFT NODES   PRIORITY
4632685  espinosa pawsey0001  workq        F.runFoam.sh       n/a PD  Resources     17:09:28     17:19:28      10:00     1      75190 

Observe the output of the job with tail -f at runtime (use <Ctrl-C> to exit the command):

zeus-1:*-v1912> tail -f slurm-4632685.out

.
.
.
Time = 0.2
   
PIMPLE: iteration 1
smoothSolver:  Solving for Ux, Initial residual = 0.0118746, Final residual = 1.89249e-06, No Iterations 3
smoothSolver:  Solving for Uy, Initial residual = 0.0617212, Final residual = 1.68113e-06, No Iterations 4
smoothSolver:  Solving for Uz, Initial residual = 0.0589944, Final residual = 9.70923e-06, No Iterations 3
Pressure gradient source: uncorrected Ubar = 0.13369, pressure gradient = -0.000964871
GAMG:  Solving for p, Initial residual = 0.213844, Final residual = 0.00414884, No Iterations 2
time step continuity errors : sum local = 5.82807e-06, global = -1.41211e-19, cumulative = -1.41211e-19
Pressure gradient source: uncorrected Ubar = 0.133687, pressure gradient = -0.000947989
GAMG:  Solving for p, Initial residual = 0.0222643, Final residual = 4.30412e-07, No Iterations 7
time step continuity errors : sum local = 5.63638e-10, global = -2.40486e-19, cumulative = -3.81697e-19
Pressure gradient source: uncorrected Ubar = 0.133687, pressure gradient = -0.000947874
ExecutionTime = 0.25 s  ClockTime = 0 s
.
.
.

Press <Ctrl-C> to exit tail

Check that the solver gave some results:

zeus-1:*-v1912> ls ./run/channel395/processor*

./run/channel395/processors4_0-1:
0  10  8.2  8.4  8.6  8.8  9  9.2  9.4  9.6  9.8  constant
   
./run/channel395/processors4_2-3:
0  10  8.2  8.4  8.6  8.8  9  9.2  9.4  9.6  9.8  constant

You should also check for success/errors in:
- the slurm output file: slurm-<SLURM_JOBID>.out
- the log files created when executing the OpenFOAM tools in: ./run/channel395/logs/run/

Z. Further notes on how to use OpenFOAM and OpenFOAM containers at Pawsey

The usage of OpenFOAM and OpenFOAM containers at Pawsey has already been described in our documentation: OpenFOAM documentation at Pawsey

and in a technical newsletter note: https://support.pawsey.org.au/documentation/display/US/Pawsey+Technical+Newsletter+2020-04

Key Points

Define a host directory that will play the role of WM_PROJECT_USER_DIR

For example, projectUserDir=./anyDirectory

Then bind that directory to the path defined inside for WM_PROJECT_USER_DIR

For this exercise, singularity exec -B $projectUserDir:/home/ofuser/OpenFOAM/ofuser-v1912 $theImage <mySolver> <myOptions>

Building an OpenFOAM container with MPICH

Overview

Teaching: 30 min
Exercises: 30 min

Questions

How can I build my own OpenFOAM container with MPICH?

Objectives

Explain our current recommended process for building OpenFOAM containers

0. Introduction

Why you need to build your own container?

If the version of OpenFOAM that you need to use is not maintained by Pawsey

If you want to move your OpenFOAM “installation” easily to another system

If you want to use the OverlayFS approach to reduce the amount of reduced files (to be explained in the following episode)

Why MPICH?

To achieve the best performance in a supercomputer, we want to run in hybrid mode with the optimised host MPI

In the case of Magnus, Cray-MPI (ABI compatible to MPICH) is the only supported flavour

Therefore, we decided to support containerised MPI applications compiled with MPICH

MPICH containers also run properly on Zeus

Main drawback:

Developers’ OpenFOAM containers equipped with OpenMPI would not run properly on Magnus

OpenFOAM containers need to be built from scratch with MPICH

A. Have ready a Linux environment

If you count with a linux environment with Docker, Singularity and Git installed in your local computer, then you are ready to go to section B.
If not, for the workshops we provide a Nimbus Virtual Machines attendees:

A.I Steps for connecting to a Nimbus Virtual Machine for a live workshop:

Follow these steps to connect to your Nimbus virtual machine:

Save the ssh-key provided by the instructors in a known directory, for example $HOME/pawseyTraining:

yourLocalComputer$ cd $HOME/pawseyTraining
yourLocalComputer$ ls

cfd_key.gz

yourLocalComputer$ gunzip cfd_cfd2020.gz

yourLocalComputer$ ls -lat

drwxr-xr-x   4 esp025  515598196   128 26 May 08:11 .
drwxr-xr-x+ 75 esp025  515598196  2400 26 May 08:08 ..
-rw-r--r--   1 esp025  515598196  3434 22 May 11:26 rsa_cfd2020

The point of the process is for you to count with the file: rsa_cfd2020 in a known location

Make sure that the ssh-key has read-write permissions only for you:

yourLocalComputer$ chmod 600 rsa_cfd2020
yourLocalComputer$ ls -lat

total 24
drwxr-xr-x   4 esp025  515598196   128 26 May 08:11 .
drwxr-xr-x+ 75 esp025  515598196  2400 26 May 08:08 ..
-rw-------   1 esp025  515598196  3434 22 May 11:26 rsa_cfd2020

Connect to the VM using that ssh-key:

Choose an IP from the list of available VMs given by the instructors.
The chosen IP address should be something like 123.123.123.123
Now, use the ssh-key to connect to the VM with that IP address (username is ubuntu):

yourLocalComputer$ ssh -i rsa_cfd2020 ubuntu@123.123.123.123

Enter passphrase for key 'rsa_cfd2020':

Welcome to Ubuntu 18.04.4 LTS (GNU/Linux 5.3.0-53-generic x86_64)
   
 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage
   
  System information as of Tue May 26 00:31:58 UTC 2020
   
  System load:  0.0                Processes:              102
  Usage of /:   33.9% of 77.36GB   Users logged in:        0
  Memory usage: 2%                 IP address for ens3:    192.168.1.17
  Swap usage:   0%                 IP address for docker0: 172.17.0.1
   
 * MicroK8s passes 9 million downloads. Thank you to all our contributors!
   
     https://microk8s.io/
   
0 packages can be updated.
0 updates are security updates.
   
Your Hardware Enablement Stack (HWE) is supported until April 2023.
   
Last login: Mon May 25 23:18:39 2020 from 27.33.35.111
ubuntu@vm:~$ 

B. Clone the Git repository for the workshop exercises

You’ll need to clone the git repository into your linux environment:

B.I Steps for cloning the Git repository

Create a directory for the training and clone the repository

ubuntu@vm:~$ mkdir pawseyTraining
ubuntu@vm:~$ cd pawseyTraining
ubuntu@vm:pawseyTraining$ git clone https://github.com/PawseySC/containers-openfoam-workshop-scripts.git

Cloning into 'containers-openfoam-workshop-scripts'...
remote: Enumerating objects: 30, done.
remote: Counting objects: 100% (30/30), done.
remote: Compressing objects: 100% (16/16), done.
remote: Total 30 (delta 10), reused 30 (delta 10), pack-reused 0
Unpacking objects: 100% (30/30), done.

ubuntu@vm:pawseyTraining$ ls

containers-openfoam-workshop-scripts

C. Building a first Docker image with MPICH

General advice:

As mentioned in the previous seminars, builiding containers of large applications will need some trial-and-error process

The building of an OpenFOAM image needs trial-and-error. (Installations of OpenFOAM are sometimes very time consuming)

Therefore, we recommended to build the image first with Docker and convert it to Singularity afterwards

This also allows more portability

C.I Steps for building our first image based on pawsey/mpich-base
cd into the exercise directory:
ubuntu@vm:pawseyTraining$ cd containers-openfoam-workshop-scripts/04_buildingAnOpenFOAMContainer/openfoam-2.4.x 
ubuntu@vm:openfoam-2.4.x$ ls 
01_Docker  02_PortingToSingularity
The 01_Docker directory contains the definition files for building the Docker image

The 02_PortingToSingularity directory contains the definition file for building the Singularity image
ubuntu@vm:openfoam-2.4.x$ cd 01_Docker
ubuntu@vm:01_Docker$ ls
Dockerfile.01  Dockerfile.02
If you wish, shrink your prompt, execute this:
PS1='${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\W\[\033[00m\]\$ '
a shrinked prompt $ 
Let’s take a look into the first Dockerfile (Dockerfile.01):
Main sections of the Dockerfile.01 to discuss:
# 0. Initial main definintion
# Defining the base container to build from
# In this case: mpich 3.1.4 and ubuntu 16.04; MPICH is needed for crays
FROM pawsey/mpich-base:3.1.4_ubuntu16.04

LABEL maintainer="Alexis.Espinosa@pawsey.org.au"
#OpenFOAM version to install
ARG OFVERSION="2.4.x"
#Using bash from now on
SHELL ["/bin/bash", "-c"]
Will build from the mpich-base image with MPICH-3.1.4 on Ubuntu-16.04

The “docker-variable” (argument) OFVERSION holds the version of OpenFOAM to be built
# I. Installing additional tools useful for interactive sessions
RUN apt-get update -qq\
 &&  apt-get -y --no-install-recommends install \
            vim time\
            cron gosu \
            bc \
 && apt-get clean all \
 && rm -r /var/lib/apt/lists/*
Using apt-get to install some auxiliary tools
# III. INSTALLING OPENFOAM.
#This section is for installing OpenFOAM
#Will follow PARTIALLY the official installation instructions:
#https://openfoam.org/download/2-4-0-source/
#
#Will follow PARTIALLY the instructions for OpenFOAM-2.4.x  available in the wiki:
#https://openfoamwiki.net/index.php/Installation/Linux/OpenFOAM-2.4.x/Ubuntu#Ubuntu_16.04
#
#Then, Will follow a combination of both
#(Also checking wiki for OpenFOAM-2.4.0)
#
#Where recipe deviates from the instructions mentioned above, comments from the maintainer are labelled as: AEG

#...........
#Definition of the installation directory within the container
ARG OFINSTDIR=/opt/OpenFOAM
ARG OFUSERDIR=/home/ofuser/OpenFOAM
WORKDIR $OFINSTDIR
Beginning of the main section for the OpenFOAM installation

Here we keep some comments about the guides used for defining this recipe

In this case: https://openfoam.org/download/2-4-0-source/

and https://openfoamwiki.net/index.php/Installation/Linux/OpenFOAM-2.4.x/Ubuntu#Ubuntu_16.04

Differences from the guides are noted by the maintainer with comments starting with AEG

We are also defining the path where the installation will reside (OFINSTDIR)

And the path where theoretical user stuff will reside (OFUSERDIR)

WORKDIR indicates the builder to create that directory and to move into it
#...........
#Step 1.
#Install necessary packages
#
RUN apt-get update -qq\
 &&  apt-get -y --no-install-recommends --no-install-suggests install \
   build-essential\
   flex bison git-core cmake zlib1g-dev \
   libboost-system-dev libboost-thread-dev \
#AEG:No OpenMPI because MPICH will be used (installed in the parent FROM container)
#AEG:NoOpenMPI:   libopenmpi-dev openmpi-bin \
#AEG:(using libncurses-dev, as in official instructions, and not libncurses5-dev, as in wiki)
   gnuplot libreadline-dev libncurses-dev libxt-dev \
   qt4-dev-tools libqt4-dev libqt4-opengl-dev \ 
   freeglut3-dev libqtwebkit-dev \
#AEG:No scotch because it installs openmpi which later messes up with MPICH
#    Therefore, ThirdParty scotch is the one to be installed and used by openfoam.
#AEG:NoScotch:   libscotch-dev \
   libcgal-dev \
#AEG:These libraries are needed for CGAL (system and third party) (if needed, change libgmp-dev for libgmp3-dev):
   libgmp-dev libmpfr-dev\
#AEG: Some more suggestions from the wiki instructions:
   python python-dev \
   libglu1-mesa-dev \
#AEG:I found the following was needed to install  FlexLexer.h
   libfl-dev \
 && apt-get clean all \
 && rm -r /var/lib/apt/lists/*
Here we install the auxiliary packages for OpenFOAM listed in the instructions guides

With some differences (check comments by AEG)

Like: OpenMPI is NOT installed

In your exercise, this section may appear disabled/commented (this is just to reduce time for the exercise, but in the real Dockerfile it is fully functional)
#...........
#Step 2. Download
#Change to the installation dir, clone OpenFOAM directories
ARG OFVERSIONGIT=$OFVERSION
WORKDIR $OFINSTDIR

RUN git clone git://github.com/OpenFOAM/OpenFOAM-${OFVERSIONGIT}.git \
 && git clone git://github.com/OpenFOAM/ThirdParty-${OFVERSIONGIT}.git
Cloning the source directories for OpenFOAM installation
Now lets use the first Dockerfile to build a first Docker container (not the final main installation)
ubuntu@vm:01_Docker$ sudo docker build -f Dockerfile.01 -t myuser/openfoam:2.4.x.01 .
With the -f option you can choose a specific Dockerfile

Important: Do not forget the dot . (which means: current directory) at the end

Note the tag used: 2.4.x.01 as we know this is a first test and not the final image

Important: The myuser name is important. This should be modified to correspond to your Dockerhub account. And will be used when the final version of the image is ready to be pushed it into DockerHub
Sending build context to Docker daemon   21.5kB^M^M
Step 1/15 : FROM pawsey/mpich-base:3.1.4_ubuntu16.04
 ---> b2cb97823381
Step 2/15 : LABEL maintainer="Alexis.Espinosa@pawsey.org.au"
 ---> Running in a4f596b9337f
Removing intermediate container a4f596b9337f
 ---> a58d84a78040
Step 3/15 : ARG OFVERSION="2.4.x"
 ---> Running in 453210a478c6
Removing intermediate container 453210a478c6
 ---> 6d3d0e059b3a
Step 4/15 : SHELL ["/bin/bash", "-c"]
.
.
.
Step 13/15 : ARG OFVERSIONGIT=$OFVERSION
 ---> Running in 0f20db6d29f3
Removing intermediate container 0f20db6d29f3
 ---> 46ee1dca0673
Step 14/15 : WORKDIR $OFINSTDIR
 ---> Running in 40ecd1e76151
Removing intermediate container 40ecd1e76151
 ---> 0432945652d9
Step 15/15 : RUN git clone git://github.com/OpenFOAM/OpenFOAM-${OFVERSIONGIT}.git  && git clone git://github.com/OpenFOAM/ThirdParty-${OFVERSIONGIT}.git
 ---> Running in 5fdd676ad176
^[[91mCloning into 'OpenFOAM-2.4.x'...
^[[0m^[[91mCloning into 'ThirdParty-2.4.x'...
^[[0mRemoving intermediate container 5fdd676ad176
 ---> bdbe2e3bac9e
Successfully built bdbe2e3bac9e
Successfully tagged myuser/openfoam:2.4.x.01
Check that the image exists:
ubuntu@vm:01_Docker$ docker image ls
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
myuser/openfoam     2.4.x.01            bdbe2e3bac9e        49 minutes ago      866MB
pawsey/openfoam     2.4.x               4917982ed825        2 weeks ago         7.42GB
pawsey/openfoam     v1912               c4360b0a0f3f        2 weeks ago         11.9GB
pawsey/mpich-base   3.1.4_ubuntu16.04   b2cb97823381        3 weeks ago         548MB
Our image is now first in the list

(For those using the virtual machines provided by Pawsey to the atendees, note that some pawsey images were already pulled before hand)

D. More on the Dockerfile

Non-default settings in the installation are:

For the OpenFOAM installation to understand the use of the system-MPI (MPICH in this case), we need to modify the default settings

We also recommend to modify the path for the OpenFOAM installation

We also recommend the path where the user’s own solvers and libraries will be stored (WM_PROJECT_USER_DIR)

These three modifications are set in the files bashrc and prefs.sh

The `Dockerfile.02` file (main parts for discussion)

#...........
#Step 3. Definitions for the prefs and bashrc files.
ARG OFPREFS=${OFINSTDIR}/OpenFOAM-${OFVERSION}/etc/prefs.sh
ARG OFBASHRC=${OFINSTDIR}/OpenFOAM-${OFVERSION}/etc/bashrc

We’ll make use of these variables (arguments) to refer to the prefs.sh and bashrc OpenFOAM settings files

#...........
#Defining the prefs.sh:
RUN head -23 ${OFINSTDIR}/OpenFOAM-${OFVERSION}/etc/config/example/prefs.sh > $OFPREFS \
 && echo '#------------------------------------------------------------------------------' >> ${OFPREFS} \
#Using a combination of the variable definition recommended for the use of system mpich in this link:
#   https://bugs.openfoam.org/view.php?id=1167
#And in the file .../OpenFOAM-$OFVERSION/wmake/rules/General/mplibMPICH
#(These MPI_* environmental variables are set in the prefs.sh,
# and this file will be sourced automatically by the bashrc when the bashrc is sourced)
#
#--As suggested in the link above and mplibMPICH file:
 && echo 'export WM_MPLIB=SYSTEMMPI' >> ${OFPREFS} \
 && echo 'export MPI_ROOT="/usr"' >> ${OFPREFS} \
 && echo 'export MPI_ARCH_FLAGS="-DMPICH_SKIP_MPICXX"' >> ${OFPREFS} \
 && echo 'export MPI_ARCH_INC="-I${MPI_ROOT}/include"' >> ${OFPREFS} \
 && echo 'export MPI_ARCH_LIBS="-L${MPI_ROOT}/lib${WM_COMPILER_LIB_ARCH} -L${MPI_ROOT}/lib -lmpich -lrt"' >> ${OFPREFS} \
 && echo ''

These are the settings for the use of the system installed MPICH
Using echo command to write the settings into the prefs.sh file

#...........
#Modifying the bashrc file
RUN cp ${OFBASHRC} ${OFBASHRC}.original \
#Changing the installation directory within the bashrc file (This is not in the openfoamwiki instructions)
 && sed -i '/^foamInstall=$HOME.*/afoamInstall='"${OFINSTDIR}" ${OFBASHRC} \
 && sed -i '0,/^foamInstall=$HOME/s//# foamInstall=$HOME/' ${OFBASHRC} \
#Changing the place for your own tools/solvers (WM_PROJECT_USER_DIR directory) within the bashrc file 
 && sed -i '/^export WM_PROJECT_USER_DIR=.*/aexport WM_PROJECT_USER_DIR="'"${OFUSERDIR}/ofuser"'-$WM_PROJECT_VERSION"' ${OFBASHRC} \
 && sed -i '0,/^export WM_PROJECT_USER_DIR/s//# export WM_PROJECT_USER_DIR/' ${OFBASHRC} \
 && echo ''

These are the settings for the installation path and the WM_PROJECT_USER_DIR path
Using sed command to replace settings in the bashrc file
The syntax shown here adds the new setting immediately below the original setting
Then comments out the line of the original setting
Note also that the original file is first copied into bashrc.original

The RUN StopHere trick in the Dockerfile.02:
#...........
# REMOVE THIS AFTER TESTING
RUN StopHere #Trick for stopping the recipe at this point
#...........
We recommend the use of this trick to stop the building process at some point and check the status of the installation up to that point

We like to use this trick instead of commenting the rest of the Dockerfile

Note that the building process will end with an error, but previous commands are cached in layers

We can still access any succesful cached layer before the error and use it as an image

D.I Steps for the second build:

Build the docker container using Dockerfile.02

ubuntu@vm:01_Docker$ sudo docker build -f Dockerfile.02 -t myuser/openfoam:2.4.x.02 .

Sending build context to Docker daemon     64kB^M^M
Step 1/39 : FROM pawsey/mpich-base:3.1.4_ubuntu16.04
 ---> b2cb97823381
Step 2/39 : LABEL maintainer="Alexis.Espinosa@pawsey.org.au"
 ---> Using cache
 ---> a58d84a78040
Step 3/39 : ARG OFVERSION="2.4.x"
 ---> Using cache
 ---> 6d3d0e059b3a
Step 4/39 : SHELL ["/bin/bash", "-c"]
 ---> Using cache
 ---> 9812983b6344
.
.
.
Removing intermediate container 5e181146e2bd
 ---> 46e4f0fcea46
Step 19/39 : RUN cp ${OFBASHRC} ${OFBASHRC}.original  && sed -i '/^foamInstall=$HOME.*/afoamInstall='"${OFINSTDIR}" ${OFBASHRC}  && sed -i '0,/^foamInstall=$HOME/s//# foamInstall=$HOME/' ${OFBASHRC}  && sed -i '/^export WM_PROJECT_USER_DIR=.*/aexport WM_PROJECT_USER_DIR="'"${OFUSERDIR}/ofuser"'-$WM_PROJECT_VERSION"' ${OFBASHRC}  && sed -i '0,/^export WM_PROJECT_USER_DIR/s//# export WM_PROJECT_USER_DIR/' ${OFBASHRC}  && echo ''
 ---> Running in 569bea603d57

Removing intermediate container 569bea603d57
 ---> e4ae93bc92f1
Step 20/39 : RUN StopHere #Trick for stopping the recipe at this point
 ---> Running in 35a6be1d6690
^[[91m/bin/bash: StopHere: command not found
^[[0mThe command '/bin/bash -c StopHere #Trick for stopping the recipe at this point' returned a non-zero code: 127

Everything went fine, except for Step 20/39 : RUN StopHere
Exactly above that line there is a hexadecimal number, in this case: ---> e4ae93bc92f1
That number is the “ID” of the cache layer before the error, and we can access it as an image

Use the ID of the latest cached layer (copy/paste) for running that layer as an image interactively.
```
ubuntu@vm:01_Docker$ docker run -it --rm e4ae93bc92f1
```
- The use of an interactive session for this kind of checks is very practical
- The -it indicates interactive
- The --rm indicates docker to remove the running session (container) from the docker engine after exiting
- (After exiting --rm removes the container, but not the image. The image is safe)
```
root@202a0bf870bb:/opt/OpenFOAM#
```
- We are now in an interactive session inside the container

Inside the container, check the contents of the prefs.sh file:

root@202a0bf870bb:/opt/OpenFOAM# pwd

/opt/OpenFOAM

root@202a0bf870bb:/opt/OpenFOAM# ls

OpenFOAM-2.4.x  ThirdParty-2.4.x

root@202a0bf870bb:/opt/OpenFOAM# cd OpenFOAM-2.4.x/etc
root@202a0bf870bb:/opt/OpenFOAM/OpenFOAM-2.4.x/etc# cat prefs.sh

#----------------------------------*-sh-*--------------------------------------
# =========                 |
# \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox
#  \\    /   O peration     |
#   \\  /    A nd           | Copyright (C) 2011-2013 OpenFOAM Foundation
#    \\/     M anipulation  |
#------------------------------------------------------------------------------
# License
#     This file is part of OpenFOAM.
#
#     OpenFOAM is free software: you can redistribute it and/or modify it
#     under the terms of the GNU General Public License as published by
#     the Free Software Foundation, either version 3 of the License, or
#     (at your option) any later version.
#
#     OpenFOAM is distributed in the hope that it will be useful, but WITHOUT
#     ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
#     FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
#     for more details.
#
#     You should have received a copy of the GNU General Public License
#     along with OpenFOAM.  If not, see <http://www.gnu.org/licenses/>.
#
#------------------------------------------------------------------------------
export WM_MPLIB=SYSTEMMPI
export MPI_ROOT="/usr"
export MPI_ARCH_FLAGS="-DMPICH_SKIP_MPICXX"
export MPI_ARCH_INC="-I${MPI_ROOT}/include"
export MPI_ARCH_LIBS="-L${MPI_ROOT}/lib${WM_COMPILER_LIB_ARCH} -L${MPI_ROOT}/lib -lmpich -lrt"

Settings are as expected

Also check the modifications inside the bashrc file

root@202a0bf870bb:/opt/OpenFOAM/OpenFOAM-2.4.x/etc# grep "foamInstall" bashrc

# 'foamInstall' below to where OpenFOAM is installed
# foamInstall=$HOME/$WM_PROJECT
foamInstall=/opt/OpenFOAM
# foamInstall=~$WM_PROJECT
# foamInstall=/opt/$WM_PROJECT
# foamInstall=/usr/local/$WM_PROJECT
: ${FOAM_INST_DIR:=$foamInstall}; export FOAM_INST_DIR
unset cleaned foamClean foamInstall foamOldDirs

The only real active definition of foamInstall is correct, the rest are commented

root@202a0bf870bb:/opt/OpenFOAM/OpenFOAM-2.4.x/etc# grep "PROJECT_USER_DIR" bashrc

# export WM_PROJECT_USER_DIR=$HOME/$WM_PROJECT/$USER-$WM_PROJECT_VERSION
export WM_PROJECT_USER_DIR="/home/ofuser/OpenFOAM/ofuser-$WM_PROJECT_VERSION"

The setting of WM_PROJECT_USER_DIR is correct

Exit the interactive session

root@202a0bf870bb:/opt/OpenFOAM/OpenFOAM-2.4.x/etc# exit

E. Final remarks on the building of OpenFOAM containers at Pawsey with Docker

General recommendations:

Pawsey maintains several images that can be used for running applications or for building new ones

The official maintained Dockerfile for OpenFOAM-2.4.x is in the PawseySC GitHUB: https://github.com/PawseySC/pawsey-containers

The DockerHub repository (where the corresponding Docker images are) is: https://hub.docker.com/u/pawsey

Use pawsey Dockerfiles as examples to build your own container

Then write your own Dockerfile and modify it following the instructions from the Developers and from the OpenFOAM wiki: https://openfoamwiki.net/index.php/Category:Installing_OpenFOAM_on_Linux)

The process may need some trial and error until you reach the correct recipe (the use of the StopHere trick shown in the previous exercise may help to debug the process and save some time)

After reaching a final correct Dockerfile recipe you can then build your Docker image and convert it to Singularity in a second stage

Compilation sections in the Dockerfile and the bashrc file
#Third party compilation
RUN . ${OFBASHRC} \
 && cd $WM_THIRD_PARTY_DIR \
 && ./Allwmake 2>&1 | tee log.Allwmake
#OpenFOAM compilation 
ENV WM_NCOMPPROCS=4
RUN . ${OFBASHRC} \
 && cd $WM_PROJECT_DIR \
 && export QT_SELECT=qt4 \
 && ./Allwmake 2>&1 | tee log.Allwmake
In each RUN command, source the bashrc file (in this case using the argument OFBASHRC) before compilation steps in a layer

This is because environment settings are lost after each RUN command

The only way to keep the environment variables is to apply a ENV command for each of the OpenFOAM variables, which is a very tedious process and we do not recommend it

Instead, when executing the Docker container, the bashrc file will need to be sourced to set the OpenFOAM environment

Nevertheless, we are recommending to use Singularity to run your OpenFOAM containers, so it is possible that you will not be using the Docker image except for testing

Fortunately, when builiding the Singularity image, there is an easy way to indicate to source the bashrc file every time the container is executed

Then, for Singularity containers, there will be no need to manually source the bashrc file

E.I Final steps in the creation of the full Docker container
The final building step (after reaching the right Dockerfile recipe) would be:
ubuntu@vm:01_Docker$ sudo docker build -t myuser/openfoam:2.4.x .
Without the -f option, then a file named Dockerfile will be used by default as the recipe

(This command will not work in the current state of the exercise) because that file has not been created yet

But still we are presenting the instruction here for completeness

To really build the full image you would need to create the correct Dockerfile (the provided file *.02 was produced from the official recipe, but it has been modified by commenting some lines (marked with #commentedForExercise# string) and the additional line with RUN StopHere

To create your own correct Dockerfile you can remove the comments and the StopHere line, or clone the git repository mentioned a few paragraphs above and take the correct recipe from there

The full building process of the openfoam:2.4.x takes around 6 hours
When using the correct Dockerfile the output will like like this:
Sending build context to Docker daemon  16.38kB^M^M
Step 1/38 : FROM pawsey/mpich-base:3.1.4_ubuntu16.04
 ---> b2cb97823381
Step 2/38 : LABEL maintainer="Alexis.Espinosa@pawsey.org.au"
 ---> Using cache
 ---> 2d15a7d6c656
Step 3/38 : ARG OFVERSION="2.4.x"
 ---> Running in 0126259b03e1
.
.
.
Removing intermediate container 7a3762b35d26
 ---> 9e966f97d1fb
Step 37/38 : USER ofuser
 ---> Running in 11bbf508ba96
Removing intermediate container 11bbf508ba96
 ---> 16869a33968e
Step 38/38 : WORKDIR /home/ofuser
 ---> Running in 121eb0c2151e
Removing intermediate container 121eb0c2151e
 ---> 4917982ed825
Successfully built 4917982ed825
Successfully tagged myuser/openfoam:2.4.x
The name myuser is important and should be modified to match your username in DockerHub

This because it will indicate the owner when pushing the container image
Then we test that the internal OpenFOAM installation works properly

This can easily be checked with an interactive session

(Here we perform the test with the pawsey/openfoam:2.4.x image because the myuser/openfoam:2.4.x image does not exist yet):
ubuntu@vm:01_Docker$ docker run -it --rm pawsey/openfoam:2.4.x
ofuser@49146943821c:~$ source /opt/OpenFOAM/OpenFOAM-2.4.x/etc/bashrc
This will also show the way to run the Docker container in an interactive session

To set the OpenFOAM environment in our container, the user needs to source manually the bashrc file
ofuser@49146943821c:~$ cd $FOAM_TUTORIALS
ofuser@49146943821c:tutorials$ ls
Allclean  DNS         compressible      financial       lagrangian  resources
Allrun    basic       discreteMethods   heatTransfer    mesh        stressAnalysis
Alltest   combustion  electromagnetics  incompressible  multiphase
ofuser@49146943821c:tutorials$ cd incompressible/pimpleFoam/channel395
ofuser@49146943821c:channel395$ ./Allrun
./Allrun scripts executes all the workflow for the tutorial

In a common installation of OpenFOAM, usually you do not want to execute the tutorials from the original directory (because you can modify your original source)

But in this case, it is fine the original image will not be modified and the changes we observe here will be lost when exiting the container

If you want to keep the results, you would need to extract the tutorial into a local host directory first and execute the containerised solver from there (mounting the local directory in the docker command) (not explained here)

Or you could also copy out the results (also by mounting the local directory in the docker command, so you have a directory where to copy the results) (not explained here)

(Refer to the Docker documentation if you want to use the Docker image in your computer or elsewhere)
Running blockMesh on /opt/OpenFOAM/OpenFOAM-2.4.x/tutorials/incompressible/pimpleFoam/channel395
Running decomposePar on /opt/OpenFOAM/OpenFOAM-2.4.x/tutorials/incompressible/pimpleFoam/channel395
Running pimpleFoam in parallel on /opt/OpenFOAM/OpenFOAM-2.4.x/tutorials/incompressible/pimpleFoam/channel395 using 5 processes 
OpenFOAM execution is working properly.

This may take several minutes (~15 to ~30 min). Use <Ctrl-C> to kill the solver
ofuser@49146943821c:channel395$ exit
After testing that the Docker container works properly, the final step is to push your container to DockerHub.
As mentioned some paragraphs above, (the myuser name in the commands needs to be replaced with your real account in DockerHub)

(Refer to Docker documentation and website to obtain your DockerHub account) (not explained here)
For example:
ubuntu@vm:01_Docker$ docker push myuser/openfoam:2.4.x
The push refers to repository [docker.io/myuser/openfoam]
6b38074bb2ff: Preparing 
30bc9c9ededb: Preparing 
a238555104b7: Preparing 
30bc9c9ededb: Mounted from pawsey/openfoam 
c49ae6578794: Mounted from pawsey/openfoam 
45cc33e9b7df: Waiting 
502628a9589d: Waiting
.
.
.
e79142719515: Layer already exists 
aeda103e78c9: Layer already exists 
2558e637fbff: Layer already exists 
f749b9b0fb21: Layer already exists 
2.4.x: digest: sha256:717d3e4153c52b517273e2afaadf2e38651c7ecf1d68ce09658fc68e2df806a0 size: 7227

F. Converting the container into Singularity format

The Singularity.def file:
Bootstrap: docker
From: pawsey/openfoam:2.4.x

%post
/bin/mv /bin/sh /bin/sh.original
/bin/ln -s /bin/bash /bin/sh
echo ". /opt/OpenFOAM/OpenFOAM-2.4.x/etc/bashrc" >> $SINGULARITY_ENVIRONMENT
This is the whole definition file for the conversion to singularity. Very simple!

The singularity image will be built from the Pawsey one: pawsey/openfoam:2.4.x

The image in the From: command need to exist in the DockerHub registry (repository)

The first two lines in the %post section instructs the builder to use bash as the shell

This is needed because OpenFOAM scripts have some instructions that can only be interpreted by bash (called bash-isms by geeks)

The last line writes the instruction of sourcing the bashrc file every time the a container instance is created with this image

F.I Steps for porting the image into Singularity:

cd into the 02_PortingToSingularity directory

ubuntu@vm:*-2.4.x$ cd 02_PortingToSingularity
ubuntu@vm:02_*Singularity$ ls

Singularity.def

Use the singularity build:

ubuntu@vm:02_*Singularity$ sudo singularity build openfoam-2.4.x-myuser.sif Singularity.def

You must have sudo/root privileges to execute build
conversion may take around 10 minutes

INFO:    Starting build...
Getting image source signatures
Copying blob f7277927d38a done
Copying blob 8d3eac894db4 done
Copying blob edf72af6d627 done
Copying blob 3e4f86211d23 done
Copying blob e69ac5970cdd done
Copying blob af93086ab594 done
Copying blob 9121f6d149f9 done
.
.
.
2020/05/27 01:34:22  info unpack layer: sha256:11c314071f65ec67414a027ec31d85459cf15f93be7d5d5945d54e9e2cf765c3
2020/05/27 01:34:22  info unpack layer: sha256:a5ff4ed9ae6c32eee3053982461f781edb56b0427a551c8b9541bccc0541234d
2020/05/27 01:34:22  info unpack layer: sha256:29d7c1e14df37b4b7c80226021b633150801041cc9c4401f7d35449608f3d9cf
2020/05/27 01:34:22  info unpack layer: sha256:06d6bd89758266248c1b8cf78a2d21a3fbcef805820f7b91d78ebf1b88d61621
2020/05/27 01:34:47  info unpack layer: sha256:3a96376985c106d721cc9d686d10acaad6084f535e326e4c4799149cccfaa39e
INFO:    Running post scriptlet
+ /bin/mv /bin/sh /bin/sh.original
+ /bin/ln -s /bin/bash /bin/sh
+ echo . /opt/OpenFOAM/OpenFOAM-2.4.x/etc/bashrc
INFO:    Creating SIF file...
INFO:    Build complete: openfoam-2.4.x-myuser.sif

ubuntu@vm:02_*Singularity$ ls

Singularity.def  openfoam-2.4.x-myuser.sif

F.II Test the Singularity image

Run the container interactively and copy the tutorial to the host:

ubuntu@vm:02_*Singularity$ singularity run openfoam-2.4.x-myuser.sif

Singularity> mkdir run
Singularity> cd run
Singularity> cp -r $FOAM_TUTORIALS/incompressible/pimpleFoam/channel395 .
Singularity> cd channel395
Singularity> ls

Note that the directory from where singularity was called is mounted by default, so you can copy stuff from the interior of the container to the host file system

0  0.org  Allrun  constant  system

Use the OpenFOAM tools to if they work. First, create the mesh:

Singularity> blockMesh

/*---------------------------------------------------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  2.4.x                                 |
|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
Build  : 2.4.x-2b147f41daf9
Exec   : blockMesh
.
.
.
----------------
  patch 0 (start: 175300 size: 1200) name: bottomWall
  patch 1 (start: 176500 size: 1200) name: topWall
  patch 2 (start: 177700 size: 1000) name: sides1_half0
  patch 3 (start: 178700 size: 1000) name: sides1_half1
  patch 4 (start: 179700 size: 1000) name: sides2_half0
  patch 5 (start: 180700 size: 1000) name: sides2_half1
  patch 6 (start: 181700 size: 750) name: inout1_half0
  patch 7 (start: 182450 size: 750) name: inout1_half1
  patch 8 (start: 183200 size: 750) name: inout2_half0
  patch 9 (start: 183950 size: 750) name: inout2_half1
   
End

Run the decomposer

Singularity> decomposePar

/*---------------------------------------------------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  2.4.x                                 |
|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
Build  : 2.4.x-2b147f41daf9
Exec   : decomposePar
.
.
.
Time = 0
   
Processor 0: field transfer
Processor 1: field transfer
Processor 2: field transfer
Processor 3: field transfer
Processor 4: field transfer
   
End.

Modify the system/controlDict to write results every time step (just for this particular test)
- set the line of writeInterval to read:
- writeInterval 1;
- You can edit the file, or simply use the following sed replacement:
```
Singularity> sed -i 's,^writeInterval.*,writeInterval  1;,' system/controlDict 
```
- Check the setting:
```
Singularity> grep "writeInterval" system/controlDict 
```
```
writeInterval  1;
```

Run the parallel solver

Singularity> mpiexec -n 5 pimpleFoam -parallel

/*---------------------------------------------------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  2.4.x                                 |
|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
Build  : 2.4.x-2b147f41daf9
Exec   : pimpleFoam -parallel
.
.
.
Courant Number mean: 0.30276 max: 0.526366
Time = 0.4
   
PIMPLE: iteration 1
smoothSolver:  Solving for Ux, Initial residual = 0.0112386, Final residual = 2.57986e-06, No Iterations 3
smoothSolver:  Solving for Uy, Initial residual = 0.0600746, Final residual = 2.34423e-06, No Iterations 4
smoothSolver:  Solving for Uz, Initial residual = 0.0576859, Final residual = 2.06087e-06, No Iterations 4
Pressure gradient source: uncorrected Ubar = 0.133673, pressure gradient = -0.000891632
GAMG:  Solving for p, Initial residual = 0.201055, Final residual = 0.00409835, No Iterations 2
time step continuity errors : sum local = 5.70346e-06, global = -4.59793e-20, cumulative = -4.59793e-20
Pressure gradient source: uncorrected Ubar = 0.133669, pressure gradient = -0.000868433
GAMG:  Solving for p, Initial residual = 0.0314739, Final residual = 3.67096e-07, No Iterations 8
time step continuity errors : sum local = 4.57115e-10, global = -5.07387e-20, cumulative = -9.6718e-20
Pressure gradient source: uncorrected Ubar = 0.133668, pressure gradient = -0.000866965
smoothSolver:  Solving for k, Initial residual = 0.0654047, Final residual = 1.25654e-06, No Iterations 3
bounding k, min: 0 max: 0.000670144 average: 5.74343e-05
ExecutionTime = 16.69 s  ClockTime = 40 s
.
.
.

Important in an interactive session, the MPI installation that is used is the one present inside the container
This is what we call, use of the internal MPI (MPICH in this case)
For the use of the host MPI installation, the container needs to be executed in “hybrid-mode”, as we have explained for the use at Pawsey supercomputers
For executing the singularity container in “hybrid-mode” in your own linux installation, you will need to have a ABI compatible MPICH installed in the host, and follow the instructions from the Singularity documentation (not explained here)
The solver may take several minutes (~10 to ~20 min)
We do not have time to wait, so use <Ctrl-c> to kill the solver after a few time steps

Exit the container and check the results in your local disk

Singularity> exit

ubuntu@vm:02_*Singularity$ cd run/channel395/
ubuntu@vm:channel395$ ls

0  0.org  Allrun  constant  processor0  processor1  processor2  processor3  processor4  system

ubuntu@vm:channel395$ ls -lat processor*/

processor1/:
total 32
drwxr-xr-x  3 ubuntu ubuntu 4096 May 28 12:12 0.8
drwxr-xr-x  8 ubuntu ubuntu 4096 May 28 12:11 .
drwxr-xr-x  3 ubuntu ubuntu 4096 May 28 12:11 0.6
drwxr-xr-x  3 ubuntu ubuntu 4096 May 28 12:11 0.4
drwxr-xr-x  3 ubuntu ubuntu 4096 May 28 12:10 0.2
drwxr-xr-x  2 ubuntu ubuntu 4096 May 28 11:40 0
drwxr-xr-x 11 ubuntu ubuntu 4096 May 28 11:40 ..
drwxr-xr-x  3 ubuntu ubuntu 4096 May 28 11:40 constant
   
processor3/:
total 32
drwxr-xr-x  3 ubuntu ubuntu 4096 May 28 12:12 0.8
drwxr-xr-x  8 ubuntu ubuntu 4096 May 28 12:11 .
drwxr-xr-x  3 ubuntu ubuntu 4096 May 28 12:11 0.6
drwxr-xr-x  3 ubuntu ubuntu 4096 May 28 12:11 0.4
drwxr-xr-x  3 ubuntu ubuntu 4096 May 28 12:10 0.2
drwxr-xr-x  2 ubuntu ubuntu 4096 May 28 11:40 0
drwxr-xr-x 11 ubuntu ubuntu 4096 May 28 11:40 ..
drwxr-xr-x  3 ubuntu ubuntu 4096 May 28 11:40 constant
   
processor2/:
total 32
drwxr-xr-x  3 ubuntu ubuntu 4096 May 28 12:12 0.8
drwxr-xr-x  8 ubuntu ubuntu 4096 May 28 12:11 .
drwxr-xr-x  3 ubuntu ubuntu 4096 May 28 12:11 0.6
drwxr-xr-x  3 ubuntu ubuntu 4096 May 28 12:11 0.4
drwxr-xr-x  3 ubuntu ubuntu 4096 May 28 12:10 0.2
drwxr-xr-x  2 ubuntu ubuntu 4096 May 28 11:40 0
drwxr-xr-x 11 ubuntu ubuntu 4096 May 28 11:40 ..
drwxr-xr-x  3 ubuntu ubuntu 4096 May 28 11:40 constant
   
processor4/:
total 32
drwxr-xr-x  3 ubuntu ubuntu 4096 May 28 12:12 0.8
drwxr-xr-x  8 ubuntu ubuntu 4096 May 28 12:11 .
drwxr-xr-x  3 ubuntu ubuntu 4096 May 28 12:11 0.6
drwxr-xr-x  3 ubuntu ubuntu 4096 May 28 12:11 0.4
drwxr-xr-x  3 ubuntu ubuntu 4096 May 28 12:10 0.2
drwxr-xr-x  2 ubuntu ubuntu 4096 May 28 11:40 0
drwxr-xr-x  3 ubuntu ubuntu 4096 May 28 11:40 constant
drwxr-xr-x 11 ubuntu ubuntu 4096 May 28 11:40 ..
   
processor0/:
total 32
drwxr-xr-x  3 ubuntu ubuntu 4096 May 28 12:12 0.8
drwxr-xr-x  8 ubuntu ubuntu 4096 May 28 12:11 .
drwxr-xr-x  3 ubuntu ubuntu 4096 May 28 12:11 0.6
drwxr-xr-x  3 ubuntu ubuntu 4096 May 28 12:11 0.4
drwxr-xr-x  3 ubuntu ubuntu 4096 May 28 12:10 0.2
drwxr-xr-x  2 ubuntu ubuntu 4096 May 28 11:40 0
drwxr-xr-x 11 ubuntu ubuntu 4096 May 28 11:40 ..
drwxr-xr-x  3 ubuntu ubuntu 4096 May 28 11:40 constant

G. Deploy the image to the system of your preference

The image is good to go!

Copy it to any system where you want to use it

The system will need to have Singularity installed

And, in order to run parallel solvers in the most efficient way (hybrid mode), it needs to count with an ABI compatible MPICH

Key Points

Take it easy, be patient

Use existing definition files examples and available guides to define the right installation recipe

Main difference with a standard OpenFOAM installation guide are’:’

1.Avoiding any step that performs an installation of OpenMPI

2.Settings in prefs.sh for using WM_MPLIB=SYSTEMMPI (MPICH in this case)

3.Settings in bashrc for defining the new location for installation

4.Settings in bashrc for defining WM_PROJECT_USER_DIR

Use OverlayFS to reduce the number of result files

Overview

Teaching: 30 min
Exercises: 30 min

Questions

For old versions of OpenFOAM (without “collated” option available), how can I reduce the amount of result files?

Objectives

Use OverlayFS for saving results and reduce the number of files in the host file system

I. Introduction

The problem with a large amount of files

The creation of a large amount of result files has been a major problem for Supercomputing centres due to the overloading of their file management systems

This major problem affects the performace of the whole system (affecting all the user base)

The collated option is of great benefit for reducing the amount of result files and solving this problematic

fileHandler collated + ioRanks

But it is only fully functional for versions greater-than or equal to - OpenFOAM-6 (from OpenFOAM foundation) - and OpenFOAM-v1812 (from ESI-OpenCFD)

For other versions/flavours of OpenFOAM this setting is not functional or does not exists at all

Then why use other versions/flavours ?

We, indeed, encourage users to update their case settings and tools to tha latest vesions of OpenFOAM with functional collated+ioRanks option

But still users may need to keep using other versions of OpenFOAM:

Because they have a complicated case defined already for another version/flavour

Because they may have own tools written for another version/flavour

Because they may be using additional tools (like waves2Foam or CFDEM) that were written for another version/flavour

Because only that other flavour (mainly Foam-extend) may contain the solver/tool the user needs

And, of course, to update those case settings/tools towards a recent version of OpenFOAM would require extensive effort and time investment

What can Pawsey do to help their user base ?

We need to avoid the overload of our file system in order to keep a performant

Files older than 30 days are purged from /scratch

We have already restricted the user’s quota to a maximum of 1 Million inodes (files/directories)

Users need to modify their workflows (solution-analysis-deletion) for not reaching their quota limit

Use of virtual file systems to store files within (Thanks to Dr. Andrew King for the suggestion!)

Virtual file systems super basics

Extriclty speaking, it is much more complicated than this, but we can simplyfy the concept as:

a big single file that can be formatted to keep several files in its interior

The metadata server will only observe this big file which reduces the overload problem dramatically

Here we make use of OverlayFS due to its proved correct behaviour when working with Singularity

There may be some other useful virtual file system technologies/formats/options (like FUSE)

II. Logic for the use of OverlayFS to store results

a) Start case preparation normally, then rename the standard decomposed directories

Proceed with settings and decomposition of your case as normal

Rename the processor0, processor1 … processor4 to bak.processor0, bak.processor1 … bak.processor4

Now the initial conditions and decomposed mesh are in the bak.processor* subdirectories

You can access to the information directly from those directories

If you need an OpenFOAM tool to access that information, you’ll need to make use of soft links to “trick” OpenFOAM. But, in principle, soft links pointing to the bak.processor* directories would not be needed during execution. They will only be needed for the reconstruction (see reconstruction steps in the following section).

(Do not get confused, for this exercise, the tutorial for OpenFOAM-2.4.x uses 5 subdomains: 0,1,…,4)

b) Create several OverlayFS files and prepare them to store the results

Create a writable OverlayFS file: overlay0

To create this writable OverlayFS file you will need an “ubuntu-based” container version 18.04 or higher

In the local host, copy that file to create the needed replicas of OverlayFS: overlay1, overlay2 … overlay4

For each ovelay* create a corresponding processor* directory in the internal directory structure (using an ubuntu-based container):

insideDir=/overlayOpenFOAM/run/channel395

mkdir -p $insideDir/processor0 in overlay0

…

mkdir -p $insideDir/processor4 in overlay4

For each overlay* copy the initial conditions and the mesh information from the corresponding directory bak.processor* into the processor* directory in the internal structure of the corresponding overlay* file (using an ubuntu-based container):

cp -r bak.processor0/* $insideDir/processor0

…

cp -r bak.processor4/* $insideDir/processors4

c) Use soft links to trick the OpenFOAM tools to write towards the interior of the OverlayFS

In the case directory, create soft-links named processor* that point to the internal directories:

ln -s $insideDir/processor0 processor0

…

ln -s $insideDir/processor4 processor4

The links will appear broken to the host system, but functional to the containers that load the OverlayFS files

d) Execute the containerised solver in hybrid mode, mounting one OverlayFS file per task

Execute the solver in hybrid-mode, allowing MPI task ID to mount its corresponding overlay${ID} file

Each MPI task spawned by srun will have an ID given by the slurm variable: SLURM_PROCID with values 0,1,…,4

For example, if SLURM_PROCID=0, then the mounted OverlayFS file would be overlay0

This allows that:

As SLURM_PROCID=0, then OpenFOAM will read/write to processor0 (which is a link that points towards the interlnal structure of overlay0, specifically towards the directory $insideDir/processor0)

The same for the other MPI tasks

Results will exist inside the OverlayFS files

III. General instructions for following the content

III.I Accessing the scripts for this episode

In this whole episode, we make use of a series of scripts to cover a typical compilation/execution workflow. Lets start by listing the scripts.
cd into the directory where the provided scripts are. In this case we’ll use OpenFOAM-v1912.
zeus-1:~> cd $MYSCRATCH/pawseyTraining/containers-openfoam-workshop-scripts
zeus-1:*-scripts> cd 05_useOverlayFSForReducingNumberOfFiles/example_OpenFOAM-2.4.x
zeus-1:*-2.4.x> ls
A.extractAndAdaptTutorial.sh  caseSettingsFoam.sh      D.runFoam.sh                 imageSettingsSingularity.sh
B.decomposeFoam.sh            C.setupOverlayFoam.sh    E.reconstructFromOverlay.sh  run

Sections and scripts for this episode

In the following sections, there are instructions for submitting these job scripts for execution in the supercomputer one by one:

A.extractAndAdaptTutorial.sh (already pre-executed) is for copying an adapting a case to solve

B.decomposeFoam.sh is decomposing the case to solve

C.setupOverlayFoam.sh is for creating the OverlayFS files to store the result

D.runFoam.sh is for executing the solver (and writing results to the interior of the overlay files)

E.reconstructFromOverlay.sh is for reconstructing a result time initially inside the overlay files

caseSettingsFoam.sh is a script that defines the settings for using OpenFOAM within all the other scripts (it is being sourced from all the A,B,C,..G scripts)

imageSettingsSingularity.sh is a script that defines the settings for using Singularity within all the other scripts (it is being sourced from all the workflow scripts)

So how will this episode flow?

The script of section “A” has already been pre-executed.

We’ll start our explanation at section “B.Decomposition”

but will concentrate our efforts on section “C. Setup OverlayFS”.

Users will then move to section D. and proceed by themselves afterwards.

At the end, we’ll discuss the main instructions within the scripts and the whole process.

A. Extract and adapt the tutorial to be solved - [Pre-Executed]

The `A.extractAndAdaptTutorial.sh` script (main parts to be discussed):

#1. Loading the container settings, case settings and auxiliary functions (order is important)
source $SLURM_SUBMIT_DIR/imageSettingsSingularity.sh
source $SLURM_SUBMIT_DIR/caseSettingsFoam.sh

The `imageSettingsSingularity.sh` script (main sections to be discussed):

#Module environment
module load singularity

#Defining the container to be used
theRepo=/group/singularity/pawseyRepository/OpenFOAM
theContainerBaseName=openfoam
theVersion=2.4.x
theProvider=pawsey
theImage=$theRepo/$theContainerBaseName-$theVersion-$theProvider.sif

The `caseSettingsFoam.sh` script (main sections to be discussed):

#Choosing the tutorial case
tutorialAppDir=incompressible/pimpleFoam
tutorialName=channel395
tutorialCase=$tutorialAppDir/$tutorialName

#Choosing the working directory for the case to solve
baseWorkingDir=$SLURM_SUBMIT_DIR/run
if ! [ -d $baseWorkingDir ]; then
    echo "Creating baseWorkingDir=$baseWorkingDir"
    mkdir -p $baseWorkingDir
fi
caseName=$tutorialName
caseDir=$baseWorkingDir/$caseName

A.I Steps for dealing with the extraction and adaptation of the case to be solved
Submit the job (no need for reservation as the script uses the copyq partition)
zeus-1:*-2.4.x> sbatch A.extractAndAdaptTutorial.sh 
Submitted batch job 4632758
Check that the tutorial has been copied to our host file system
zeus-1:*-2.4.x> ls ./run/channel395/
0  0.orig  Allrun  constant  system
Read the controlDict dictionary:
zeus-1:*-2.4.x> view ./run/channel395/system/controlDict
~
~
~
:q
The settings that were adapted in ..../run/channel395/system/controlDict
To use binary writing format to accelerate writing and reduce the size of the files
 writeFormat     binary;
Never use runTimeModifiable. This option creates permanent reading of dictionaries (each time step) which overloads the shared file system.
 runTimeModifiable false;

B. Decomposition

The `B.decomposeFoam.sh` script (main points to be discussed):

#1. Loading the container settings, case settings and auxiliary functions (order is important)
source $SLURM_SUBMIT_DIR/imageSettingsSingularity.sh
source $SLURM_SUBMIT_DIR/caseSettingsFoam.sh

The `imageSettingsSingularity.sh` script (main sections to be discussed):

#Module environment
module load singularity

#Defining the container to be used
theRepo=/group/singularity/pawseyRepository/OpenFOAM
theContainerBaseName=openfoam
theVersion=2.4.x
theProvider=pawsey
theImage=$theRepo/$theContainerBaseName-$theVersion-$theProvider.sif

The `caseSettingsFoam.sh` script (main sections to be discussed):

#Choosing the working directory for the case to solve
baseWorkingDir=$SLURM_SUBMIT_DIR/run
if ! [ -d $baseWorkingDir ]; then
    echo "Creating baseWorkingDir=$baseWorkingDir"
    mkdir -p $baseWorkingDir
fi
caseName=$tutorialName
caseDir=$baseWorkingDir/$caseName

#7. Perform all preprocessing OpenFOAM steps up to decomposition
srun -n 1 -N 1 singularity exec $theImage blockMesh 2>&1 | tee $logsDir/log.blockMesh
srun -n 1 -N 1 singularity exec $theImage decomposePar -cellDist 2>&1 | tee $logsDir/log.decomposePar

B.I Steps for dealing with decomposition:
Submit the decomposition script from the scripts directory (use the reservation for the workshop if available)
zeus-1:*-2.4.x> myReservation=containers
zeus-1:*-2.4.x> sbatch --reservation=$myReservation B.decomposeFoam.sh 
If a reservation is not available, do not use the option. (Or you can use the debugq: --partition=debugq instead.)
Submitted batch job 4632558
Check that the decomposition has been performed:
zeus-1:*-2.4.x> ls ./run/channel395/processor*
./run/channel395/processor0:
0  constant
   
./run/channel395/processor1:
0  constant
   
./run/channel395/processor2:
0  constant
   
./run/channel395/processor3:
0  constant
   
./run/channel395/processor4:
0  constant
Note that a processor* directory was created per numberOfSubdomains (The number of subdomains is set in the system/decomposeParDict dictionary)

Also note that this tutorial uses 5 subdomains (and 5 cores when executing the solver (below))
You should also check for success/errors in:

the slurm output file: slurm-<SLURM_JOBID>.out

the log files created when executing the OpenFOAM tools in: ./run/channel395/logs/pre/

C. Setup the overlayFS

The `C.setupOverlayFoam.sh` script (main points to be discussed):

#SBATCH --ntasks=4 #Several tasks will be used for copying files. (Independent from the numberOfSubdomains)

#1. Loading the container settings, case settings and auxiliary functions (order is important)
source $SLURM_SUBMIT_DIR/imageSettingsSingularity.sh
source $SLURM_SUBMIT_DIR/caseSettingsFoam.sh

The `imageSettingsSingularity.sh` script (main sections to be discussed):

#Module environment
module load singularity

#Defining the container to be used
theRepo=/group/singularity/pawseyRepository/OpenFOAM
theContainerBaseName=openfoam
theVersion=2.4.x
theProvider=pawsey
theImage=$theRepo/$theContainerBaseName-$theVersion-$theProvider.sif

#Defining settings for the OverlayFS
overlaySizeGb=1

The `caseSettingsFoam.sh` script (main sections to be discussed):

#Choosing the working directory for the case to solve
baseWorkingDir=$SLURM_SUBMIT_DIR/run
if ! [ -d $baseWorkingDir ]; then
    echo "Creating baseWorkingDir=$baseWorkingDir"
    mkdir -p $baseWorkingDir
fi
caseName=$tutorialName
caseDir=$baseWorkingDir/$caseName

#Defining the name of the directory inside the overlay* files at which results will be saved
baseInsideDir=/overlayOpenFOAM/run
insideName=$caseName
insideDir=$baseInsideDir/$insideName

#4. Rename the processor* directories into bak.processor*
#(OpenFOAM wont be able to see these directories)
#(Access will be performed through soft links)
echo "Renaming the processor directories"
rename processor bak.processor processor*

#5. Creating a first overlay file (overlay0)
#(Needs to use ubuntu:18.04 or higher to use the -d <root-dir> option to make them writable by simple users)
echo "Creating the overlay0 file"
echo "The size in Gb is overlaySizeGb=$overlaySizeGb"
if [ $overlaySizeGb -gt 0 ]; then
   countSize=$(( overlaySizeGb * 1024 * 1024 ))
   srun -n 1 -N 1 singularity exec docker://ubuntu:18.04 bash -c " \
        mkdir -p overlay_tmp/upper && \
        dd if=/dev/zero of=overlay0 count=$countSize bs=1024 && \
        mkfs.ext3 -d overlay_tmp overlay0 && rm -rf overlay_tmp \
        "
else
   echo "Variable overlaySizeGb was not set correctly"
   echo "In theory, this should have been set together with the singularity settings"
   echo "Exiting";exit 1
fi

#6. Replicating the overlay0 file into the needed number of overlay* files (as many as processors*)
echo "Replication overlay0 into the rest of the overlay* files"
for ii in $(seq 1 $(( foam_numberOfSubdomains - 1 ))); do
   if [ -f overlay${ii} ]; then
      echo "overlay${ii} already exists"
      echo "Deal with it first and remove it from the working directory"
      echo "Exiting";exit 1
   else
      echo "Replicating overlay0 into overlay${ii}"
      srun -n 1 -N 1 --mem-per-cpu=0 --exclusive cp overlay0 overlay${ii} &
   fi
done
wait

#7. Creating inside processor* directories inside the overlayFS 
echo "Creating the directories inside the overlays"
for ii in $(seq 0 $(( foam_numberOfSubdomains - 1 ))); do
   echo "Creating processor${ii} inside overlay${ii}"
   srun -n 1 -N 1 --mem-per-cpu=0 --exclusive singularity exec --overlay overlay${ii} $theImage mkdir -p $insideDir/processor${ii} &
done
wait

#8. Transfer the content of the bak.processor* directories into the overlayFS
echo "Copying OpenFOAM files inside bak.processor* into the overlays"
for ii in $(seq 0 $(( foam_numberOfSubdomains - 1 ))); do
    echo "Writing into overlay${ii}"
    srun -n 1 -N 1 --mem-per-cpu=0 --exclusive singularity exec --overlay overlay${ii} $theImage cp -r bak.processor${ii}/* $insideDir/processor${ii}/ &
done
wait

#9. List the content of directories inside the overlay* files
echo "Listing the content in overlay0 $insideDir/processor0"
srun -n 1 -N 1 singularity exec --overlay overlay0 $theImage ls -lat $insideDir/processor0/

C.I Steps for dealing with the Overlay setup
Submit the solver script (from the scripts directory)
zeus-1:*-2.4.x> sbatch --reservation=$myReservation C.prepareOverlayFoam.sh 
If a reservation is not available, do not use the option. (Or you can use the debugq: --partition=debugq instead.)
Submitted batch job 4642685 on cluster zeus
Check that the overlay files were created and the processor* directories renamed to bak.processor*:
zeus-1:*-2.4.x> cd run/channel395
zeus-1:channel395> ls
0      Allrun          bak.processor1  bak.processor3  constant  overlay0  overlay2  overlay4
0.org  bak.processor0  bak.processor2  bak.processor4  logs      overlay1  overlay3  system
There are now 5 overlay* files

All processor* directories have been renamed to bak.processor*
Explore the content of one of the overlay files:
zeus-1:channel395> module load singularity
zeus-1:channel395> theImage=/group/singularity/pawseyRepository/OpenFOAM/openfoam-2.4.x-pawsey.sif
zeus-1:channel395> insideDir=/overlayOpenFOAM/run/channel395
zeus-1:channel395> singularity exec --overlay overlay1 $theImage ls -lat $insideDir/processor1/
total 16
drwxr-s---+ 4 espinosa pawsey0001 4096 May 24 20:38 .
drwxr-s---+ 2 espinosa pawsey0001 4096 May 24 20:38 0
drwxr-s---+ 3 espinosa pawsey0001 4096 May 24 20:38 constant
drwxr-s---+ 3 espinosa pawsey0001 4096 May 24 20:38 ..

D. Executing the solver

The D.runFoam script (main points to be discussed):
#SBATCH --ntasks=5
#5. Defining OpenFOAM controlDict settings for this run
foam_endTime=10
foam_writeInterval=1
foam_purgeWrite=0 #Just for testing in this exercise. In reality this should have a reasonable value if possible
#foam_purgeWrite=10 #Just 10 times will be preserved
#7. Creating soft links towards directories inside the overlayFS files
#These links and directories will be recognized by each mpi instance of the container
#(Initially these links will appear broken as they are pointing towards the interior of the overlay* files.
# They will only be recognized within the containers)
echo "Creating the soft links to point towards the interior of the overlay files"

for ii in $(seq 0 $(( foam_numberOfSubdomains -1 ))); do
   echo "Linking to $insideDir/processor${ii} in overlay${ii}"
   srun -n 1 -N 1 --mem-per-cpu=0 --exclusive ln -s $insideDir/processor${ii} processor${ii} &
done
wait
#8. Execute the case using the softlinks to write inside the overlays
echo "About to execute the case"
srun -n $SLURM_NTASKS -N $SLURM_JOB_NUM_NODES bash -c "singularity exec --overlay "'overlay${SLURM_PROCID}'" $theImage pimpleFoam -parallel 2>&1" | tee $logsDir/log.pimpleFoam.$SLURM_JOBID
echo "Execution finished"
VERY IMPORTANT: Note that the singularity command is called inside a bash -c command

This is the way we allow each MPI task to pick a different overlay file through the SLURM_PROCID variable

Here, SLURM_PROCID is slurm environment variable which needs to be evaluated when executing the container, so we use the section in single quotes '...' to allow the internal evaluation of that variable

Here, theImage is not a global environment variable, is evaluated by the host shell in a section with double quotes "..." at the command line

The total string passed to bash -c is the concatenation of two doble quotes sections with a single quotes section in between
#9. List the existing times inside the overlays 
echo "Listing the available times inside overlay0"
srun -n 1 -N 1 singularity exec --overlay overlay0 $theImage ls -lat processor0/

D.I Steps for dealing with the solver

Submit the solver script (from the scripts directory)
```
zeus-1:*-2.4.x> sbatch --reservation=$myReservation D.runFoam.sh 
```
- If a reservation is not available, do not use the option. (Or you can use the debugq: --partition=debugq instead.)
```
Submitted batch job 4632685 on cluster zeus
```

Check that the solver is running:

zeus-1:*-2.4.x> squeue -u $USER

JOBID    USER     ACCOUNT     PARTITION            NAME EXEC_HOST ST     REASON   START_TIME     END_TIME  TIME_LEFT NODES   PRIORITY
4632685  espinosa pawsey0001  workq        D.runFoam.sh       n/a PD  Resources     17:09:28     17:19:28      10:00     1      75190 

Observe the output of the job with tail -f at runtime (use <Ctrl-C> to exit the command):

zeus-1:*-2.4.x> tail -f slurm-4632685.out

.
.
.
Time = 0.2
   
PIMPLE: iteration 1
smoothSolver:  Solving for Ux, Initial residual = 0.0118746, Final residual = 1.89249e-06, No Iterations 3
smoothSolver:  Solving for Uy, Initial residual = 0.0617212, Final residual = 1.68113e-06, No Iterations 4
smoothSolver:  Solving for Uz, Initial residual = 0.0589944, Final residual = 9.70923e-06, No Iterations 3
Pressure gradient source: uncorrected Ubar = 0.13369, pressure gradient = -0.000964871
GAMG:  Solving for p, Initial residual = 0.213844, Final residual = 0.00414884, No Iterations 2
time step continuity errors : sum local = 5.82807e-06, global = -1.41211e-19, cumulative = -1.41211e-19
Pressure gradient source: uncorrected Ubar = 0.133687, pressure gradient = -0.000947989
GAMG:  Solving for p, Initial residual = 0.0222643, Final residual = 4.30412e-07, No Iterations 7
time step continuity errors : sum local = 5.63638e-10, global = -2.40486e-19, cumulative = -3.81697e-19
Pressure gradient source: uncorrected Ubar = 0.133687, pressure gradient = -0.000947874
ExecutionTime = 0.25 s  ClockTime = 0 s
.
.
.

You can see in the case directory that now there are several processor* soft links

zeus-1:*-2.4.x> cd run/channel395
zeus-1:channel395> ls -lat 

total 5242952
-rw-rw----+  1 espinosa pawsey0001 1073741824 May 25 14:55 overlay0
-rw-rw----+  1 espinosa pawsey0001 1073741824 May 25 14:55 overlay1
-rw-rw----+  1 espinosa pawsey0001 1073741824 May 25 14:55 overlay2
-rw-rw----+  1 espinosa pawsey0001 1073741824 May 25 14:55 overlay3
-rw-rw----+  1 espinosa pawsey0001 1073741824 May 25 14:55 overlay4
drwxr-s---+ 12 espinosa pawsey0001       4096 May 25 14:55 .
drwxrws---+  4 espinosa pawsey0001       4096 May 25 14:55 logs
lrwxrwxrwx   1 espinosa pawsey0001         42 May 25 14:55 processor0 -> /overlayOpenFOAM/run/channel395/processor0
lrwxrwxrwx   1 espinosa pawsey0001         42 May 25 14:55 processor1 -> /overlayOpenFOAM/run/channel395/processor1
lrwxrwxrwx   1 espinosa pawsey0001         42 May 25 14:55 processor2 -> /overlayOpenFOAM/run/channel395/processor2
lrwxrwxrwx   1 espinosa pawsey0001         42 May 25 14:55 processor3 -> /overlayOpenFOAM/run/channel395/processor3
lrwxrwxrwx   1 espinosa pawsey0001         42 May 25 14:55 processor4 -> /overlayOpenFOAM/run/channel395/processor4
drwxr-s---+  2 espinosa pawsey0001       4096 May 25 14:55 system
drwxrws---+  4 espinosa pawsey0001       4096 May 25 14:53 bak.processor0
drwxrws---+  4 espinosa pawsey0001       4096 May 25 14:53 bak.processor1
drwxrws---+  4 espinosa pawsey0001       4096 May 25 14:53 bak.processor2
drwxrws---+  4 espinosa pawsey0001       4096 May 25 14:53 bak.processor3
drwxrws---+  4 espinosa pawsey0001       4096 May 25 14:53 bak.processor4
drwxr-s---+  2 espinosa pawsey0001       4096 May 25 14:53 0
drwxr-s---+  3 espinosa pawsey0001       4096 May 25 14:53 constant
drwxrws---+  3 espinosa pawsey0001       4096 May 25 14:03 ..
drwxr-s---+  2 espinosa pawsey0001       4096 May 25 14:03 0.org
-rwxrwx---+  1 espinosa pawsey0001        483 May 25 14:03 Allrun

The processor* soft links are pointing to the directories inside the overlay* files

zeus-1:channel395> ls -la processor1/

ls: cannot access 'processor1/': No such file or directory

The host shell cannot read the internal directory structure that lives the overlay* files, and that is why the links appear broken

Check that the solver gave some results by listing the interior of an overlay file:

zeus-1:channel395> module load singularity
zeus-1:channel395> theImage=/group/singularity/pawseyRepository/OpenFOAM/openfoam-2.4.x-pawsey.sif
zeus-1:channel395> singularity exec --overlay overlay1 $theImage ls processor1/

  0.6  1.2  1.8  2.2  2.8  3.4  4	4.6  5.2  5.8  6.4  7	 7.6  8.2  8.8	9.4  constant
2  0.8  1.4  10   2.4  3    3.6  4.2	4.8  5.4  6    6.6  7.2  7.8  8.4  9	9.6
4  1	  1.6  2    2.6  3.2  3.8  4.4	5    5.6  6.2  6.8  7.4  8    8.6  9.2	9.8

You should also check for success/errors in:
- the slurm output file: slurm-<SLURM_JOBID>.out
- the log files created when executing the OpenFOAM tools in: ./run/channel395/logs/run/

E. Reconstruction

No, unfortunately a container cannot mount more than 1 OverlayFS file at the same time

Yes, this implies that the results need to be copied back to the host file system before reconstruction

This is the inverse operation to the process of copying the initial decomposition into the OverlayFS files (explained at the beginning of this episode)

But in order to avoid the presence of many files in the host, this should be done by small batches:

Copy small batch of results from the internal structure of overlay* towards the bak.processor* directories in the host file system

Now create processor* soft links to point towards the corresponding bak.processor* directories and not to the OverlayFS interior

Reconstruct that small batch (here we use batch size=1, but the following episode uses larger batches)

After reconstruction, the reconstructed time-results now exist in the main case directory. Then, remove the files/directories from the decomposed result-times inside the bak.processor* as they are not needed anymore (and because we do not want to keep a large number of files).

The `E.reconstructFromOverlay.sh` script (main points to be discussed):

#SBATCH --ntasks=4 #Several tasks will be used for copying files. (Independent from the numberOfSubdomains)

#4. Transfer the content of the overlayFS into the bak.processor* directories
reconstructionTime=10
echo "Copying the times to reconstruct from the overlays into bak.processor*"
for ii in $(seq 0 $(( foam_numberOfSubdomains - 1 ))); do
   echo "Writing into bak.processor${ii}"
   srun -n 1 -N 1 --mem-per-cpu=0 --exclusive singularity exec --overlay overlay${ii} $theImage cp -r $insideDir/processor${ii}/$reconstructionTime bak.processor${ii} &
done
wait

The time to be reconstructed is reconstructionTime=10
(Here we only reconstruct a single time in each attempt, but batches of larger size are explained in the following episode)
To be able to reconstruct a specific time-result, information needs to be transferred to the corresponding bak.processor* directories

#5. Point the soft links to the bak.processor* directories
echo "Creating the soft links to point towards the bak.processor* directories"
for ii in $(seq 0 $(( foam_numberOfSubdomains -1 ))); do
   echo "Linking to bak.processor${ii}"
   srun -n 1 -N 1 --mem-per-cpu=0 --exclusive ln -s bak.processor${ii} processor${ii} &
done
wait

Now new processor* soft links will point towards the bak.processor* physical directories

#6. Reconstruct the indicated time
echo "Start reconstruction"
srun -n 1 -N 1 singularity exec $theImage reconstructPar -time ${reconstructionTime} 2>&1 | tee $logsDir/log.reconstructPar.$SLURM_JOBID
if grep -i 'error\|exiting' $logsDir/log.reconstructPar.$SLURM_JOBID; then
   echo "The reconstruction of time ${reconstructionTime} failed"
   echo "Exiting";exit 1
fi

E.I Steps for dealing with reconstruction:
Submit the reconstruction script (from the scripts directory)
zeus-1:*-2.4.x> sbatch --reservation=$myReservation E.reconstructFromOverlay.sh 
If a reservation is not available, do not use the option. (Or you can use the debugq: --partition=debugq instead.)
Submitted batch job 4632899 on cluster zeus
Check that the reconstruction has been performed (a directory for the last time of the solution, 10 in this case, should appear at in the case directory):
zeus-1:*-2.4.x> ls ./run/channel395/
0      Allrun          bak.processor2  constant  overlay1  overlay4    processor2  system
0.org  bak.processor0  bak.processor3  logs      overlay2  processor0  processor3
10     bak.processor1  bak.processor4  overlay0  overlay3  processor1  processor4
You should also check for success/errors in:

the slurm output file: slurm-<SLURM_JOBID>.out

the log files created when executing the OpenFOAM tools in: ./run/channel395/logs/post/

Z. Further notes on how to use OpenFOAM and OpenFOAM containers at Pawsey

More on OverlayFS for singularity in: https://sylabs.io/guides/3.5/user-guide/persistent_overlays.html

The usage of OpenFOAM and OpenFOAM containers at Pawsey has already been described in our documentation: OpenFOAM documentation at Pawsey

and in a technical newsletter note: https://support.pawsey.org.au/documentation/display/US/Pawsey+Technical+Newsletter+2020-04

Key Points

Singularity can deal with an OverlayFS, but only one OverlayFS can be mounted per container instance

As each core writes results to a single processor*, this works for saving results inside the corresponding overlay*

Unfortunately, the reconstructPar tool cannot read results from several overlay* files at the same time. Therefore, decomposed results must be copied back to the host file system before reconstruction.

Last point may seem like a killer, but extraction and reconstruction may be performed in small batches avoiding the appearence of many files at the same time in the host file system.

Here the small batch is of size=1 (just a single time result), but the following episode deals with batches of larger size

Advanced scripts for postprocessing with OverlayFS

Overview

Teaching: 30 min
Exercises: 30 min

Questions

How can I postprocess a large number of results?

Objectives

Use provided scripts to postprocess a large number of results without increasing dramatically the existing number of files in the system

0. Introduction

Postprocessing results in batches

The workflow presented here is extremelly similar to that in the previous episode (please study previous one first)

Some minor differences in this episode are: 1) the overlay* files are kept in the directory ./overlayFSDir, and 2) the renamed bak.processor* directories are moved to the directory ./bakDir

And the main difference from the previous episode is: the reconstruction of the existing results can handle many time-results in the same job

Nevertheless, in order to avoid the creation of a large number of files in the host file system, the extraction of the result-times from the overlayFS files is performed in batches of small size

All the decomposed result-times of each small batch are postprocessed (reconstructed) and then deleted before the following batch is extracted/processed

Use of bash functions

Another difference in the workflow scripts for this episode is the use of bash functions

These functions are defined in a separate bash script contained in the directory named ../../A1_auxiliaryScripts

That script is sourced from all the workflow (A,B,C..G) scripts

In practice, users should keep this directory in a known location and source the definition of bash functions from there

0.I Accessing the scripts for this episode

In this episode, we make use of a series of scripts to cover a typical compilation/execution workflow. Lets start by listing the scripts.
List the content of the A1_auxiliaryScripts directory:
zeus-1:~> cd $MYSCRATCH/pawseyTraining/containers-openfoam-workshop-scripts
zeus-1:*-scripts> ls A1_auxiliaryScripts
ofContainersOverlayFunctions.sh
This file contains the definition of several functions utilised within the workflow scripts
cd into the directory that contains the scripts for executing the workflow of this exercise. In this case we’ll use OpenFOAM-2.4.x.
zeus-1:*-scripts> cd 06_advancedScriptsForPostProcessingWithOverlayFS/example_OpenFOAM-2.4.x
zeus-1:*-2.4.x> ls
A.extractAndAdpatTutorial.sh  C.setupOverlayFoam.sh        F.extractFromOverlayIntoBak.sh  run
B.decomposeFoam.sh            D.runFoam.sh                 G.reconstructFromBak.sh
caseSettingsFoam.sh           E.reconstructFromOverlay.sh  imageSettingsSingularity.sh

Sections and scripts for this episode

In the following sections, there are instructions for submitting these workflow scripts for execution in the supercomputer one by one:

A.extractAndAdaptTutorial.sh (already pre-executed) is for copying an adapting a case to solve

B.decomposeFoam.sh is decomposing the case to solve

C.setupOverlayFoam.sh is for creating the OverlayFS files to store the result

D.runFoam.sh is for executing the solver (and writing results to the interior of the overlay files)

E.reconstructFromOverlay.sh is for reconstructing several result-times in batches

F.extractFromOverlayIntoBak.sh is for extracting a batch of result-times from the overlay files (no reconstruction)

G.reconstructFromBak.sh is for reconstructing existing results in the bak.processor* directories

caseSettingsFoam.sh is a script that defines the settings for using OpenFOAM within all the workflow scripts (it is being sourced from all the A,B,C,..G scripts)

imageSettingsSingularity.sh is a script that defines the settings for using Singularity within all the other scripts (it is being sourced from all the workflow scripts)

../../A1_auxiliaryScripts/ofContainersOverlayFunctions.sh is the script that contains the definition of the bash functions utilised in the workflow scripts (This script is sourced from all the workflow scripts)

So how will this episode flow?

The script of section “A. has already been pre-executed.

The usage of these scripts is extremely similar to that of previous episode (we recommend to practise previous episode first)

Additional explanations are dedicated to the use of the bash functions

And on the reconstruction made in batches (section E.)

Sections F. and G. are left to the user to explore by themselves

A. Extract and adapt the tutorial to be solved - [Pre-Executed]

The `A.extractAndAdaptTutorial.sh` script (main parts to be discussed):

#1. Loading the container settings, case settings and auxiliary functions (order is important)
source $SLURM_SUBMIT_DIR/imageSettingsSingularity.sh
source $SLURM_SUBMIT_DIR/caseSettingsFoam.sh
overlayFunctionsScript=$auxScriptsDir/ofContainersOverlayFunctions.sh

The `imageSettingsSingularity.sh` script (main sections to be discussed):

#Module environment
module load singularity

#Defining the container to be used
theRepo=/group/singularity/pawseyRepository/OpenFOAM
theContainerBaseName=openfoam
theVersion=2.4.x
theProvider=pawsey
theImage=$theRepo/$theContainerBaseName-$theVersion-$theProvider.sif

#Defining the path of the auxiliary scripts for dealing with overlayFS
#(Define the path to a more permanent directory for production workflows)
auxScriptsDir=$SLURM_SUBMIT_DIR/../../A1_auxiliaryScripts

The `caseSettingsFoam.sh` script (main sections to be discussed):

#Choosing the tutorial case
tutorialAppDir=incompressible/pimpleFoam
tutorialName=channel395
tutorialCase=$tutorialAppDir/$tutorialName

#Choosing the working directory for the case to solve
baseWorkingDir=$SLURM_SUBMIT_DIR/run
if ! [ -d $baseWorkingDir ]; then
    echo "Creating baseWorkingDir=$baseWorkingDir"
    mkdir -p $baseWorkingDir
fi
caseName=$tutorialName
caseDir=$baseWorkingDir/$caseName

A.I Steps for dealing with the extraction and adaptation of the case to be solved
Submit the job (no need for reservation as the script uses the copyq partition)
zeus-1:*-2.4.x> sbatch A.extractAndAdaptTutorial.sh 
Submitted batch job 4632758
Check that the tutorial has been copied to our host file system
zeus-1:*-2.4.x> ls ./run/channel395/
0  0.orig  Allrun  constant  system
Read the controlDict dictionary:
zeus-1:*-2.4.x> view ./run/channel395/system/controlDict
~
~
~
:q
The settings that were adapted in ./run/channel395/system/controlDict
To use binary writing format to accelerate writing and reduce the size of the files
 writeFormat     binary;
Never use runTimeModifiable. This option creates permanent reading of dictionaries (each time step) which overloads the shared file system.
 runTimeModifiable false;

B. Decomposition

The `B.decomposeFoam.sh` script (main points to be discussed):

#1. Loading the container settings, case settings and auxiliary functions (order is important)
source $SLURM_SUBMIT_DIR/imageSettingsSingularity.sh
source $SLURM_SUBMIT_DIR/caseSettingsFoam.sh
overlayFunctionsScript=$auxScriptsDir/ofContainersOverlayFunctions.sh

The `imageSettingsSingularity.sh` script (main sections to be discussed):

#Module environment
module load singularity

#Defining the container to be used
theRepo=/group/singularity/pawseyRepository/OpenFOAM
theContainerBaseName=openfoam
theVersion=2.4.x
theProvider=pawsey
theImage=$theRepo/$theContainerBaseName-$theVersion-$theProvider.sif

#Defining the path of the auxiliary scripts for dealing with overlayFS
#(Define the path to a more permanent directory for production workflows)
auxScriptsDir=$SLURM_SUBMIT_DIR/../../A1_auxiliaryScripts

The `caseSettingsFoam.sh` script (main sections to be discussed):

#Choosing the working directory for the case to solve
baseWorkingDir=$SLURM_SUBMIT_DIR/run
if ! [ -d $baseWorkingDir ]; then
    echo "Creating baseWorkingDir=$baseWorkingDir"
    mkdir -p $baseWorkingDir
fi
caseName=$tutorialName
caseDir=$baseWorkingDir/$caseName

#7. Perform all preprocessing OpenFOAM steps up to decomposition
srun -n 1 -N 1 singularity exec $theImage blockMesh 2>&1 | tee $logsDir/log.blockMesh
srun -n 1 -N 1 singularity exec $theImage decomposePar -cellDist 2>&1 | tee $logsDir/log.decomposePar

B.I Steps for dealing with decomposition:
Submit the decomposition script from the scripts directory (use the reservation for the workshop if available)
zeus-1:*-2.4.x> myReservation=containers
zeus-1:*-2.4.x> sbatch --reservation=$myReservation B.decomposeFoam.sh 
If a reservation is not available, do not use the option. (Or you can use the debugq: --partition=debugq instead.)
Submitted batch job 4632558
Check that the decomposition has been performed:
zeus-1:*-2.4.x> ls ./run/channel395/processor*
./run/channel395/processor0:
0  constant
   
./run/channel395/processor1:
0  constant
   
./run/channel395/processor2:
0  constant
   
./run/channel395/processor3:
0  constant
   
./run/channel395/processor4:
0  constant
Note that one processor* directory was created per numberOfSubdomains (The number of subdomains is set in the system/decomposeParDict dictionary)

Also note that this tutorial uses 5 subdomains (and 5 cores when executing the solver (below))
You should also check for success/errors in:

the slurm output file: slurm-<SLURM_JOBID>.out

the log files created when executing the OpenFOAM tools in: ./run/channel395/logs/pre/

C. Setup the overlayFS

The `C.setupOverlayFoam.sh` script (main points to be discussed):

#SBATCH --ntasks=4 #Several tasks will be used for copying files. (Independent from the numberOfSubdomains)

#1. Loading the container settings, case settings and auxiliary functions (order is important)
source $SLURM_SUBMIT_DIR/imageSettingsSingularity.sh
source $SLURM_SUBMIT_DIR/caseSettingsFoam.sh
overlayFunctionsScript=$auxScriptsDir/ofContainersOverlayFunctions.sh

The `imageSettingsSingularity.sh` script (main sections to be discussed):

#Module environment
module load singularity

#Defining the container to be used
theRepo=/group/singularity/pawseyRepository/OpenFOAM
theContainerBaseName=openfoam
theVersion=2.4.x
theProvider=pawsey
theImage=$theRepo/$theContainerBaseName-$theVersion-$theProvider.sif

#Defining settings for the OverlayFS
overlaySizeGb=1

#Defining the path of the auxiliary scripts for dealing with overlayFS
#(Define the path to a more permanent directory for production workflows)
auxScriptsDir=$SLURM_SUBMIT_DIR/../../A1_auxiliaryScripts

The `caseSettingsFoam.sh` script (main sections to be discussed):

#Choosing the working directory for the case to solve
baseWorkingDir=$SLURM_SUBMIT_DIR/run
if ! [ -d $baseWorkingDir ]; then
    echo "Creating baseWorkingDir=$baseWorkingDir"
    mkdir -p $baseWorkingDir
fi
caseName=$tutorialName
caseDir=$baseWorkingDir/$caseName

#Defining the name of the directory inside the overlay* files at which results will be saved
baseInsideDir=/overlayOpenFOAM/run
insideName=$caseName
insideDir=$baseInsideDir/$insideName

#3. Create the directory where the OverlayFS files are going to be kept
if ! [ -d ./overlayFSDir ]; then
   echo "Creating the directory ./overlayFSDir which will contain the overlay* files:"
   mkdir -p ./overlayFSDir
else
   echo "For some reason, the directory ./overlayFSDir for saving the overlay* files already exists:"
   echo "Warning:No creation needed"
fi

#5. Rename the processor* directories into bak.processor* and move them into ./bakDir
#(OpenFOAM wont be able to see these directories)
#(Access will be performed through soft links)
echo "Renaming the processor directories"
rename processor bak.processor processor*
if ! [ -d ./bakDir ]; then
   echo "Creating the directory ./bakDir that will contain the bak.processor* directories:"
   mkdir -p ./bakDir
else
   echo "For some reason, the directory ./bakDir for containing the bak.processor* dirs already exists:"
   echo "Warning:No creation needed"
fi

if ! [ -d ./bakDir/bak.processor0 ]; then
   echo "Moving all bak.processor* directories into ./bakDir"
   mv bak.processor* ./bakDir
else
   echo "The directory ./bakDir/bak.processor0 already exists"
   echo "No move/replacement of bak.processor* directories will be performed"
   echo "Exiting";exit 1
fi

#6. Creating a first ./overlayFSDir/overlayII file (./overlayFSDir/overlay0)
createOverlay0 $overlaySizeGb;success=$? #Calling the function for creating the ./overlayFSDir/overlay0 file
if [ $success -eq 222 ]; then 
   echo "./overlayFSDir/overlay0 already exists"
   echo "Exiting";exit 1
elif [ $success -ne 0 ]; then 
   echo "Failed creating ./overlayFSDir/overlay0, exiting"
   echo "Exiting";exit 1
fi

Note the use of the createOverlay0 function
The function receives as an argument the size (in Gb) of the file to be created
The returned value of the function is saved in the success variable and then checked
Read the definition of the function in ../../A1_auxiliaryScripts/ofContainersOverlayFunctions.sh

#7. Replicating the ./overlayFSDir/overlay0 file into the needed number of ./overlayFSDir/overlay* files (as many as processors*)
echo "Replication ./overlayFSDir/overlay0 into the rest of the ./overlayFSDir/overlay* files"
for ii in $(seq 1 $(( foam_numberOfSubdomains - 1 ))); do
    if [ -f ./overlayFSDir/overlay${ii} ]; then
       echo "./overlayFSDir/overlay${ii} already exists"
       echo "Deal with it first and remove it from the working directory"
       echo "Exiting";exit 1
    else
       echo "Replicating ./overlayFSDir/overlay0 into ./overlayFSDir/overlay${ii}"
       srun -n 1 -N 1 --mem-per-cpu=0 --exclusive cp ./overlayFSDir/overlay0 ./overlayFSDir/overlay${ii} &
    fi
done
wait

#8. Creating the processor* directories inside the ./overlayFSDir/overlay* files
createInsideProcessorDirs $insideDir $foam_numberOfSubdomains;success=$? #Calling the function for creatingthe inside directories 
if [ $success -eq 222 ]; then 
   echo "$insideDir/processor0 already exists inside the ./overlayFSDir/overlay0 file"
   echo "Exiting";exit 1
elif [ $success -ne 0 ]; then 
   echo "Failed creating the inside directories, exiting"
   echo "Exiting";exit 1
fi

Note the use of the createInsideProcessorDirs function
The function receives as arguments: 1) the path inside the OverlayFS files where to create the processor* dirs, and 2) the number of subdomains.
The returned value of the function is saved in the success variable and then checked
Read the definition of the function in ../../A1_auxiliaryScripts/ofContainersOverlayFunctions.sh

#9. Transfer the content of the ./bakDir/bak.processor* directories into the ./overlayFSDir/overlay* files
echo "Copying OpenFOAM the files inside ./bakDir/bak.processor* into the ./overlayFSDir/overlay* files"
copyIntoOverlayII './bakDir/bak.processor${ii}/*' "$insideDir/"'processor${ii}/' "$foam_numberOfSubdomains" "true";success=$? 
if [ $success -ne 0 ]; then 
   echo "Failed creating the inside directories, exiting"
   echo "Exiting";exit 1
fi

Note the use of the copyIntoOverlayII function
The function receives as arguments: 1) the source of the copy, 2) the destination, 3) the number of subdomains and 4) a boolean for replacing content or not
Note the use of single quotes for passing the wildcard ‘*’ to the function without evaluation
Also note the use of single quotes ‘…${ii}…’ in the place where the number of the overlay${ii} (or processor${ii}) is needed
The returned value of the function is saved in the success variable and then checked
Read the definition of the function in ../../A1_auxiliaryScripts/ofContainersOverlayFunctions.sh

#10. Mark the initial conditions time directory as already fully reconstructed
echo "Marking the time directory \"0\" as fully reconstructed"
touch 0/.reconstructDone

A dummy empty and hidden file named .reconstructDone is used to mark those result-times that have been successfully reconstructed
In this case, as the time 0 is originally reconstructed by default, it is marked

#11. List the content of directories inside the ./overlayFSDir/overlay* files
echo "Listing the content in ./overlayFSDir/overlay0 $insideDir/processor0"
srun -n 1 -N 1 singularity exec --overlay ./overlayFSDir/overlay0 $theImage ls -lat $insideDir/processor0/

C.I Steps for dealing with the Overlay setup
Submit the solver script (from the scripts directory)
zeus-1:*-2.4.x> sbatch --reservation=$myReservation C.prepareOverlayFoam.sh 
If a reservation is not available, do not use the option. (Or you can use the debugq: --partition=debugq instead.)
Submitted batch job 4642685 on cluster zeus
Check that the processor* directories were renamed/moved to ./bakDir/bak.processor*:
zeus-1:*-2.4.x> cd run/channel395
zeus-1:channel395> ls ./bakDir
bak.processor0  bak.processor1  bak.processor2  bak.processor3  bak.processor4
All processor* directories have been renamed/moved to ./bakDir/bak.processor*
Check that ./overlayFSDir/overlay* files exist:
zeus-1:channel395> ls ./overlayFSDir
overlay0  overlay1  overlay2  overlay3  overlay4
All ./overlayFSDir/overlay* files exist
Explore the content of one of the overlay files:
zeus-1:channel395> module load singularity
zeus-1:channel395> theImage=/group/singularity/pawseyRepository/OpenFOAM/openfoam-2.4.x-pawsey.sif
zeus-1:channel395> insideDir=/overlayOpenFOAM/run/channel395
zeus-1:channel395> singularity exec --overlay ./overlayFSDir/overlay1 $theImage ls -lat $insideDir/processor1/
total 16
drwxr-s---+ 4 espinosa pawsey0001 4096 May 24 20:38 .
drwxr-s---+ 2 espinosa pawsey0001 4096 May 24 20:38 0
drwxr-s---+ 3 espinosa pawsey0001 4096 May 24 20:38 constant
drwxr-s---+ 3 espinosa pawsey0001 4096 May 24 20:38 ..

D. Executing the solver

The D.runFoam script (main points to be discussed):
#SBATCH --ntasks=5
#5. Defining OpenFOAM controlDict settings for this run
foam_endTime=40
foam_writeInterval=1
foam_purgeWrite=0 #Just for testing in this exercise. In reality this should have a reasonable value if possible
#foam_purgeWrite=10 #Just 10 times will be preserved
#7. Creating soft links towards directories inside the ./overlayFSDir/overlay* files
#These links and directories will be recognized by each mpi instance of the container
#(Initially these links will appear broken as they are pointing towards the interior of the ./overlayFSDir/overlay* files.
# They will only be recognized within the containers)
pointToOverlay $insideDir $foam_numberOfSubdomains;success=$? #Calling function to point towards the interior
if [ $success -ne 0 ]; then
   echo "Failed creating the soft links"
   echo "Exiting";exit 1
fi
Note the use of the pointToOverlay function

The function receives as arguments: 1) the path inside the OverlayFS files where to create the processor* dirs, and 2) the number of subdomains.

The returned value of the function is saved in the success variable and then checked

Read the definition of the function in ../../A1_auxiliaryScripts/ofContainersOverlayFunctions.sh
#8. Execute the case 
echo "About to execute the case"
srun -n $SLURM_NTASKS -N $SLURM_JOB_NUM_NODES bash -c "singularity exec --overlay ./overlayFSDir/"'overlay${SLURM_PROCID}'" $theImage pimpleFoam -parallel 2>&1" | tee $logsDir/log.pimpleFoam.$SLURM_JOBID
echo "Execution finished"
VERY IMPORTANT: Note that the singularity command is called inside a bash -c command

This is the way we allow each MPI task to pick a different ./overlayFSDir/overlay* file through the SLURM_PROCID variable

Here, SLURM_PROCID is slurm environment variable which needs to be evaluated when executing the container, so we use the section in single quotes '...' to allow the internal evaluation of that variable

Here, theImage is not a global environment variable, is evaluated by the host shell in a section with double quotes "..." at the command line

The total string passed to bash -c is the concatenation of two doble quotes sections with a single quotes section in between
#10. List the existing times inside the ./overlayFSDir/overlay0 
echo "Listing the available times inside ./overlayFSDir/overlay0"
srun -n 1 -N 1 singularity exec --overlay ./overlayFSDir/overlay0 $theImage ls -lat processor0/

D.I Steps for dealing with the solver

Submit the solver script (from the scripts directory)
```
zeus-1:*-2.4.x> sbatch --reservation=$myReservation D.runFoam.sh 
```
- If a reservation is not available, do not use the option. (Or you can use the debugq: --partition=debugq instead.)
```
Submitted batch job 4632685 on cluster zeus
```

Check that the solver is running:

zeus-1:*-2.4.x> squeue -u $USER

JOBID    USER     ACCOUNT     PARTITION            NAME EXEC_HOST ST     REASON   START_TIME     END_TIME  TIME_LEFT NODES   PRIORITY
4632685  espinosa pawsey0001  workq        D.runFoam.sh       n/a PD  Resources     17:09:28     17:19:28      10:00     1      75190 

Observe the output of the job with tail -f at runtime (use <Ctrl-C> to exit the command):

zeus-1:*-2.4.x> tail -f slurm-4632685.out

.
.
.
Time = 0.2
   
PIMPLE: iteration 1
smoothSolver:  Solving for Ux, Initial residual = 0.0118746, Final residual = 1.89249e-06, No Iterations 3
smoothSolver:  Solving for Uy, Initial residual = 0.0617212, Final residual = 1.68113e-06, No Iterations 4
smoothSolver:  Solving for Uz, Initial residual = 0.0589944, Final residual = 9.70923e-06, No Iterations 3
Pressure gradient source: uncorrected Ubar = 0.13369, pressure gradient = -0.000964871
GAMG:  Solving for p, Initial residual = 0.213844, Final residual = 0.00414884, No Iterations 2
time step continuity errors : sum local = 5.82807e-06, global = -1.41211e-19, cumulative = -1.41211e-19
Pressure gradient source: uncorrected Ubar = 0.133687, pressure gradient = -0.000947989
GAMG:  Solving for p, Initial residual = 0.0222643, Final residual = 4.30412e-07, No Iterations 7
time step continuity errors : sum local = 5.63638e-10, global = -2.40486e-19, cumulative = -3.81697e-19
Pressure gradient source: uncorrected Ubar = 0.133687, pressure gradient = -0.000947874
ExecutionTime = 0.25 s  ClockTime = 0 s
.
.
.

You can see in the case directory that now there are several processor* soft links

zeus-1:*-2.4.x> cd run/channel395
zeus-1:channel395> ls -lat 

total 5242952
drwxr-s---+ 12 espinosa pawsey0001       4096 May 25 14:55 .
drwxrws---+  4 espinosa pawsey0001       4096 May 25 14:55 logs
lrwxrwxrwx   1 espinosa pawsey0001         42 May 25 14:55 processor0 -> /overlayOpenFOAM/run/channel395/processor0
lrwxrwxrwx   1 espinosa pawsey0001         42 May 25 14:55 processor1 -> /overlayOpenFOAM/run/channel395/processor1
lrwxrwxrwx   1 espinosa pawsey0001         42 May 25 14:55 processor2 -> /overlayOpenFOAM/run/channel395/processor2
lrwxrwxrwx   1 espinosa pawsey0001         42 May 25 14:55 processor3 -> /overlayOpenFOAM/run/channel395/processor3
lrwxrwxrwx   1 espinosa pawsey0001         42 May 25 14:55 processor4 -> /overlayOpenFOAM/run/channel395/processor4
drwxr-s---+  2 espinosa pawsey0001       4096 May 25 14:55 system
drwxrws---+  4 espinosa pawsey0001       4096 May 25 14:53 bakDir
drwxrws---+  4 espinosa pawsey0001       4096 May 25 14:53 overlayFSDir
drwxr-s---+  2 espinosa pawsey0001       4096 May 25 14:53 0
drwxr-s---+  3 espinosa pawsey0001       4096 May 25 14:53 constant
drwxrws---+  3 espinosa pawsey0001       4096 May 25 14:03 ..
drwxr-s---+  2 espinosa pawsey0001       4096 May 25 14:03 0.org
-rwxrwx---+  1 espinosa pawsey0001        483 May 25 14:03 Allrun

The processor* soft links are pointing to the directory structure that “lives” inside the ./overlayFSDir/overlay* files

zeus-1:channel395> ls -la processor1/

ls: cannot access 'processor1/': No such file or directory

The host shell cannot read the directory structure inside the ./overlayFSDir/overlay* files, and that is why the links appear broken

Check that the solver gave some results by listing the interior of an overlay file:

zeus-1:channel395> module load singularity
zeus-1:channel395> theImage=/group/singularity/pawseyRepository/OpenFOAM/openfoam-2.4.x-pawsey.sif
zeus-1:channel395> singularity exec --overlay ./overlayFSDir/overlay1 $theImage ls processor1/

  10.2  12.4  14.6  16.8  19    20.2  22.4  24.6  26.8  29	 30.2  32.4  34.6  36.8  39    5    7.2  9.4
2  10.4  12.6  14.8  17    19.2  20.4  22.6  24.8  27    29.2  30.4  32.6  34.8  37	 39.2  5.2  7.4  9.6
4  10.6  12.8  15    17.2  19.4  20.6  22.8  25    27.2  29.4  30.6  32.8  35    37.2  39.4  5.4  7.6  9.8
6  10.8  13	 15.2  17.4  19.6  20.8  23    25.2  27.4  29.6  30.8  33    35.2  37.4  39.6  5.6  7.8  constant
8  11    13.2  15.4  17.6  19.8  21	 23.2  25.4  27.6  29.8  31    33.2  35.4  37.6  39.8  5.8  8
  11.2  13.4  15.6  17.8  2	   21.2  23.4  25.6  27.8  3	 31.2  33.4  35.6  37.8  4     6    8.2
2  11.4  13.6  15.8  18    2.2   21.4  23.6  25.8  28    3.2	 31.4  33.6  35.8  38	 4.2   6.2  8.4
4  11.6  13.8  16    18.2  2.4   21.6  23.8  26    28.2  3.4	 31.6  33.8  36    38.2  4.4   6.4  8.6
6  11.8  14	 16.2  18.4  2.6   21.8  24    26.2  28.4  3.6	 31.8  34    36.2  38.4  4.6   6.6  8.8
8  12    14.2  16.4  18.6  2.8   22	 24.2  26.4  28.6  3.8	 32    34.2  36.4  38.6  4.8   6.8  9
 12.2  14.4  16.6  18.8  20    22.2  24.4  26.6  28.8  30	 32.2  34.4  36.6  38.8  40    7    9.2

(In this example the final time was set to be 40 to allow for the creation of more results)

You should also check for success/errors in:
- the slurm output file: slurm-<SLURM_JOBID>.out
- the log files created when executing the OpenFOAM tools in: ./run/channel395/logs/run/

E. Reconstruction

No, unfortunately a container cannot mount more than 1 OverlayFS file at the same time

Yes, this implies that the results need to be copied back to the host file system before reconstruction

This is the inverse operation to the process of copying the initial decomposition into the OverlayFS files (explained at the beginning of this episode)

But in order to avoid the presence of many files in the host, this should be done by small batches:

Copy small batch of results from the interior to the bak.processor* directories

Now create processor* soft links to point to bak.processor* directories and not to the OverlayFS interior

Reconstruct that small batch

Remove the reconstructed result-times from the bak.processor* directories

Continue the cycle in 1. again until postprocessing all the result-times needed

The E.reconstructFromOverlay.sh script (main points to be discussed):
#SBATCH --ntasks=4 #Several tasks will be used for copying files. (Independent from the numberOfSubdomains)
#4. Create the reconstruction array, intended times to be reconstructed are set with the reconstructTimes var
#These formats are the only accepted by function "generateReconstructArray" (check the function definition for further information)
#reconstructTimes="all"
#reconstructTimes="-1"
#reconstructTimes="20"
#reconstructTimes="50,60,70,80,90"
reconstructTimes="0:10"
unset arrayReconstruct #This global variable will be re-created in the function below
generateReconstructArray "$reconstructTimes" "$insideDir";success=$? #Calling fucntion to generate "arrayReconstruct"
if [ $success -ne 0 ]; then
   echo "Failed creating the arrayReconstruct"
   echo "Exiting";exit 1
fi
Note the use of the generateReconstructArray function

The function receives as arguments: 1)the string defining the result-times to be reconstructed, and 2) the path inside the OverlayFS files where to create the processor* dirs

The returned value of the function is saved in the success variable and then checked

The GLOBAL array arrayReconstruct is unset before calling the function, and the function will create a new one with the result-times to be reconstructed

Read the definition of the function in ../../A1_auxiliaryScripts/ofContainersOverlayFunctions.sh
#5. Point the soft links to the ./bakDir/bak.processor* directories
pointToBak $foam_numberOfSubdomains;success=$? #Calling function to point towards the bak.processors
if [ $success -ne 0 ]; then
   echo "Failed creating the soft links"
   echo "Exiting";exit 1
fi
Note the use of the pointToBak function

The function receives as argument the number of subdomains.

The returned value of the function is saved in the success variable and then checked

Read the definition of the function in ../../A1_auxiliaryScripts/ofContainersOverlayFunctions.sh
maxTimeTransfersFromOverlays=10
This variable sets the size of the batches to be postprocessed
The following sections are executed inside a loop (not shown here) as many times needed to postprocess all the result-times required batch by batch:
  ## 9. Copy from the ./overlayFSDir/overlay* the full batch into ./bakDir/bak.processor*
  unset arrayCopyIntoBak
  arrayCopyIntoBak=("${hereToDoReconstruct[@]}")
  replace="true"
  copyResultsIntoBak "$insideDir" "$foam_numberOfSubdomains" "$replace" "${arrayCopyIntoBak[@]}";success=$? #Calling the function to copy time directories into bak.processor*
  if [ $success -ne 0 ]; then
     echo "Failed transferring files into bak.processor* directories"
     echo "Exiting";exit 1
  fi
The array hereToDoReconstruct has the result-times to be processed in the current batch

Note the use of the copyResultsIntoBak function

The function receives as arguments: 1) the path inside the overlays, 2) the number of subdomains, 3) the indication for replacing or not already existing result-times in the bak directories and 4) the array with the result-times to process.

The returned value of the function is saved in the success variable and then checked

Read the definition of the function in ../../A1_auxiliaryScripts/ofContainersOverlayFunctions.sh
  ## 10. Reconstruct all times for this batch.
  echo "Start reconstruction"
  logFileHere=$logsDir/log.reconstructPar.${SLURM_JOBID}_${hereToDoReconstruct[0]}-${hereToDoReconstruct[-1]}
  srun -n 1 -N 1 singularity exec $theImage reconstructPar -time ${timeString} 2>&1 | tee $logFileHere
The command in the srun line executes the reconstruction of the batch

The variable timeString has the list of the result-times to be reconstructed in each batch. This was set in a previous step (in step ## 8., not shown here)

The output of the reconstruction is saved in the file logFileHere (this file needs to be checked when reconstruction errors happen)
For more details of the logic of the loop refer to the full script itself

E.I Steps for dealing with reconstruction:
Submit the reconstruction script (from the scripts directory)
zeus-1:*-2.4.x> sbatch --reservation=$myReservation E.reconstructFromOverlay.sh 
If a reservation is not available, do not use the option. (Or you can use the debugq: --partition=debugq instead.)
Submitted batch job 4632899 on cluster zeus
Check that the reconstruction is being performed:
zeus-1:*-2.4.x> cd run/channel395/
zeus-1:channel395> watch ls ./bakDir/bak.processor0/ 
The command watch executes the ls of the content of ./bakDir/bak.processor0 every 2 seconds
Every 2.0s: ls ./bakDir/bak.processor0/                                     Mon Jun 15 18:33:16 2020
   
0
0.2
0.4
0.6
0.8
1
1.2
constant
You can watch the progress of the first batch being copied to the ./bakDir/bak.processor0 directory

The same is happening for the resto of the ./bakDir/bak.processor* directories

In this case maxTimeTransfersFromOverlays=10 (set within the script) is the size of the batches

After the copy of the first batch is finished, those result-times will be reconstructed

After succesful reconstruction, those result-times will be removed from the host file system
Every 2.0s: ls ./bakDir/bak.processor0/                                     Mon Jun 15 18:35:36 2020
   
0
2.2
2.4
2.6
constant
A few minutes later, the second batch is being copied to the bak.processor* directories

Note that the first batch of files have been removed already from the system
Every 2.0s: ls ./bakDir/bak.processor0/                                    Mon Jun 15 18:39:16 2020
   
0
10
constant
When finished, the earliest and the latest result-times were kept in the bak.processor* directories (although this can be modified within the scripts if desired)
<CTRL>-C (to exit the watch command)
Check for the existence of the reconstructed times:
zeus-1:channel395> ls
0    0.org  1.6  2.6  3.6  4.6  5.6  6.6  7.6  8.6  9.6             bak.processor2  overlay0  processor0  system
0.2  1      1.8  2.8  3.8  4.8  5.8  6.8  7.8  8.8  9.8             bak.processor3  overlay1  processor1
0.4  10     2    3    4    5    6    7    8    9    Allrun          bak.processor4  overlay2  processor2
0.6  1.2    2.2  3.2  4.2  5.2  6.2  7.2  8.2  9.2  bak.processor0  constant        overlay3  processor3
0.8  1.4    2.4  3.4  4.4  5.4  6.4  7.4  8.4  9.4  bak.processor1  logs            overlay4  processor4
You should also check for success/errors in:

the slurm output file: slurm-<SLURM_JOBID>.out

the log files created when executing the OpenFOAM tools in: ./run/channel395/logs/post/ (check the command for the reconstructPar command for understanding the naming convention of the log files. In this case, they contain the SLURM_JOBID and the first-last result-times reconstructed in each batch.

F. Extract from Overlay into Bak; and G. Reconstruct From Bak

F.G.I These scripts are left for the user to try by themselves

F.extractFromOverlayIntoBak.sh is for extracting a batch of result-times from the overlay files (no reconstruction)

G.reconstructFromBak.sh is for reconstructing existing results in the bak.processor* directories

Z. Further notes on how to use OpenFOAM and OpenFOAM containers at Pawsey

More on OverlayFS for singularity in: https://sylabs.io/guides/3.5/user-guide/persistent_overlays.html

The usage of OpenFOAM and OpenFOAM containers at Pawsey has already been described in our documentation: OpenFOAM documentation at Pawsey

and in a technical newsletter note: https://support.pawsey.org.au/documentation/display/US/Pawsey+Technical+Newsletter+2020-04

Key Points

No, unfortunately a container cannot mount more than 1 OverlayFS file at the same time

Yes, this implies that the results need to be copied back to the host file system before reconstruction

In order to avoid the presence of many files in the host, this should be done by small batches - 1.Copy small batch of results from the interior of the ./overlayFSDir/overlay* files towards the ./bakDir/bak.processor* directories in the host file system - 2.Now create processor* soft links to point to ./bakDir/bak.processor* directories and not to the directory structure inside the OverlayFS files - 3.Reconstruct that small batch - 4.Remove the decomposed result-times from the ./bakDir/bak.processor* directories. Only the fully reconstructed result-times are kept in the host. And the original decomposed results are only kept inside the OverlayFS files. - 5.Continue the cycle until postprocessing all the result-times needed

OpenFOAM containers at Pawsey

Basic information for OpenFOAM containers at Pawsey

Overview

Meaning of icons and colours (just for the context of this tutorial)

Series of steps (commands) to be typed in users terminal (guided by instructor)

Series of steps (commands) to be typed in users terminal by themselves (in breakout room)

Collapsed blocks, with additional information, important parts of the scripts or pre-executed steps

The rest are just blocks …

… with general information

A. OpenFOAM containers maintained by Pawsey

OpenFOAM Singularity images maintained by Pawsey are in the following directory:

A.I List the content of our repository

A.II Perform a basic mini-test

Yes, you can also use your own containers

B. OpenFOAM containers need MPICH to run on Crays

Singularity Hybrid Mode for MPI applications

Why do we prefer Hybrid-mode?

Why MPICH?

Main drawback:

B.I Briefly check the settings at Pawsey

Key Points

Executing the full workflow of a case

Overview

0. Introduction

Typical workflow

0.I Accessing the scripts for this episode

Sections and scripts for this episode

So how will this episode flow?

A. Extraction of the tutorial: channel395

The A.extractTutorial.sh script

Main command in the script:

Other important parts of the script:

A.I Steps for dealing with the extraction of the “channel395” case: - [Pre-Executed]

B. Adapt the case to your needs (and Pawsey best practices)

The B.adaptCase.sh script

Main command in the script

Other important parts of the script:

B.I Initial steps for dealing with the adaptation of the case - [Pre-Executed]

B.II Final steps. To check the adapted settings:

The settings that were adapted in ..../run/channel395/system/controlDict

C. Decomposition

The C.decomposeFoam.sh script

Main command in the script:

Other important parts of the script:

C.I Steps for dealing with decomposition:

If you do not have a reservation

D. Executing the solver

The D.runFoam.sh script

Main command in the script:

Other important parts of the script:

D.I Steps for dealing with the solver

E. Reconstruction

The E.reconstructFoam.sh script

Main command in the script:

Other important parts of the script:

E.1 Steps for dealing with reconstruction:

Y. The collated option

Information about the fileHandler collated option

Z. Further notes on how to use OpenFOAM and OpenFOAM containers at Pawsey

Key Points

Compile and execute user's own tools

Overview

0. Introduction

Users own tools

0.I Accessing the scripts for this episode

Sections and scripts for this episode

So how will this episode flow?

A. Cloning a standard solver into user’s own solver myPimpleFoam

The A.cloneMyPimpleFoam.sh script

Main command in the script:

Other important parts of the script:

A.I Initial steps for dealing with this section - [Pre-Executed]

A.II Final steps. To check the source code and important settings for the compilation:

A.III The FOAM_USER_APPBIN, the WM_PROJECT_USER_DIR and other OpenFOAM environment variables

The bash -c command and the single quotes

B. Compilation of myPimpleFoam

The binding

The B.compileMyPimpleFoam.sh script

Main command in the script

Other important parts of the script:

The `A.extractTutorial.sh` script

The `B.adaptCase.sh` script

The settings that were adapted in `..../run/channel395/system/controlDict`

The `C.decomposeFoam.sh` script

The `D.runFoam.sh` script

The `E.reconstructFoam.sh` script

Information about the `fileHandler collated` option

A. Cloning a standard solver into user’s own solver `myPimpleFoam`

The `A.cloneMyPimpleFoam.sh` script

A.III The `FOAM_USER_APPBIN`, the `WM_PROJECT_USER_DIR` and other OpenFOAM environment variables

The `bash -c` command and the single quotes

The `B.compileMyPimpleFoam.sh` script

The `C.extractTutorial.sh` script (main parts to be discussed):

The `D.adaptCase.sh` script (main parts to be discussed):

The settings that were adapted in `..../run/channel395/system/controlDict`

The `E.decomposeFoam.sh` script (main parts to be discussed):

The `F.runFoam.sh` script

C.I Steps for building our first image based on `pawsey/mpich-base`

Main sections of the `Dockerfile.01` to discuss:

The `Dockerfile.02` file (main parts for discussion)

The `RUN StopHere` trick in the `Dockerfile.02`:

Compilation sections in the `Dockerfile` and the `bashrc` file

The `Singularity.def` file:

The `A.extractAndAdaptTutorial.sh` script (main parts to be discussed):

The `imageSettingsSingularity.sh` script (main sections to be discussed):

The `caseSettingsFoam.sh` script (main sections to be discussed):