Cluster specific settings for flusi

Let us collect cluster specific tips and tricks here to make live easier for eveybody

Turing

FFTW: preinstalled, load it:

  module load fftw/3.3.3
  export FFT_ROOT=/bglocal/cn/pub/FFTW/3.3.3

P3DFFT: FLUSI now relies on the version 2.6.1 of p3dfft. You can find it in the extra-files branch, get it with git checkout extra_files. Now you have p3dfft-dist.2.6.1.tar and p3dfft-dist.2.6.1.patch in your directory. Copy both to ${HOME}/src, extract it tar xf p3dfft-dist.2.6.1.tar Go to the P3DFFT directory and configure it:
```
  ./configure  --prefix=${PWD} --enable-ibm --enable-fftw --with-fftw=${FFT_ROOT} --enable-measure --enable-stride1 LDFLAGS="-lm" FC=mpixlf90_r CC=mpixlc_r 
  unset MAKEFLAGS
  make
  make install
  cd lib
  ln -s ../build/libp3dfft.a
  export P3DFFT_ROOT="${HOME}/src/p3dfft-dist.2.6.1/"
```

Currently, make will fail with errors (during C examples), but that is fine! Everything is okay if you have the file build/libp3dfft.a.

HDF5: is preinstalled, so

  module load hdf5/mpi/1.8.14   
  export HDF_ROOT=/smplocal/pub/HDF5/1.8.14/par/  
  export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/smplocal/pub/HDF5/1.8.14/par/lib

It is necessary to include the HDF lib directory in the LD_LIBRARY_PATH in order to avoid run-time errors for missing objects (that's the last command)

SUMMARY: After installation, in my .bash_profile, the following is appended (for simpler copy/paste):

  export CC=mpixlc_r
  export CXX=mpixlcxx_r
  export MPICXX=mpixlcxx_r
  export FC=mpixlf90_r

  module load fftw/3.3.3
  export FFT_ROOT=/bglocal/cn/pub/FFTW/3.3.3

  export P3DFFT_ROOT="${HOME}/src/p3dfft-dist.2.6.1/"

  module load hdf5/mpi/1.8.9
  export HDF_ROOT=/bglocal/cn/pub/HDF5/1.8.9/par
  export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/smplocal/pub/HDF5/1.8.9/par/lib

  export FFLAGS="-O5"
  export MAKEFLAGS='-j32'

Ada

Ada is the hybrid cluster at IDRIS. Note on ada you do not directly have a .bashrc file
The comilers and the MPI environment are automatically loaded with the module compilerwrappers/yes(default). Before doing anything, set

    export mpif90=ifort
    export FC=mpiifort

to use the ifort compiler. You should add this to the .bashrc.

FFTW: no special remarks, works as specified in the install guide
```
  export FFT_ROOT="${HOME}/src/fftw-3.3.3"
```
P3DFFT:

HDF5: is preinstalled, so

  module load hdf5/mpi/1.8.14  
  module load intel/2017.2

Note you need the newer compiler to use the newer pre-installed HDF. Then set the path for HDF

    export HDF_ROOT=/smplocal/pub/HDF5/1.8.14/par/

It is necessary to include the HDF lib directory in the LD_LIBRARY_PATH in order to avoid run-time errors for missing objects:

    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HDF_ROOT/lib

please note you have to specify that in the jobfiles as well.

FLUSI: set compiler flags:
```
  export FFLAGS="-O3"
```

SUMMARY: After installation, in my .bash_profile, the following is appended (for simpler copy/paste):

  # stuff for FLUSI
  export mpif90=mpiifort
  export FC=mpiifort
  export FFT_ROOT="${HOME}/src/fftw-3.3.3"
  module load hdf5/mpi/1.8.14
  
  # either these two...
  export HDF_ROOT=/smplocal/pub/HDF5/1.8.14/par/
  module load intel/2017.2
  
  # ...or this line
  # export HDF_ROOT=/linkhome/rech/sch/rsch512/src/hdf5-1.8.15-patch1
  
  export FFLAGS="-O3"
  export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HDF_ROOT/lib
  export P3DFFT_ROOT=/linkhome/rech/sch/rsch512/src/p3dfft-pseudo/
  module load lapack95/intel
  export mpi_command="mpirun -n 4"

Job-file

a sample file for IBM's load leveler:

    #!/bin/bash
    #=========== directives ===========
    # @ job_name = NAME
    # @ output = $(job_name).$(jobid).$(step_name)
    # @ error = $(output)
      
    #=========== Step 1 directives ===========
    #======= Sequential preprocessing ========
    # @ step_name = pre
    # @ job_type = serial
    # @ wall_clock_limit = 600
    # @ class = archive
    # @ queue
      
    #=========== Step 2 directives ===========
    #============= Parallel step =============
    # @ step_name = main
    # @ dependency = (pre == 0)
    #   (executed only if previous step completed without error)
    # @ job_type = parallel
    # @ total_tasks = 32
    # @ wall_clock_limit = 10:00:00
    # @ queue
      
    #=========== Step 3 directives ===========
    #======= Sequential postprocessing =======
    # @ step_name = post
    # @ dependency = (main >= 0) && (pre == 0)
    # @ job_type = serial
    # @ class = archive
    # @ wall_clock_limit = 1200
    # @ queue
      
    
    PARAMSFILE=part.ini
    PROGFILE=flusi
    dir_gaya=
    files_input=($PARAMSFILE $PROGFILE particle.in)
    backup=no
    
    case ${LOADL_STEP_NAME} in
      
      #============ Step 1 commands ============
      #======= Sequential preprocessing ========
      pre )
        set -x
    
        if [ ! -f $PROGFILE ]; then
            echo $LOADL_STEP_INITDIR/$PROGFILE "not found!"
            exit
        fi
        if [ ! -f $PARAMSFILE ]; then
            echo $LOADL_STEP_INITDIR/$PARAMSFILE "not found!"
            exit
        fi
    
    	# copy all input files to the work directory
    	for file in "${files_input[@]}"
        do
          cp $LOADL_STEP_INITDIR/$file $tmpdir  
    	done
    
        # symbolic link to workdir
    	rm -f workdir
        ln -s $tmpdir workdir
        echo "working dir: " $tmpdir
        cd $TMPDIR
    
# --------------------------------------------------------------------------------------------- 
# backup resuming goes here
if [ "$backup" == "yes" ]; then
	if [ "$dir_gaya" == "" ]; then
	  echo "no directory on gaya, trying to resume from startdir"
	  cp $LOADL_STEP_INITDIR/*.t ./
	  cp $LOADL_STEP_INITDIR/runtime* ./
	else
	  echo "fetchin input files from ergon..."
	  mfget $dir_gaya/*.t
	  mfget $dir_gaya/runti*
	fi
fi
# ---------------------------------------------------------------------------------------------

echo "files in starting dir:"
        ls -lh
      ;;
      
      #============ Step 2 commands ============
      #============= Parallel step =============
      main )
        set -x
        cd $TMPDIR
    	ls -l
        poe ./$PROGFILE $PARAMSFILE
      ;;
      
      #============ Step 3 commands ============
      #======= Sequential postprocessing =======
      post )
        set -x
        # remove link to workdir
        rm workdir
            
        cd $tmpdir
        echo "files in work dir"
        ls -lh

for file in *
do 
	if [ "$dir_gaya" == "" ]; then
		cp $file $LOADL_STEP_INITDIR/
	else
		mfput ${file} ${dir_gaya}/${file}
	fi
done  
      ;;
      
    esac

Mesocentre

Mesocentre is located in Marseille (nord) Before doing anything, load ifort compiler + MPI environment:

module load intel/12.1 mpich2/intel/1.4.1

FFTW: no special remarks, works as specified in the install guide
```
  export FFT_ROOT="${HOME}/src/fftw-3.3.3"
```

P3DFFT: content of makeinc (in $P3DFFT_ROOT/build):

  DFLAGS = -DFFTW -DESTIMATE -DSTRIDE1
  CC = mpicc
  FF = mpif90 -c -I${FFT_ROOT}/include
  FFLAGS = -O3 
  AR = ar
  ARFLAGS = -v -r -u
  LDFLAGS= -O3 -L${FFT_ROOT}/lib -lfftw3 -lm 
  INCL = -I${FFT_ROOT}/include

then just type make don't forget the include path:

  export P3DFFT_ROOT="${HOME}/src/p3dfft"

To install p3dfft-2.6.1, follow the instructions for Turing. By default, it is compiled with MPICH2. When compiling with OpenMPI, before configuring, type

  grep -n 'lmpichf90' configure.ac
  grep -n 'lmpichf90' configure

and then modify these two files to remove all instances of -lmpichf90.

HDF5: is preinstalled, just type

  export HDF_ROOT="/LOGINSTO/softs/hdf5/intel/1.8.8/"

FLUSI: set compiler flags:
```
  export FFLAGS="-O3"
```

Use ssh with git:

  git remote set-url origin git@github.com:pseudospectators/FLUSI.git

Use LAPACK:

  Do Nothing! Just leave line 85 of makefile ("LDFLAGS += -llapack") as it is

SUMMARY: After installation, in my .bash_profile, the following is appended (for simpler copy/paste):

For use with MPICH

  module load intel/12.1 mpich2/intel/1.4.1       
  export FFT_ROOT="${HOME}/src/fftw-3.3.3"
  export P3DFFT_ROOT="${HOME}/src/p3dfft"
  export HDF_ROOT="/LOGINSTO/softs/hdf5/intel/1.8.8/" 
  export FFLAGS="-O3"

For use with OPENMPI

  module load intel/13.1.2 openmpi/intel/1.6.3 hdf5/intel/1.8.8
  export FFT_ROOT="${HOME}/src/fftw-3.3.3"
  export P3DFFT_ROOT="${HOME}/src/p3dfft-2.6.1-openmpi"
  export HDF_ROOT="/softs/cemracs/selalib/intel"
  export FFLAGS="-O3"
  export PATH="${PATH}:/home/dkolom/mpi2vis"

Job-file

a sample OAR job file for the OARSUB system running on the mesocentre:

For use with MPICH

#!/bin/bash
#OAR -n FLUSI_A
#OAR -p cluster
#OAR -l nodes=1/core=8, walltime=24:00:00
#OAR -O FLUSI.%jobid%.output
#OAR -E FLUSI.%jobid%.ERROR
#OAR --project 13b016
#cat $OAR_NODEFILE
module load mpich2/intel/1.4.1 intel/12.1
#MPI_ARCH_PATH=/softs/openmpi/intel/1.4.5
#/softs/openmpi/intel/1.4.5/bin/mpirun -machinefile $OAR_NODEFILE -n 12 ./main PARAMS_test.ini

/softs/mpich2/intel/1.4.1/bin/mpirun -n 8 ./flusi PARAMS_test_fsi.ini

#------------------------------------
# run: oarsub -S ./job.oar
# note job.oar must be executable
#------------------------------------

For use with OPENMPI

#!/bin/bash
#OAR -n FLUSI_A
#OAR -p cluster
#OAR -q medium
##OAR -l core=32, walltime=48:00:00
#OAR -l nodes=1, walltime=48:00:00
#OAR -O FLUSI.%jobid%.output
#OAR -E FLUSI.%jobid%.ERROR
#OAR --project 14b028
module load intel/13.1.2 hdf5/intel/1.8.8 openmpi/intel/1.6.3
echo running
ulimit -s unlimited

CPUS=$(wc -l ${OAR_NODEFILE} | awk '{print $1}')
echo ${CPUS}
#mpirun -n "${CPUS}" -machinefile "${OAR_NODEFILE}" --bind-to-core -bysocket ./exeflusi.sh
#mpirun -n "${CPUS}" -machinefile "${OAR_NODEFILE}" --bind-to-core -bynode ./exeflusi.sh
mpirun -n "${CPUS}" -machinefile "${OAR_NODEFILE}" --bind-to-core ./exeflusi.sh

#------------------------------------
# run: oarsub -S ./batch.oar
# note job.oar must be executable
#------------------------------------

The content of exeflusi.sh is:

#!/bin/bash
ulimit -s unlimited
./flusi PARAMS.ini

Note that the project name changes every year. Read more about options -bysocket and -bynode on the Mesocentre website.

Unit tests

When OPENMPI is used, the tests should be submitted using OAR. Unit test directory /test and executable files flusi and mhd should be copied to /scratch disk space. Here is a sample OAR script (note that it must be executable):

#!/bin/bash
#OAR -n unit_test
#OAR -p cluster
#OAR -l nodes=1, walltime=10:00:00
#OAR -O FLUSI.%jobid%.output
#OAR -E FLUSI.%jobid%.ERROR
#OAR --project 14b028
#cat $OAR_NODEFILE
module load intel/13.1.2 hdf5/intel/1.8.8 openmpi/intel/1.6.3
echo running
ulimit -s unlimited

export nprocs=4
export mpi_command="mpirun -n ${nprocs} -machinefile "${OAR_NODEFILE}" "
export mpi_serial="mpirun -n 1 -machinefile "${OAR_NODEFILE}" "

./unittest.sh

#------------------------------------
# run: oarsub -S ./batch.oar
# note job.oar must be executable
#------------------------------------

Duke

Is M2P2's home cluster

Sugiton

This is a machine with Fedora 18 and the gfortran compiler.

HDF5 I've simply installed
```
  sudo yum install hdf5-mpich hdf5-mpich-devel
```
There is then no need to set HDF_ROOT.

P3DFFT: the contents of the makeinc are:

  CC = mpicc
  FF = f95 -c -I/home/tommy/fftw/include -I/usr/include/mpich2-x86_64
  FFLAGS = -O3 
  AR = ar
  ARFLAGS = -v -r -u
  LDFLAGS= -O3 -L/home/tommy/fftw/lib -lfftw3 -lm 
  INCL = -I/home/tommy/fftw/include

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cluster specific settings for flusi

Turing

Ada

Job-file

Mesocentre

Job-file

Unit tests

Duke

Sugiton

Clone this wiki locally