NEMO on MobiusΒΆ
QuickStart Guide to setting up NEMO v3.6 and XIOS on Mobius:
mobius: Access to mobius is via:
ssh mobius
Compiling the code:
At present this has been successfully achieved using the Intel compilers:
First set up the correct modules (can be added to ~/.bashrc):
module purge
module load shared intel/compiler/64/14.0/2013_sp1.3.174 mvapich2/intel/64/2.0b slurm/14.03.0 cluster-tools/7.0
Then obtain the code (the HEAD at time of testing was revision 6800, drop the @6800 if you want the most up-to-date version):
cd "my_chosen_dir"
mkdir NEMO
cd NEMO
svn co http://forge.ipsl.jussieu.fr/nemo/svn/trunk@6800
cd ../
svn co http://forge.ipsl.jussieu.fr/ioserver/svn/XIOS/branchs/xios-1.0@703
Next step is to compile the XIOS libraries:
cd xios-1.0/arch
Add the following code to a new file called arch-mobius_intel.path:
NETCDF_INCDIR="-I$NETCDF_INC_DIR"
NETCDF_LIBDIR='-Wl,"--allow-multiple-definition" -L$NETCDF_LIB_DIR'
NETCDF_LIB="-lnetcdf -lnetcdff "
MPI_INCDIR=""
MPI_LIBDIR=""
MPI_LIB=""
HDF5_INCDIR="-I$HDF5_INC_DIR"
HDF5_LIBDIR="-L$HDF5_LIB_DIR "
HDF5_LIB="-lhdf5_hl -lhdf5 -lz"
OASIS_INCDIR=""
OASIS_LIBDIR=""
OASIS_LIB=""
The following to arch-mobius_intel.fcm:
%CCOMPILER mpicc
%FCOMPILER mpif90
%LINKER mpif90 -nofor-main
%BASE_CFLAGS
%PROD_CFLAGS -O3 -D BOOST_DISABLE_ASSERTS
%DEV_CFLAGS -g -traceback
%DEBUG_CFLAGS -DBZ_DEBUG -g -traceback -fno-inline
%BASE_FFLAGS -D__NONE__
%PROD_FFLAGS -O3
%DEV_FFLAGS -g -O2 -traceback -fp-stack-check -check bounds
%DEBUG_FFLAGS -g -traceback
%BASE_INC -D__NONE__
%BASE_LD -lstdc++
%CPP mpicc -EP
%FPP cpp -P
%MAKE gmake
And finally to arch-mobius_intel.env:
export HDF5_INC_DIR=/login/jdha/utils/hdf5_mob_intel/include
export HDF5_LIB_DIR=/login/jdha/utils/hdf5_mob_intel/lib
export NETCDF_INC_DIR=/login/jdha/utils/netcdf_mob_intel/include
export NETCDF_LIB_DIR=/login/jdha/utils/netcdf_mob_intel/lib
To compile:
cd ../
./make_xios --full --prod --arch mobius_intel --jobs 6
Now to compile the NEMO code:
cd ../NEMO/trunk/NEMOGCM/ARCH
Add a new architecture file arch-mobius_intel.fcm, being careful to replace the %XIOS path entry to point to your newly compiled XIOS path:
# compiler options for Mobius (using intel compiler)
#
# NCDF_HOME root directory containing lib and include subdirectories for netcdf4
# HDF5_HOME root directory containing lib and include subdirectories for HDF5
# XIOS_HOME root directory containing lib for XIOS
# OASIS_HOME root directory containing lib for OASIS
#
# NCDF_INC netcdf4 include file
# NCDF_LIB netcdf4 library
# XIOS_INC xios include file (taken into accound only if key_iomput is activated)
# XIOS_LIB xios library (taken into accound only if key_iomput is activated)
# OASIS_INC oasis include file (taken into accound only if key_oasis3 is activated)
# OASIS_LIB oasis library (taken into accound only if key_oasis3 is activated)
#
# FC Fortran compiler command
# FCFLAGS Fortran compiler flags
# FFLAGS Fortran 77 compiler flags
# LD linker
# LDFLAGS linker flags, e.g. -L<lib dir> if you have libraries
# FPPFLAGS pre-processing flags
# AR assembler
# ARFLAGS assembler flags
# MK make
# USER_INC complete list of include files
# USER_LIB complete list of libraries to pass to the linker
# CC C compiler used to compile conv for AGRIF
# CFLAGS compiler flags used with CC
#
# Note that:
# - unix variables "$..." are accpeted and will be evaluated before calling fcm.
# - fcm variables are starting with a % (and not a $)
#
%NCDF_HOME /login/jdha/utils/netcdf_mob_intel
%HDF5_HOME /login/jdha/utils/hdf5_mob_intel
%XIOS_HOME "path_to_xios"
%NCDF_INC -I%NCDF_HOME/include -I%HDF5_HOME/include
%NCDF_LIB -L%HDF5_HOME/lib -L%NCDF_HOME/lib -lnetcdff -lnetcdf -lhdf5_hl -lhdf5 -lz
%XIOS_INC -I%XIOS_HOME/inc
%XIOS_LIB -L%XIOS_HOME/lib -lxios
%CPP cpp
%FC mpif90 -c -cpp
%FCFLAGS -i4 -r8 -O3 -fp-model source -fpp -warn all
%FFFLAGS -i4 -r8 -O3 -fp-model source -fpp -warn all
%FPPFLAGS -P -C -traditional
%LD mpif90
%LDFLAGS -lstdc++
%AR ar
%ARFLAGS -r
%MK gmake
%USER_INC %XIOS_INC %NCDF_INC
%USER_LIB %XIOS_LIB %NCDF_LIB
Finally to compile the GYRE_XIOS example:
cd ../CONFIG
./makenemo -n GYRE_XIOS -m mobius_intel -j 10
cd GYRE_XIOS/EXP00
ln -s "path_to_xios"/bin/xios_server.exe xios_server.exe
Create a runscript in the EXP00 directory that contains:
#!/bin/bash
# ---------------------------
#===============================================================
# CLUSTER BITS
#===============================================================
#
#SBATCH -J NEMO
#SBATCH -p inter
#SBATCH -N 3
#SBATCH -n 11
module purge
module load shared intel/compiler/64/14.0/2013_sp1.3.174 mvapich2/intel/64/2.0b slurm/14.03.0 cluster-tools/7.0
XIOSCORES=2
OCEANCORES=9
#===============================================================
# GATHER NODE INFORMATION
#===============================================================
hostlist=$(scontrol show hostname $SLURM_JOB_NODELIST)
rm -f hostfile
l=0
for f in $hostlist
do
if [ $l -lt $XIOSCORES ]; then
echo $f':1' >> hostfile
else
echo $f':16' >> hostfile
fi
let l++
done
arr=($hostlist)
echo ${arr[@]:0:$XIOSCORES}
#===============================================================
# RUN MODEL
#===============================================================
# Change to the directory that the job was submitted from
cd $SLURM_SUBMIT_DIR
echo 'In derectory: ' $SLURM_SUBMIT_DIR
export OMP_NUM_THREADS=1
time mpirun -f hostfile -np $XIOSCORES xios_server.exe : -np $OCEANCORES opa
# The use of the hostfile is to distribute the XIOS servers evenly
# if not required try the following instead:
#
# time mpirun -np $XIOSCORES xios_server.exe : -np $OCEANCORES opa
exit
Correct for a bug in the namelist in revision 6800:
sed 's/rn_ahm_0_lap/rn_ahm_0/' namelist_cfg > tmp; mv tmp namelist_cfg
And then submit to the compute nodes:
sbatch runscipt.pbs
Hopefully this is a start. More indepth instructions to follow + finer details about submitting aprun commands and the iodef.xml options!