NEMO on ARCHERΒΆ
QuickStart Guide to setting up NEMO v3.6 and XIOS on ARCHER:
login.archer.ac.uk: Access to the ARCHER HPC facility is via the login nodes. There are 8 login nodes that are automatically assigned.
Compiling the code:
At present this has been successfully acheived using the INTEL compilers (CRAY instructions to follow):
First set up the correct modules (can be added to ~/.bashrc):
module swap PrgEnv-cray PrgEnv-intel
module load cray-hdf5-parallel
module load cray-netcdf-hdf5parallel
Then obtain the code (the HEAD at time of testing was revision 6800, drop the @6800 if you want the most up-to-date version):
cd "my_chosen_dir"
mkdir NEMO
cd NEMO
svn co http://forge.ipsl.jussieu.fr/nemo/svn/trunk@6800
cd ../
svn co http://forge.ipsl.jussieu.fr/ioserver/svn/XIOS/branchs/xios-1.0@703
Next step is to compile the XIOS libraries:
cd xios-1.0/arch
Add the following code to a new file called arch-ARCHER_INTEL.path:
NETCDF_INCDIR="-I $NETCDF_INC_DIR"
NETCDF_LIBDIR='-Wl,"--allow-multiple-definition" -Wl,"-Bstatic" -L $NETCDF_LIB_DIR'
NETCDF_LIB="-lnetcdf -lnetcdff"
MPI_INCDIR=""
MPI_LIBDIR=""
MPI_LIB=""
HDF5_LIBDIR="-L $HDF5_LIB_DIR"
HDF5_LIB="-lhdf5_hl -lhdf5 -lz"
OASIS_INCDIR=""
OASIS_LIBDIR=""
OASIS_LIB=""
The following to arch-ARCHER_INTEL.fcm:
%CCOMPILER CC
%FCOMPILER ftn
%LINKER ftn -nofor-main
%BASE_CFLAGS
%PROD_CFLAGS -O3 -D BOOST_DISABLE_ASSERTS
%DEV_CFLAGS -g -traceback
%DEBUG_CFLAGS -DBZ_DEBUG -g -traceback -fno-inline
%BASE_FFLAGS -D__NONE__
%PROD_FFLAGS -O2
%DEV_FFLAGS -g -O2 -traceback
%DEBUG_FFLAGS -g -traceback
%BASE_INC -D__NONE__
%BASE_LD -lstdc++
%CPP CC -EP
%FPP cpp -P
%MAKE gmake
And finally to arch-ARCHER_INTEL.env:
export HDF5_INC_DIR=${HDF5_DIR}/include
export HDF5_LIB_DIR=${HDF5_DIR}/lib
export NETCDF_INC_DIR=${NETCDF_DIR}/include
export NETCDF_LIB_DIR=${NETCDF_DIR}/lib
To compile:
cd ../
./make_xios --full --prod --arch ARCHER_INTEL --jobs 6
Now to compile the NEMO code:
cd ../NEMO/trunk/NEMOGCM/ARCH
Add a new architecture file arch-ARCHER_INTEL.fcm, being careful to replace the XIOS path entry to point to your newly compiled XIOS path:
# compiler options for Archer CRAY XC-30 (using intel compiler)
#
# NCDF_HOME root directory containing lib and include subdirectories for netcdf4
# HDF5_HOME root directory containing lib and include subdirectories for HDF5
# XIOS_HOME root directory containing lib for XIOS
# OASIS_HOME root directory containing lib for OASIS
#
# NCDF_INC netcdf4 include file
# NCDF_LIB netcdf4 library
# XIOS_INC xios include file (taken into accound only if key_iomput is activated)
# XIOS_LIB xios library (taken into accound only if key_iomput is activated)
# OASIS_INC oasis include file (taken into accound only if key_oasis3 is activated)
# OASIS_LIB oasis library (taken into accound only if key_oasis3 is activated)
#
# FC Fortran compiler command
# FCFLAGS Fortran compiler flags
# FFLAGS Fortran 77 compiler flags
# LD linker
# LDFLAGS linker flags, e.g. -L<lib dir> if you have libraries
# FPPFLAGS pre-processing flags
# AR assembler
# ARFLAGS assembler flags
# MK make
# USER_INC complete list of include files
# USER_LIB complete list of libraries to pass to the linker
#
# Note that:
# - unix variables "$..." are accpeted and will be evaluated before calling fcm.
# - fcm variables are starting with a % (and not a $)
#
%NCDF_HOME $NETCDF_DIR
%HDF5_HOME $HDF5_DIR
%XIOS_HOME "path_to_xios"
%NCDF_INC -I%NCDF_HOME/include -I%HDF5_HOME/include
%NCDF_LIB -L%HDF5_HOME/lib -L%NCDF_HOME/lib -lnetcdff -lnetcdf -lhdf5_hl -lhdf5 -lz
%XIOS_INC -I%XIOS_HOME/inc
%XIOS_LIB -L%XIOS_HOME/lib -lxios
%CPP cpp
%FC ftn
%FCFLAGS -integer-size 32 -real-size 64 -O3 -fp-model source -zero -fpp -warn all
%FFLAGS -integer-size 32 -real-size 64 -O3 -fp-model source -zero -fpp -warn all
%LD CC -Wl,"--allow-multiple-definition"
%FPPFLAGS -P -C -traditional
%LDFLAGS
%AR ar
%ARFLAGS -r
%MK gmake
%USER_INC %XIOS_INC %NCDF_INC
%USER_LIB %XIOS_LIB %NCDF_LIB
Finally to compile the GYRE_XIOS example:
cd ../CONFIG
./makenemo -n XIOS_GYRE -m ARCHER_INTEL
cd GYRE_XIOS/EXP00
ln -s "path_to_xios"/bin/xios_server.exe xios_server.exe
Create a file called runscript.pbs in the EXP00 directory that contains:
#!/bin/bash --login
#PBS -l select=3
#PBS -l walltime=00:20:00
#PBS -A n01-NOCL
export OMP_NUM_THREADS=1
ulimit -s unlimited
XIOSCORES=2
OCEANCORES=9
#===============================================================
# RUN MODEL
#===============================================================
aprun -n $XIOSCORES -N 2 xios_server.exe : -n $OCEANCORES opa
exit
Correct for a bug in the namelist in revision 6800:
sed 's/rn_ahm_0_lap/rn_ahm_0/' namelist_cfg > tmp; mv tmp namelist_cfg
And then submit to the compute nodes:
qsub -q short runscipt.pbs
Hopefully this is a start. More indepth instructions to follow + finer details about submitting aprun commands and the iodef.xml options!