This blog entry is part one of a series on using WRF with Google Compute Engine with Debian or Ubuntu Linux Distributions.
Spinning up an instance
Using the Developers Console
The Google Developers Console walks you through configuring your machine. There are three primary sections that you’re going to want to modify as illustrated:
Instance name
Location and resources
Project access
Using the command line
While you can do this using the Developers Console, I prefer to use the command line utility, gcutil. After installing and configuring it with your project, you can create the instance described above with a simple command:
gcutil --project="<project name>" addinstance "<instance name>" --zone="us-central1-b" --machine_type="n1-standard-2" --service_account_scopes="https://www.googleapis.com/auth/userinfo.email,https://www.googleapis.com/auth/compute,https://www.googleapis.com/auth/devstorage.read_write" --image="https://www.googleapis.com/compute/v1/projects/debian-cloud/global/images/debian-7-wheezy-v20131120"
Installing utilities on GCE, via package manager
Make sure the Debian packages are uptodate:
sudo apt-get update
First installing basics software to use in the VM:
sudo apt-get install ftp gawk bzip2 file csh time
WRF must be installed from computer source, so we need to get a piece of software need to:
sudo apt-get install build-essential g++ gcc gfortran git m4 make byacc flex curl
Now we have to installed some sofware manually
cd /home/wrf
mkdir ./Sources ./Data ./Scratch ./Sources/archive-src ./Sources/configuration_systeme
Debian offers libraries required for grib :
sudo apt-get install zlib1g slib libjasper1 libjasper-dev libpng-dev libjpeg-dev libz-dev libcurl4-openssl-dev
Check if your VM is 64 bits compatible :
$ if [[ $(sed -n '/flags/{/ lm / p;q}' /proc/cpuinfo) ]] ; then echo "Compatible 64 bits" ; else echo "Non-compatible 64 bits" ; fi
Download the most recent sources of HDF-4, HDF-5, NetCDF C and Fortran and Open MPI of the different sites :
- HDF-4 &HDF-5: https://support.hdfgroup.org/products/
- NetCDF C and Fortran: https://www.unidata.ucar.edu/downloads/netcdf/
- Open MPI: https://www.open-mpi.org/software/ompi
HDF4
Il est nécessaire de compiler cette librairie sans les headers netcdf pour ne pas induire de confusion avec ceux de la librairie NetCDF v4.xx
export hdfversion=4.2.10
tar zxvf ../archive-src/hdf-${hdfversion}.tar.gz
cd ../archive-src/hdf-${hdfversion}
export CC=gcc
export CXX=g++
export FC=gfortran
export F77=gfortran
export F90=gfortran
export CFLAGS='-m64 -g'
export FFLAGS='-m64 -g'
export FCFLAGS='-m64 -g'
export CXXFLAGS='-m64 -g'
export CPPFLAGS='-DNDEBUG'
./configure --enable-shared --disable-netcdf --disable-fortran --prefix=/opt/hdf-gfortran
make
sudo make install
cd ../
rm -rf ../archive-src/hdf-${hdfversion}
HDF5
Pas d’installation de SZIP : on verra si ça cause des problèmes plus loin
export hdfversion=1.8.13
tar zxvf ../archive-src/hdf5-${hdfversion}.tar.gz
cd ../archive-src/hdf5-${hdfversion}
export CC=gcc
export CXX=g++
export FC=gfortran
export F77=gfortran
export F90=gfortran
export CFLAGS='-m64 -g'
export FFLAGS='-m64 -g'
export FCFLAGS='-m64 -g'
export CXXFLAGS='-m64 -g'
export CPPFLAGS='-DNDEBUG'
./configure --enable-shared --enable-fortran --enable-cxx --prefix=/opt/hdf5-gfortran
make
sudo make install
cd ../
rm -rf ../archive-src/hdf5-${hdfversion}
NetCDF
Installation in two steps : NetCDF for GCC then Netcdf for GFORTRAN
Before compiling, download the 2 libraries netcdf-X.X.X et netcdf-X.X.X-fortran on http://www.unidata.ucar.edu/software/netcdf
export cdfcversion=4.3.2
export cdffversion=4.4.1
export CC=gcc
export CXX=g++
export FC=gfortran
export F77=gfortran
export F90=gfortran
export CFLAGS='-m64 -g'
export FFLAGS='-m64 -g'
export FCFLAGS='-m64 -g'
export CXXFLAGS='-m64 -g'
export CPPFLAGS='-I/opt/hdf5-gfortran/include -I/opt/hdf-gfortran/include'
export LDFLAGS='-L/opt/hdf5-gfortran/lib -L/opt/hdf-gfortran/lib'
# NetCDF-C
tar zxvf ../archive-src/netcdf-${cdfcversion}.tar.gz
cd ../archive-src/netcdf-${cdfcversion}
./configure --prefix=/opt/netcdf-${FC} --enable-netcdf-4 --enable-hdf4 --enable-hdf4-file-tests
make
sudo make install
#make check
cd ../
rm -rf ../archive-src/netcdf-${cdfcversion}
# NetCDF-fortran
export CPPFLAGS='-I/opt/hdf5-gfortran/include -I/opt/netcdf-gfortran/include'
export LDFLAGS='-L/opt/hdf5-gfortran/lib -L/opt/netcdf-gfortran/lib'
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/netcdf-gfortran/lib:/opt/hdf5-gfortran/lib:/opt/hdf-gfortran/lib
tar zxvf ../archive-src/netcdf-fortran-${cdffversion}.tar.gz
cd ../archive-src/netcdf-fortran-${cdffversion}
./configure --prefix=/opt/netcdf-${FC}
make
sudo make install
#make check
cd ../
rm -rf ../archive-src/netcdf-fortran-${cdffversion}
NCO
Sous Debian, utiliser la commande d’installation du package :
sudo apt-get install nco
Pour d’autres distributions, il est nécessaire d’installer depuis le code source:
export ncoversion=4.4.6
export CC=gcc
export CXX=g++
export FC=gfortran
export F77=gfortran
export CFLAGS='-m64 -g'
export FFLAGS='-m64 -g'
export FCFLAGS='-m64 -g'
export CXXFLAGS='-m64 -g'
export CPPFLAGS='-I/opt/hdf5-gfortran/include -I/opt/netcdf-gfortran/include'
export LDFLAGS='-L/opt/hdf5-gfortran/lib -L/opt/netcdf-gfortran/lib'
export NETCDF=/opt/netcdf-gfortran
export NETCDF_LIB=$NETCDF/lib
tar zxvf ../archive-src/nco-${ncoversion}.tar.gz
cd ../archive-src/nco-${ncoversion}
./configure --prefix=/opt/nco #--disable-shared --disable-ncap2 --disable-netcdf4
make
sudo make install
cd ../
rm -rf ../archive-src/nco-${ncoversion}
OpenMPI
Download sources on http://www.open-mpi.org.
Compilation of Open MPI is made with same compilers as NetCDF. Be sure that Open MPI version has been test with your Debian distribution. Check versions of GNU compilers in https://packages.debian.org and http://www.open-mpi.org.
Example : in Debian 7.7, version of GNU compilers is 4.7.2. Best is to use version 1.6.5 of Open MPI.
export ompiversion=3.0.0
export gnu=4.7.2
export CC=gcc
export CXX=g++
export FC=gfortran
export F77=gfortran
export CFLAGS='-m64 -g'
export FFLAGS='-m64 -g'
export FCFLAGS='-m64 -g'
export CXXFLAGS='-m64 -g'
mkdir ../archive-src/
cd ../archive-src/
wget https://www.open-mpi.org/software/ompi/v3.0/downloads/openmpi-$ompiversion.tar.gz
tar zxvf ../archive-src/openmpi-$ompiversion.tar.gz
cd ../archive-src/openmpi-${ompiversion}
./configure --prefix=/opt/openmpi-$ompiversion
make
sudo make install
Configuration de l’environnement CONSOLE avant l’installation de WRF
Before installing WRF one needs to copy some lines in the configuration file for Bourne Shell $HOME/.bashrc :
# Ajout configuration E. Chaxel
export PATH=$PATH:./
# Choix du compilateur
fortran=gfortran
case ${fortran} in
gfortran)
# gfortran
export FC="gfortran -ffree-form -ffree-line-length-none -fbounds-check"
export F77="gfortran -fbounds-check"
export F90="gfortran -ffree-form -ffree-line-length-none -fbounds-check"
export FCOPT="-O2"
;;
ifort)
export FC=ifort
export F77=ifort
export F90=ifort
export FCOPT="-O2 -xW -fno-alias -prec_div -ip"
;;
esac
# MPI
#export mpiversion=1.8.2
mpitype=openmpi
mpi=/opt/${mpitype}-${fortran}
export PATH=$PATH:${mpi}/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${mpi}/lib
# HDF5
export hdf5=/opt/hdf5-${fortran}
export PATH=$PATH:$hdf5/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$hdf5/lib
# HDF4
export hdf=/opt/hdf-${fortran}
export PATH=$PATH:$hdf/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$hdf/lib
# NETCDF
export NETCDF=/opt/netcdf-${fortran}
export PATH=$PATH:$NETCDF/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$NETCDF/lib
# NCO
export NCO=/opt/nco
export PATH=$PATH:${NCO}/bin
WRF
L’installation de WRF peut commencer :
mkdir -p $HOME/Sources/WRF_V3.6/
cd $HOME/Sources/WRF_V3.6/
tar zxvf $HOME/Sources/archive-src/WRFV3.6.TAR.gz
cd ./WRFV3
export WRFIO_NCD_LARGE_FILE_SUPPORT=1
export NETCDF=/opt/netcdf-gfortran
./configure # choisir l'item 34. x86_64 Linux, gfortran compiler with gcc (dmpar)
Différents problèmes : consulter http://www2.mmm.ucar.edu/wrf/users/FAQ_files/FAQ_wrf_installation.html
First do a ‘clean -a’, then reconfigure
- Add -DMPI2_SUPPORT on line « DM_CC = »
Save the modified configure.wrf file and then recompile.
./compile em_real
WPS
Ensuite installer WPS dans $HOME/Sources/WRF_V3.6/
mkdir -p $HOME/Sources/WRF_V3.6/
cd $HOME/Sources/WRF_V3.6/
tar zxvf $HOME/Sources/archive-src/WPSV3.6.TAR.gz
cd ./WPS
./configure
Choose item 3 : Linux x86_64, gfortran (dmpar)
In file configure.wps, modify:
- « mpif90 -f90=gfortran » by « mpif90 »
- « mpicc -cc=gcc » by « mpicc »
- add -DMPI2_SUPPORT on line « DM_CC = »
- add « -lgomp » à la section WRF_LIB (if option smpar is used)
COMPRESSION_LIBS = -ljasper -lpng -lz
COMPRESSION_INC = -I/usr/include/libpng -I/usr/include/jasper
Save the modified configure.wps file and then compile using:
./compile
WRF is now installed on your system.
Installing NCL
See the tutorial : Tutorial : installation of NCL (new NCAR Graphics) for WRF vizualisation