Tutorial : installing WRF on Google Compute Engine Linux / Debian / Ubuntu

This blog entry is part one of a series on using WRF with Google Compute Engine with Debian or Ubuntu Linux Distributions.

Spinning up an instance

Using the Developers Console

The Google Developers Console walks you through configuring your machine. There are three primary sections that you’re going to want to modify as illustrated:

Instance name

Location and resources

Project access

Using the command line

While you can do this using the Developers Console, I prefer to use the command line utility, gcutil. After installing and configuring it with your project, you can create the instance described above with a simple command:

 gcutil --project="<project name>" addinstance "<instance name>" --zone="us-central1-b" --machine_type="n1-standard-2" --service_account_scopes="https://www.googleapis.com/auth/userinfo.email,https://www.googleapis.com/auth/compute,https://www.googleapis.com/auth/devstorage.read_write" --image="https://www.googleapis.com/compute/v1/projects/debian-cloud/global/images/debian-7-wheezy-v20131120"  

Installing utilities on GCE, via package manager

Make sure the Debian packages are uptodate:

 sudo apt-get update  

First installing basics software to use in the VM:

 sudo apt-get install ftp gawk  bzip2 file csh time

WRF must be installed from computer source, so we need to get a piece of software need to:

 sudo apt-get install build-essential g++ gcc gfortran git m4 make byacc flex curl

Now we have to installed some sofware manually

 cd /home/wrf
 mkdir ./Sources ./Data ./Scratch ./Sources/archive-src ./Sources/configuration_systeme

Debian offers libraries required for grib :

 sudo apt-get install zlib1g slib libjasper1 libjasper-dev libpng-dev libjpeg-dev libz-dev libcurl4-openssl-dev

Check if your VM is 64 bits compatible :

$ if [[ $(sed -n '/flags/{/ lm / p;q}' /proc/cpuinfo) ]] ; then echo "Compatible 64 bits" ; else echo "Non-compatible 64 bits" ; fi

Download the most recent sources of HDF-4, HDF-5, NetCDF C and Fortran and Open MPI of the different sites :

HDF4

Il est nécessaire de compiler cette librairie sans les headers netcdf pour ne pas induire de confusion avec ceux de la librairie NetCDF v4.xx

export hdfversion=4.2.10
tar zxvf ../archive-src/hdf-${hdfversion}.tar.gz
cd ../archive-src/hdf-${hdfversion}
export CC=gcc
export CXX=g++
export FC=gfortran
export F77=gfortran
export F90=gfortran
export CFLAGS='-m64 -g'
export FFLAGS='-m64 -g'
export FCFLAGS='-m64 -g'
export CXXFLAGS='-m64 -g'
export CPPFLAGS='-DNDEBUG'
./configure --enable-shared --disable-netcdf --disable-fortran --prefix=/opt/hdf-gfortran
make
sudo make install
cd ../
rm -rf ../archive-src/hdf-${hdfversion}

HDF5

Pas d’installation de SZIP : on verra si ça cause des problèmes plus loin

export hdfversion=1.8.13
tar zxvf ../archive-src/hdf5-${hdfversion}.tar.gz
cd ../archive-src/hdf5-${hdfversion}
export CC=gcc
export CXX=g++
export FC=gfortran
export F77=gfortran
export F90=gfortran
export CFLAGS='-m64 -g'
export FFLAGS='-m64 -g'
export FCFLAGS='-m64 -g'
export CXXFLAGS='-m64 -g'
export CPPFLAGS='-DNDEBUG'
./configure --enable-shared --enable-fortran --enable-cxx --prefix=/opt/hdf5-gfortran
make
sudo make install
cd ../
rm -rf ../archive-src/hdf5-${hdfversion}

NetCDF

Installation in two steps : NetCDF for GCC then Netcdf for GFORTRAN
Before compiling, download the 2 libraries netcdf-X.X.X et netcdf-X.X.X-fortran on http://www.unidata.ucar.edu/software/netcdf

export cdfcversion=4.3.2
export cdffversion=4.4.1
export CC=gcc
export CXX=g++
export FC=gfortran
export F77=gfortran
export F90=gfortran
export CFLAGS='-m64 -g'
export FFLAGS='-m64 -g'
export FCFLAGS='-m64 -g'
export CXXFLAGS='-m64 -g'
export CPPFLAGS='-I/opt/hdf5-gfortran/include -I/opt/hdf-gfortran/include'
export LDFLAGS='-L/opt/hdf5-gfortran/lib -L/opt/hdf-gfortran/lib'

# NetCDF-C
tar zxvf ../archive-src/netcdf-${cdfcversion}.tar.gz 
cd ../archive-src/netcdf-${cdfcversion}
./configure  --prefix=/opt/netcdf-${FC} --enable-netcdf-4 --enable-hdf4 --enable-hdf4-file-tests
make
sudo make install
#make check
cd ../
rm -rf ../archive-src/netcdf-${cdfcversion}

# NetCDF-fortran
export CPPFLAGS='-I/opt/hdf5-gfortran/include -I/opt/netcdf-gfortran/include'
export LDFLAGS='-L/opt/hdf5-gfortran/lib -L/opt/netcdf-gfortran/lib'
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/netcdf-gfortran/lib:/opt/hdf5-gfortran/lib:/opt/hdf-gfortran/lib

tar zxvf ../archive-src/netcdf-fortran-${cdffversion}.tar.gz 
cd ../archive-src/netcdf-fortran-${cdffversion}
./configure  --prefix=/opt/netcdf-${FC}
make
sudo make install
#make check
cd ../
rm -rf ../archive-src/netcdf-fortran-${cdffversion}

NCO

Sous Debian, utiliser la commande d’installation du package :

 sudo apt-get install nco

Pour d’autres distributions, il est nécessaire d’installer depuis le code source:

 export ncoversion=4.4.6
 export CC=gcc
 export CXX=g++
 export FC=gfortran
 export F77=gfortran
 export CFLAGS='-m64 -g'
 export FFLAGS='-m64 -g'
 export FCFLAGS='-m64 -g'
 export CXXFLAGS='-m64 -g'
 export CPPFLAGS='-I/opt/hdf5-gfortran/include -I/opt/netcdf-gfortran/include'
 export LDFLAGS='-L/opt/hdf5-gfortran/lib -L/opt/netcdf-gfortran/lib'
 export NETCDF=/opt/netcdf-gfortran
 export NETCDF_LIB=$NETCDF/lib
 tar zxvf ../archive-src/nco-${ncoversion}.tar.gz
 cd ../archive-src/nco-${ncoversion}
 ./configure --prefix=/opt/nco #--disable-shared --disable-ncap2 --disable-netcdf4 
 make
 sudo make install
 cd ../
 rm -rf ../archive-src/nco-${ncoversion}

OpenMPI

Download sources on http://www.open-mpi.org.

Compilation of Open MPI is made with same compilers as NetCDF. Be sure that Open MPI version has been test with your Debian distribution. Check versions of GNU compilers in https://packages.debian.org and http://www.open-mpi.org.

Example : in Debian 7.7, version of GNU compilers is 4.7.2. Best is to use version 1.6.5 of Open MPI.

 export ompiversion=3.0.0
 export gnu=4.7.2
 export CC=gcc
 export CXX=g++
 export FC=gfortran
 export F77=gfortran
 export CFLAGS='-m64 -g'
 export FFLAGS='-m64 -g'
 export FCFLAGS='-m64 -g'
 export CXXFLAGS='-m64 -g'
 mkdir ../archive-src/
 cd ../archive-src/
 wget https://www.open-mpi.org/software/ompi/v3.0/downloads/openmpi-$ompiversion.tar.gz
 tar zxvf ../archive-src/openmpi-$ompiversion.tar.gz
 cd ../archive-src/openmpi-${ompiversion}
 ./configure --prefix=/opt/openmpi-$ompiversion
 make
 sudo make install

Configuration de l’environnement CONSOLE avant l’installation de WRF

Before installing WRF one needs to copy some lines in the configuration file for Bourne Shell  $HOME/.bashrc :

# Ajout configuration E. Chaxel
export PATH=$PATH:./

# Choix du compilateur 
fortran=gfortran
case ${fortran} in
gfortran)
# gfortran
export FC="gfortran -ffree-form -ffree-line-length-none -fbounds-check"
export F77="gfortran -fbounds-check"
export F90="gfortran -ffree-form -ffree-line-length-none -fbounds-check"
export FCOPT="-O2"
;;
ifort)
export FC=ifort
export F77=ifort
export F90=ifort
export FCOPT="-O2 -xW -fno-alias -prec_div -ip"
;;
esac

# MPI
#export mpiversion=1.8.2
mpitype=openmpi
mpi=/opt/${mpitype}-${fortran}
export PATH=$PATH:${mpi}/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${mpi}/lib

# HDF5
export hdf5=/opt/hdf5-${fortran}
export PATH=$PATH:$hdf5/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$hdf5/lib

# HDF4
export hdf=/opt/hdf-${fortran}
export PATH=$PATH:$hdf/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$hdf/lib

# NETCDF
export NETCDF=/opt/netcdf-${fortran}
export PATH=$PATH:$NETCDF/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$NETCDF/lib

# NCO
export NCO=/opt/nco
export PATH=$PATH:${NCO}/bin

WRF

L’installation de WRF peut commencer :

mkdir -p $HOME/Sources/WRF_V3.6/
cd $HOME/Sources/WRF_V3.6/
tar zxvf $HOME/Sources/archive-src/WRFV3.6.TAR.gz
cd ./WRFV3
export WRFIO_NCD_LARGE_FILE_SUPPORT=1
export NETCDF=/opt/netcdf-gfortran
./configure # choisir l'item 34. x86_64 Linux, gfortran compiler with gcc   (dmpar)

Différents problèmes : consulter http://www2.mmm.ucar.edu/wrf/users/FAQ_files/FAQ_wrf_installation.html

First do a ‘clean -a’, then reconfigure

  • Add -DMPI2_SUPPORT on line « DM_CC = »

Save the modified configure.wrf file and then recompile.

./compile em_real

WPS

Ensuite installer WPS dans $HOME/Sources/WRF_V3.6/

mkdir -p $HOME/Sources/WRF_V3.6/
cd $HOME/Sources/WRF_V3.6/
tar zxvf $HOME/Sources/archive-src/WPSV3.6.TAR.gz
cd ./WPS
./configure 

Choose item 3 :  Linux x86_64, gfortran    (dmpar)

In file configure.wps, modify:

  • « mpif90 -f90=gfortran » by « mpif90 »
  • « mpicc -cc=gcc »  by « mpicc »
  • add -DMPI2_SUPPORT on line « DM_CC = »
  • add « -lgomp » à la section WRF_LIB (if option smpar is used)

COMPRESSION_LIBS = -ljasper -lpng -lz

COMPRESSION_INC = -I/usr/include/libpng -I/usr/include/jasper

Save the modified configure.wps file and then compile using:

./compile

WRF is now installed on your system.

Installing NCL

See the tutorial : Tutorial : installation of NCL (new NCAR Graphics) for WRF vizualisation