Compilation Guideline of OpenSeeMP on Linux Machines

Compilation Guideline of OpenSeeMP on Linux Machines

This guide is developed in a systematic manner assuming ActiveTcl, MPICH2 and other packages are not pre-installed in the system. But even if those are available slight modification will enable us to compile OpenSeesMP.

The following tarballs are used to compile OpenSeesMP in this manual. This instruction only works with OpenSees 2.3.2.2 version(svn code – 4985). Package’s version has importance for compatibility issues.

Required Packages

ActiveTcl8.5.11.1.295590-linux-x86_64-threaded.tar.gz
db-5.3.21.NC.tar.gz
lapack-3.4.1.tgz
metis-4.0.3.tar.gz
mpiblacs-patch03.tgz
mpiblacs.tgz –files inside needs to be overwritten by files of mpiblacs-patch03.tgz
mpich2-1.1.tar.gz
MUMPS_4.10.0.tar.gz
openssl-1.0.1c.tar.gz
ParMetis-3.2.0.tar.gz
scalapack-2.0.2.tgz
xblas-1.0.248.tar.gz

You can find the following packages in internet, all of them are free, some of them require an email to the developer.

Some Terminal Commands

Some knowledge of bash commands are necessary, parallel machines often do not have graphical user interface.

mkdir mydir - makes a directory named mydir
chmod 777 a.txt - change write permission of files to full permission
nano a.txt - opens a.txt for editing
cd mydir - gets inside mydir
cd .. - gets out mydir to parent directory
which mpd - checks which mpd is used and returns the location

Some system environment packages

Before compiling any of these packages we need to perform the following tasks - (apt-get command is for ubuntu machines for redhat machines we can use yum install) sudo in and type the following:

mkdir lib
mkdir bin
sudo apt-get update 
sudo apt-get install build-essential
sudo apt-get install subversion
svn co svn://opensees.berkeley.edu/usr/local/svn/OpenSees/trunk@4985 OpenSees

Use-

chmod 777 OpenSees 

To change the editing permission of the directory.

sudo apt-get install make
sudo apt-get install gcc
sudo apt-get install g++
sudo apt-get install gfortran
sudo apt-get install mysql

Compilation of Necessary Packages

After completion of the above mentioned process we need to customize compile and install some packages, which should be done in a ordered manner-

To be systematic and implementation on cluster HPC we will create a directory in home called “PARALLEL”. And put all the tarballs required extracts – in simple named folder like scalapack for sclalapack-2.0.2. Hence we get all packs in one master folder.

The directory structure should be setup as following-

HOME=/home/yourusername

PARALLEL=$(HOME)/PARALLEL

level0 |   level1        |level2

HOME   |   PARALLEL      |blacs
              		 |mpich2
              		 |lapack
             		 |scalapack
             		 |blas
             		 |xblas
             		 |metis
             		 |parmetis
             		 |mumps
              		 |---
Fig – Example directory structure

Install Activetcl 8.5.11 - x86-64 (Requires Admin Access)

Untar the installation package, go inside the directory , type –

install.sh 

By default tcl will be installed in the following directory –

TCLdir=/opt/ActiveTcl-8.5

TCL Directories to remember

TCL_BIN=$(TCLdir)/bin

TCL_INC=-I$(TCLdir)/include

TCL_LIB=-L$(TCLdir)/lib -ltcl8.5 -ltk8.5

We need to remember one thing!! tcl.init file location $(tcldir)/lib/tcl8.5/tcl.int

Install mpich2-1.1 for compatibility issues with mumps

Configuration steps ---

a. Setting prefix/ installation directory – for csh and tcsh:

     ./configure --prefix=/home/yourusername/PARALLEL/mpich2-install |& tee c.txt

for bash and sh:

     ./configure --prefix=/home/yourusername/PARALLEL/mpich2-install 2>&1 | tee c.txt

b. Build MPICH2:

for csh and tcsh:

     make |& tee m.txt

for bash and sh:

     make 2>&1 | tee m.txt

c. Install the MPICH2 commands:

for csh and tcsh:

     make install |& tee mi.txt

for bash and sh:

     make install 2>&1 | tee mi.txt

d. Add the bin subdirectory of the installation directory to your path:

for csh and tcsh:

    setenv PATH=/home/yourusername/PARALLEL/mpich2-install/bin:$PATH

for bash and sh:

    export PATH=/home/yourusername/PARALLEL/mpich2-install/bin:$PATH

Make permanent change to system variable to recognise mpich2. Go to user home (don’t confuse with OpenSees compilation home). Add the line of above section d. at the last line.

chmod 777 .bashrc
nano .bashrc

Run the following command to se wheather system variables are changed or not -

which mpd
which mpiexec
which mpirun

Run this command to check everything is working ok. e. mpd run Begin by placing in your home directory a file named .mpd.conf (/etc/mpd.conf if root), containing the line

     secretword=<secretword>

where <secretword> is a string known only to yourself. It should NOT be your normal Unix password. Make this file readable and writable only by you:

     chmod 600 .mpd.conf

The first sanity check consists of bringing up a ring of one mpd on the local machine, testing one mpd command, and bringing the "ring" down.

mpd &
mpdtrace
mpdallexit

The output of mpdtrace should be the hostname of the machine you are running on. The mpdallexit causes the mpd daemon to exit. If you have problems getting the mpd ring established, see the Installation Guide of MPICH2 for instructions on how to diagnose problems with your system configuration that may be preventing it.

We will notedown the important directory location of MPICH2-

MPIdir=$(HOME)/PARALLEL/mpich2-install

MPI_BIN=$(MPIdir)/bin

MPI_INC=-I$(MPIdir)/include

MPI_LIB=-L$(MPIdir)/lib

Use the full directory of all the address of string like $(HOME), $(MPIdir) as system will not recognise such strings used in following 3 to 12 step. After editing the makefiles run relevant make commands for each individual packages.

Install BLAS

Edit the make makefile –

PLAT =
FORTRAN  = $(MPI_BIN)/mpif90
LOADER   = $(MPI_BIN)/mpif90

Install XBLAS

Edit the make makefile –

CC = $(MPI_BIN)/mpicc
EXTRA_LIBS =  -lm

Search for the location of libm.a if not found automatically, enter the location of the file in place of -lm.

Install LAPACK 3.4.1

Edit the make makefile –

FORTRAN	= $(MPI_BIN)/mpif90
LOADER	= $(MPI_BIN)/mpif90
USEXBLAS	= Yes
XBLASLIB	=$(PARALLEL)/xblas/libxblas.a
BLASLIB	= $(PARALLEL)/blas/blas.a

Install mpiBLACS

Edit the following in the makefile –

COMMLIB =
PLAT =
BTopdir =$(HOME)/PARALLEL/blacs
MPIdir =$(HOME)/PARALLEL/mpich2-install
MPILIBdir = $(MPIdir)/lib
MPIINCdir = $(MPIdir)/include
MPILIB = $(MPILIBdir)/libmpich.a

SYSINC = -I$(MPIINCdir)

INTFACE = -DAdd_***

• The Fortran 77 to C interface to be used. If you are unsure of the correct setting for your platform, compile and run BLACS/INSTALL/xintface. Choices are: Add_, NoChange, UpCase, or f77IsF2C.

Go to blacs/INSTALL/EXE

make xintface
./xintface #to check interface type use similar command to check others.

TRANSCOMM = -DUseMpich***
WHATMPI = ***
SYSERRORS =

• above parameterhas to be identified from running all the executable in BLACS/INSTALL/EXE directory

  F77            = $(MPI_BIN)/mpif77
  CC             = $(MPI_BIN)/mpicc
  CCLOADER       = $(MPI_BIN)/mpicc

Install SCALAPACK 2.0.2

Edit the make makefile –

CDEFS		= -DAdd_ # found from mpiBLACS section
FC		= $(MPI_BIN)/mpifc 
CC		= $(MPI_BIN)/mpicc
BLASLIB	= $(PARALLEL)/blas/blas.a
LAPACKLIB	= $(PARALLEL)/lapack/liblapack.a

Install metis 4.x.x

Edit the make makefile –

CC            = $(MPI_BIN)/mpicc

Install parmetis 3.x.x

Edit the make makefile –

CC		= $(MPI_BIN)/mpicc
INCDIR 	= $(MPIdir)/include
LD		= $(MPI_BIN)/mpicc

Install MUMPS_4.10.0

Edit the makefile like following -

LMETISDIR = $(PARALLEL)/parmetis
IMETIS    = -I$(LMETISDIR)
LMETIS    = -L$(LMETISDIR) -lparmetis –lmetis
ORDERINGSF  = -Dmetis -Dpord –Dparmetis
PLAT    =
CC      = $(MPI_BIN)/mpicc 
FC      = $(MPI_BIN)/mpif90
FL      = $(MPI_BIN)/mpif90
##FOR SCLAPACK BLACS SANDWITCHING IS NECESSARY DON’T REMOVE UNLESS YOU KNOW WHAT YOU ARE DOING
SCALAP  = -L$(PARALLEL)/scalapack -lscalapack \
       $(PARALLEL)/blacs/LIB/blacs.a \
       $(PARALLEL)/blacs/LIB/blacsCinit.a \
       $(PARALLEL)/blacs/LIB/blacs.a \
       $(PARALLEL)/blacs/LIB/blacsF77init.a\
       $(PARALLEL)/blacs/LIB/blacs.a \
       $(PARALLEL)/lapack/liblapack.a
INCPAR  = -I$(MPIdir)/include
LIBPAR  = $(SCALAP) \
          -L$(MPIdir)/lib

LIBBLAS = $(PARALLEL)/blas/blas.a
CDEFS = -DAdd_# identified earlier by mpiBLACS check in previous section

BerkeleyDB5.3

Perform the following command-

cd build_unix
../dist/configure
make
make install

You can use prefix to install at your preferred directory, if you don't have enough permission. Be careful to notedown the installation directory.

openssl 1.0.1

Use ./configure and track the location of include directory we will need it later

• For other installed files, find location using whereis command

libdl
libm
libssl
libdb
libcrypto
libgfortran

If not found automatically, try to find manually from all library folders of machine.

Edit The Makefile.def

Copy all the text from following section. Make a new Makefile.def in your machine. Edit the sections which are marked to be edited carefully. Watch this comment in the file and adjust to your machines specificity -"!!!!!!!!!!!!!USE MACHINES SPECIFIC LOCATIONS!!!!!!!!"

<tcl>Makefile.def.HPC.KM.tcl</tcl>

MAKE

Inside OpenSees Folder type make. If it compiles successfully you will get a notification and OpenSeesMP will be created on bin folder.

Some Final Touch-ups

• Copy init.tcl from $(TCLdir)/lib/tcl8.5/tcl.int to directory $(HOME)/lib/tcl8.5/
• Entry of OpenSees to Environment $PATH , go to user home directory -

nano .bashrc

Enter the following line at the end.

export PATH=/home/bin/OpenSeesMP:$PATH

• Running examples –

cd to directory where examples are kept Use following command –

mpirun –np 2 OpenSeesMP example.tcl

Be sure to make mpich2 up and running before trying this.

Created By : 	Khaled Mashfiq, MECRES, Sapienza University of Rome
Courtesy :  	Prof. Rosario Gigliotti, Department of Structural Engineering, Sapienza University of Rome
                Dr. Marco Faggella, Department of Structural Engineering, Sapienza University of Rome
                Pietro Maioli, Department of Structural and Electrical Engineering, Sapienza University of Rome
                SM Iftekharul Alam, Purdue University.