View source for Compilation Guideline of OpenSeeMP on Linux Machines

== Compilation Guideline of OpenSeeMP on Linux Machines ==

This guide is developed in a systematic manner assuming ActiveTcl, MPICH2 and other packages are not pre-installed in the system. But even if those are available slight modification will enable us to compile OpenSeesMP.

The following tarballs are used to compile OpenSeesMP in this manual. This instruction only works with OpenSees 2.3.2.2 version(svn code – 4985). Package’s version has importance for compatibility issues.


===NOTE: Possible Simpler Option Which Worked For Me====
using the scalapack_installer and the correct location of mpi.h

./setup.py --prefix=/home/ubuntu --downall --mpiincdir=/usr/include/mpich2/mpi.h

this installs lapack, parmetis, metis, scalapack, blacs, blas

===Required Packages===
 ActiveTcl8.5.11.1.295590-linux-x86_64-threaded.tar.gz
 db-5.3.21.NC.tar.gz
 lapack-3.4.1.tgz
 metis-4.0.3.tar.gz
 mpiblacs-patch03.tgz
 mpiblacs.tgz –files inside needs to be overwritten by files of mpiblacs-patch03.tgz
 mpich2-1.1.tar.gz
 MUMPS_4.10.0.tar.gz
 openssl-1.0.1c.tar.gz
 ParMetis-3.2.0.tar.gz
 scalapack-2.0.2.tgz
 xblas-1.0.248.tar.gz

You can find the following packages in internet, all of them are free, some of them require an email to the developer.


===Some Terminal Commands===
Some knowledge of bash commands are necessary, parallel machines often do not have graphical user interface.

 mkdir mydir - makes a directory named mydir
 chmod 777 a.txt - change write permission of files to full permission
 nano a.txt - opens a.txt for editing
 cd mydir - gets inside mydir
 cd .. - gets out mydir to parent directory
 which mpd - checks which mpd is used and returns the location

===Some system environment packages===

Before compiling any of these packages we need to perform the following tasks - 
(apt-get command is for ubuntu machines for redhat machines we can use yum install)
sudo in and type the following:
 mkdir lib
 mkdir bin
 sudo apt-get update 
 sudo apt-get install build-essential
 sudo apt-get install subversion
 svn co svn://opensees.berkeley.edu/usr/local/svn/OpenSees/trunk@4985 OpenSees
Use- 
 chmod 777 OpenSees 
To change the editing permission of the directory.
 sudo apt-get install make
 sudo apt-get install gcc
 sudo apt-get install g++
 sudo apt-get install gfortran
 sudo apt-get install mysql

===Compilation of Necessary Packages===

After completion of the above mentioned process we need to customize compile and install some packages, which should be done in a ordered manner- 

To be systematic and implementation on cluster HPC we will create a directory in home called “PARALLEL”. And put all the tarballs required extracts – in simple named folder like scalapack for sclalapack-2.0.2. Hence we get all packs in one master folder.

The directory structure should be setup as following-
;HOME=/home/yourusername
;PARALLEL=$(HOME)/PARALLEL

 level0 |   level1        |level2
 
 HOME   |   PARALLEL      |blacs
               		 |mpich2
               		 |lapack
              		 |scalapack
              		 |blas
              		 |xblas
              		 |metis
              		 |parmetis
              		 |mumps
               		 |---
 Fig – Example directory structure



====Install Activetcl 8.5.11 - x86-64 (Requires Admin Access)====
Untar the installation package, go inside the directory , type –

 install.sh 

By default tcl will be installed in the following directory –

TCLdir=/opt/ActiveTcl-8.5

TCL Directories to remember

TCL_BIN=$(TCLdir)/bin

TCL_INC=-I$(TCLdir)/include

TCL_LIB=-L$(TCLdir)/lib -ltcl8.5 -ltk8.5


 We need to remember one thing!! tcl.init file location $(tcldir)/lib/tcl8.5/tcl.int


====Install mpich2-1.1 for compatibility issues with mumps====

Configuration steps ---

a.	Setting prefix/ installation directory –
for csh and tcsh:

      ./configure --prefix=/home/yourusername/PARALLEL/mpich2-install |& tee c.txt

for bash and sh:

      ./configure --prefix=/home/yourusername/PARALLEL/mpich2-install 2>&1 | tee c.txt

b.	Build MPICH2:

for csh and tcsh:

      make |& tee m.txt

for bash and sh:

      make 2>&1 | tee m.txt
c.	Install the MPICH2 commands:

for csh and tcsh:

      make install |& tee mi.txt

for bash and sh:

      make install 2>&1 | tee mi.txt

d.	Add the bin subdirectory of the installation directory to your path:

for csh and tcsh:

     setenv PATH=/home/yourusername/PARALLEL/mpich2-install/bin:$PATH

for bash and sh:
  
     export PATH=/home/yourusername/PARALLEL/mpich2-install/bin:$PATH

Make permanent change to system variable to recognise mpich2. Go to user home (don’t confuse with OpenSees compilation home). 
Add the line of above section d. at the last line.
 chmod 777 .bashrc
 nano .bashrc

Run the following command to se wheather system variables are changed or not -

 which mpd
 which mpiexec
 which mpirun

Run this command to check everything is working ok.
e.	mpd run
Begin by placing in your home directory a file named .mpd.conf (/etc/mpd.conf if root), containing the line 

      secretword=<secretword>

where <secretword> is a string known only to yourself.  It should NOT be your normal Unix password.  Make this file readable and writable only by you:

      chmod 600 .mpd.conf

The first sanity check consists of bringing up a ring of one mpd on the local machine, testing one mpd command, and bringing the "ring" down. 

 mpd &
 mpdtrace
 mpdallexit

The output of mpdtrace should be the hostname of the machine you are running on. The mpdallexit causes the mpd daemon to exit.  If you have problems getting the mpd ring established, see the Installation Guide of MPICH2 for instructions on how to diagnose problems with your system configuration that may be preventing it.

We will notedown the important directory location of MPICH2-

MPIdir=$(HOME)/PARALLEL/mpich2-install

MPI_BIN=$(MPIdir)/bin

MPI_INC=-I$(MPIdir)/include

MPI_LIB=-L$(MPIdir)/lib


;Use the full directory of all the address of string like $(HOME), $(MPIdir) as system will not recognise such strings used in following 3 to 12 step. After editing the makefiles run relevant make commands for each individual packages.

====Install BLAS====
Edit the make makefile –

 PLAT =
 FORTRAN  = $(MPI_BIN)/mpif90
 LOADER   = $(MPI_BIN)/mpif90

====Install XBLAS====
Edit the make makefile –

 CC = $(MPI_BIN)/mpicc
 EXTRA_LIBS =  -lm
Search for the location of libm.a if not found automatically, enter the location of the file in place of -lm.


====Install LAPACK 3.4.1====
Edit the make makefile –

 FORTRAN	= $(MPI_BIN)/mpif90
 LOADER	= $(MPI_BIN)/mpif90
 USEXBLAS	= Yes
 XBLASLIB	=$(PARALLEL)/xblas/libxblas.a
 BLASLIB	= $(PARALLEL)/blas/blas.a

====Install mpiBLACS====
Edit the following in the makefile –
 COMMLIB =
 PLAT =
 BTopdir =$(HOME)/PARALLEL/blacs
 MPIdir =$(HOME)/PARALLEL/mpich2-install
 MPILIBdir = $(MPIdir)/lib
 MPIINCdir = $(MPIdir)/include
 MPILIB = $(MPILIBdir)/libmpich.a
 
 SYSINC = -I$(MPIINCdir)
 
 INTFACE = -DAdd_***

* The Fortran 77 to C interface to be used.  If you are unsure of the correct setting for your platform, compile and run BLACS/INSTALL/xintface. Choices are: Add_, NoChange, UpCase, or f77IsF2C.

Go to blacs/INSTALL/EXE
 make xintface
 ./xintface #to check interface type use similar command to check others.

 TRANSCOMM = -DUseMpich***
 WHATMPI = ***
 SYSERRORS =
* above parameterhas to be identified from running all the executable in BLACS/INSTALL/EXE directory

   F77            = $(MPI_BIN)/mpif77
   CC             = $(MPI_BIN)/mpicc
   CCLOADER       = $(MPI_BIN)/mpicc


====Install SCALAPACK 2.0.2====
Edit the make makefile –


 CDEFS		= -DAdd_ # found from mpiBLACS section
 FC		= $(MPI_BIN)/mpifc 
 CC		= $(MPI_BIN)/mpicc
 BLASLIB	= $(PARALLEL)/blas/blas.a
 LAPACKLIB	= $(PARALLEL)/lapack/liblapack.a


====Install metis 4.x.x====
Edit the make makefile –

 CC            = $(MPI_BIN)/mpicc

====Install parmetis 3.x.x====
Edit the make makefile –

 CC		= $(MPI_BIN)/mpicc
 INCDIR 	= $(MPIdir)/include
 LD		= $(MPI_BIN)/mpicc


====Install MUMPS_4.10.0====
Edit the makefile like following -

 LMETISDIR = $(PARALLEL)/parmetis
 IMETIS    = -I$(LMETISDIR)
 LMETIS    = -L$(LMETISDIR) -lparmetis –lmetis
 ORDERINGSF  = -Dmetis -Dpord –Dparmetis
 PLAT    =
 CC      = $(MPI_BIN)/mpicc 
 FC      = $(MPI_BIN)/mpif90
 FL      = $(MPI_BIN)/mpif90
 ##FOR SCLAPACK BLACS SANDWITCHING IS NECESSARY DON’T REMOVE UNLESS YOU KNOW WHAT YOU ARE DOING
 SCALAP  = -L$(PARALLEL)/scalapack -lscalapack \
        $(PARALLEL)/blacs/LIB/blacs.a \
        $(PARALLEL)/blacs/LIB/blacsCinit.a \
        $(PARALLEL)/blacs/LIB/blacs.a \
        $(PARALLEL)/blacs/LIB/blacsF77init.a\
        $(PARALLEL)/blacs/LIB/blacs.a \
        $(PARALLEL)/lapack/liblapack.a
 INCPAR  = -I$(MPIdir)/include
 LIBPAR  = $(SCALAP) \
           -L$(MPIdir)/lib
 
 LIBBLAS = $(PARALLEL)/blas/blas.a
 CDEFS = -DAdd_# identified earlier by mpiBLACS check in previous section

====BerkeleyDB5.3====
Perform the following command-
 cd build_unix
 ../dist/configure
 make
 make install

You can use prefix to install at your preferred directory, if you don't have enough permission. Be careful to notedown the installation directory.

====openssl 1.0.1====
Use ./configure and track the location of include directory we will need it later

* For other installed files, find location using whereis command

 libdl
 libm
 libssl
 libdb
 libcrypto
 libgfortran

If not found automatically, try to find manually from all library folders of machine.

===Edit The Makefile.def===

Copy all the text from following section. Make a new Makefile.def in your machine. Edit the sections which are marked to be edited carefully. Watch this comment in the file and adjust to your machines specificity -"!!!!!!!!!!!!!USE MACHINES SPECIFIC LOCATIONS!!!!!!!!"


<tcl>Makefile.def.HPC.KM.tcl</tcl>

===MAKE===
Inside OpenSees Folder type make. If it compiles successfully you will get a notification and OpenSeesMP will be created on bin folder.

===Some Final Touch-ups ===

;* Copy init.tcl from $(TCLdir)/lib/tcl8.5/tcl.int  to directory $(HOME)/lib/tcl8.5/
;* Entry of OpenSees to Environment $PATH , go to user home directory -

 nano .bashrc

Enter the following line at the end. Replace the "path_to_executable" with the path where OpenSeesMP is located.

 export PATH=/path_to_executable/OpenSeesMP:$PATH

;* Running examples –

cd to directory where examples are kept
Use following command –

 mpirun –np 2 OpenSeesMP example.tcl


Be sure to make mpich2 up and running before trying this. 




 Created By : 	Khaled Mashfiq, MECRES, Sapienza University of Rome
 Courtesy :  	Prof. Rosario Gigliotti, Department of Structural Engineering, Sapienza University of Rome
                 Dr. Marco Faggella, Department of Structural Engineering, Sapienza University of Rome
                 Pietro Maioli, Department of Structural and Electrical Engineering, Sapienza University of Rome
                 SM Iftekharul Alam, Purdue University.