Building OpenSeesMP on Kraken

[reagan]

A few questions related to building OpenSeesMP on Kraken. I'm trying to build an older revision (#5184).

1. The instructions in MAKES/Makefile.def.KRAKEN require building Tcl from source. Any reason why we can't link to the available Tcl library in /usr/lib?
2. I tried building Tcl from source by downloading tcl8.5.15. First, the configure script doesn't seem to recognize the --enable-static flag. Second, the built tclsh8.5 binary segfaults upon running.
3. If I try building OpenSees using either the existing Tcl library in /usr/lib or the ~/tcl/lib directory created by building Tcl from source, the build stops with the following error:

/nics/d/home/reagan/builds/OpenSees/SRC/system_of_eqn/linearSOE/sparseGEN/SuperLU.h(48): catastrophic error: cannot open source file "slu_ddefs.h"
#include <slu_ddefs.h>

Any ideas on how to proceed?

Thanks

Reagan

[reagan]

FYI, the --prefix=~/tcl didn't work as mentioned in the comments in MAKES/Makefile.def.KRAKEN. It says it needs the absolute path without ~. In case you wanted to make the change in the comments.

[fmk]

Reagan,

the only problem i have typcally with the tcl in the /usr directories is that it is typically an 8.4 version. It can be built using these just take out the _TCL85 in the Makefile.def. the instructions i put in when i build it, if they change this i don't keep up. ./configure --help will show the options. i can open up my tcl install if you want to link against it.

[reagan]

Hi Frank

Thanks for your reply

I think the reason I was having the problems with building OpenSees (point 3 above) was because I was using the Makefile.def.KRAKEN from revision 5184 (the older version I'm trying to build). I just saw that there was a newer version of the file in SVN. So when I download and use the new version of the Makefile.def.KRAKEN, along with the rest of the code from revision 5184, I get no errors until I reach the end where it tries to link to the tcl libraries. First, could you please confirm that it'd be alright for me to use the new version of the Makefile.def.KRAKEN with the older revision #5184 of the source code?

Now I found /usr/lib/libtcl8.5.so on Kraken and tried linking to it without removing the -TCL85 flag since it was version 8.5. However, I think OpenSees needs a static version of the library and the build aborts with the following error:

ipo: warning #11010: file format not recognized for /usr/lib/libtcl8.5.so
/usr/bin/ld: skipping incompatible /usr/lib/libdl.a when searching for -ldl
/nics/d/home/reagan/builds/lib/libOpenSees.a(packages.o): In function `getLibraryFunction(char const*, char const*, void**, void**)':
packages.cpp:(.text+0xa7): warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: attempted static link of dynamic object `/usr/lib/libtcl8.5.so'
gmake[1]: *** [tcl] Error 1
gmake[1]: Leaving directory `/nics/d/home/reagan/builds/OpenSees/SRC/modelbuilder/tcl'
make: *** [all] Error 2

It'd be great if you could let me link to your tcl install.

[fmk]

reagan,

seems i cannot log into kraken at the moment due to account inactivity. i have requested my accound be updated again.

[fmk]

Reagan,

/nics/b/home/fmckenna/tcl/include

and

/nics/b/home/fmckenna/tcl/lib

[reagan]

Hi Frank

I'm getting a permission denied error for both directories.

[reagan]

Hi Frank

Just tried one more thing:

I started wondering if the reason it wanted a static TCL library was because of the _TCL85 flag, so I removed the flag and tried linking to the dynamic /usr/lib/tcl8.5.so library, but no luck. The build would stop very near the beginning with Tcl related errors.

Any help would be appreciated.

[fmk]

Reagan,

i have just left my home directory open. can you try again
see if you can:

ls /nics/b/home/fmckenna/tcl

[reagan]

Thanks Frank. Was able to access it now.

Still no luck building OpenSeesMP though. Again, it goes through the whole compilation process, but while linking in the end, it gives me the following error:

LIBRARIES BUILT ... NOW LINKING OpenSees PROGRAM
gmake[1]: Entering directory `/nics/d/home/reagan/builds/OpenSees/SRC/tcl'
gmake[1]: Nothing to be done for `tcl'.
gmake[1]: Leaving directory `/nics/d/home/reagan/builds/OpenSees/SRC/tcl'
gmake[1]: Entering directory `/nics/d/home/reagan/builds/OpenSees/SRC/modelbuilder/tcl'
Makefile:46: warning: overriding commands for target `tcl'
Makefile:14: warning: ignoring old commands for target `tcl'
gmake[2]: Entering directory `/nics/d/home/reagan/builds/OpenSees/SRC/tcl'
gmake[2]: Nothing to be done for `tcl'.
gmake[2]: Leaving directory `/nics/d/home/reagan/builds/OpenSees/SRC/tcl'
/bin/sh: line 0: cd: /nics/d/home/reagan/builds/OpenSees/SRC/system_of_eqn/diagonal: No such file or directory
gmake[2]: Entering directory `/nics/d/home/reagan/builds/OpenSees/SRC/modelbuilder/tcl'
Makefile:46: warning: overriding commands for target `tcl'
Makefile:14: warning: ignoring old commands for target `tcl'
Unknown target mpi, try: make help
gmake[2]: Leaving directory `/nics/d/home/reagan/builds/OpenSees/SRC/modelbuilder/tcl'
gmake[2]: Entering directory `/nics/d/home/reagan/builds/OpenSees/SRC/actor/channel'
gmake[2]: Nothing to be done for `mpi'.
gmake[2]: Leaving directory `/nics/d/home/reagan/builds/OpenSees/SRC/actor/channel'
echo CC -Wl,-rpath /nics/d/home/reagan/builds/OpenSees/SRC/tcl/tclMain.o /nics/d/home/reagan/builds/OpenSees/SRC/tcl/mpiParameterMain.o /nics/d/home/reagan/builds/OpenSees/SRC/tcl/commands.o /nics/d/home/reagan/builds/OpenSees/SRC/modelbuilder/tcl/myCommands.o /nics/d/home/reagan/builds/OpenSees/SRC/actor/channel/MPI_Channel.o /nics/d/home/reagan/builds/OpenSees/SRC/actor/machineBroker/MPI_MachineBroker.o /nics/d/home/reagan/builds/OpenSees/SRC/system_of_eqn/linearSOE/diagonal/MPIDiagonalSOE.o /nics/d/home/reagan/builds/OpenSees/SRC/system_of_eqn/linearSOE/diagonal/MPIDiagonalSolver.o /nics/d/home/reagan/builds/OpenSees/SRC/domain/subdomain/ActorSubdomain.o /nics/d/home/reagan/builds/OpenSees/SRC/domain/subdomain/ShadowSubdomain.o /nics/d/home/reagan/builds/OpenSees/SRC/tcl/TclFeViewer.o /nics/d/home/reagan/builds/OpenSees/SRC/tcl/TclVideoPlayer.o \
/nics/d/home/reagan/builds/lib/libOpenSees.a -L/usr/lib -L/nics/d/home/reagan/builds/lib -ldl /nics/d/home/reagan/builds/OpenSees/SRC/system_of_eqn/linearSOE/sparseGEN/DistributeSuperLU.o /nics/d/home/reagan/builds/OpenSees/SRC/system_of_eqn/linearSOE/sparseGEN/DistributedSparseGenColLinSOE.o \
/nics/d/home/reagan/builds/lib/libArpack.a /nics/d/home/reagan/builds/lib/libSuperLU.a /nics/d/home/reagan/builds/lib/libUmfpack.a /nics/d/home/reagan/builds/lib/libAMD.a /nics/d/home/reagan/builds/lib/libDistributedSuperLU.a /nics/d/home/reagan/builds/lib/libCSparse.a /nics/d/home/reagan/builds/lib/libMetis.a /nics/d/home/reagan/builds/lib/libCBlas.a /nics/d/home/reagan/builds/tcl_fmk/lib/libtcl8.5.a \
/nics/d/home/reagan/builds/OpenSees/SRC/tcl/tclMain.o -o /nics/d/home/reagan/builds/bin/OpenSeesMP
CC -Wl,-rpath /nics/d/home/reagan/builds/OpenSees/SRC/tcl/tclMain.o /nics/d/home/reagan/builds/OpenSees/SRC/tcl/mpiParameterMain.o /nics/d/home/reagan/builds/OpenSees/SRC/tcl/commands.o /nics/d/home/reagan/builds/OpenSees/SRC/modelbuilder/tcl/myCommands.o /nics/d/home/reagan/builds/OpenSees/SRC/actor/channel/MPI_Channel.o /nics/d/home/reagan/builds/OpenSees/SRC/actor/machineBroker/MPI_MachineBroker.o /nics/d/home/reagan/builds/OpenSees/SRC/system_of_eqn/linearSOE/diagonal/MPIDiagonalSOE.o /nics/d/home/reagan/builds/OpenSees/SRC/system_of_eqn/linearSOE/diagonal/MPIDiagonalSolver.o /nics/d/home/reagan/builds/OpenSees/SRC/domain/subdomain/ActorSubdomain.o /nics/d/home/reagan/builds/OpenSees/SRC/domain/subdomain/ShadowSubdomain.o /nics/d/home/reagan/builds/OpenSees/SRC/tcl/TclFeViewer.o /nics/d/home/reagan/builds/OpenSees/SRC/tcl/TclVideoPlayer.o /nics/d/home/reagan/builds/lib/libOpenSees.a -L/usr/lib -L/nics/d/home/reagan/builds/lib -ldl /nics/d/home/reagan/builds/OpenSees/SRC/system_of_eqn/linearSOE/sparseGEN/DistributeSuperLU.o /nics/d/home/reagan/builds/OpenSees/SRC/system_of_eqn/linearSOE/sparseGEN/DistributedSparseGenColLinSOE.o /nics/d/home/reagan/builds/lib/libArpack.a /nics/d/home/reagan/builds/lib/libSuperLU.a /nics/d/home/reagan/builds/lib/libUmfpack.a /nics/d/home/reagan/builds/lib/libAMD.a /nics/d/home/reagan/builds/lib/libDistributedSuperLU.a /nics/d/home/reagan/builds/lib/libCSparse.a /nics/d/home/reagan/builds/lib/libMetis.a /nics/d/home/reagan/builds/lib/libCBlas.a /nics/d/home/reagan/builds/tcl_fmk/lib/libtcl8.5.a /nics/d/home/reagan/builds/OpenSees/SRC/tcl/tclMain.o -o /nics/d/home/reagan/builds/bin/OpenSeesMP
icpc: error #10236: File not found: '/nics/d/home/reagan/builds/OpenSees/SRC/system_of_eqn/linearSOE/sparseGEN/DistributeSuperLU.o'
gmake[1]: *** [tcl] Error 1
gmake[1]: Leaving directory `/nics/d/home/reagan/builds/OpenSees/SRC/modelbuilder/tcl'
make: *** [all] Error 2

Any suggestions? Spent a week at this now. We have about a month before Kraken is decommissioned and a huge allocation left to burn. Stampede allocation is almost empty. Would really appreciate any help to build OpenSeesMP using revision 5184 of the code. Even a few other students here are having issues with the newer versions of OpenSees.

Thanks a lot

Reagan

[fmk]

reagan,

look at the4'th last line ... DistributedSuperLU .. cd to the directory, see if file there .. if not type make and see if it is created..

what specific version are you trying to build?

[reagan]

Hi Frank

Thanks for pointing that out. I had a stupid typo in the Makefile.def. Fixed it and it actually popped out the OpenSeesMP binary. However, running it directly gives me the following error:

reagan@kraken-gsi2:~/builds/bin> ./OpenSeesMP
Assertion failed in file /ptmp/ulib/mpt/nightly/5.3/120211/mpich2/src/mpid/cray/src/adi/mpid_init.c at line 115: 0
aborting job:
(null)

Running OpenSeesMP directly (without using mpiexec/mpirun) seems to work on other computers. Not sure if Kraken needs it to be submitted as a batch job for it to work. Just to check, I tried building an OpenSees binary by changing the PROGRAMMING_MODE to SEQUENTIAL, however that gives the following errors:

LIBRARIES BUILT ... NOW LINKING OpenSees PROGRAM
gmake[1]: Entering directory `/nics/d/home/reagan/builds/OpenSees/SRC/tcl'
gmake[1]: Nothing to be done for `tcl'.
gmake[1]: Leaving directory `/nics/d/home/reagan/builds/OpenSees/SRC/tcl'
gmake[1]: Entering directory `/nics/d/home/reagan/builds/OpenSees/SRC/modelbuilder/tcl'
gmake[2]: Entering directory `/nics/d/home/reagan/builds/OpenSees/SRC/tcl'
gmake[2]: Nothing to be done for `tcl'.
gmake[2]: Leaving directory `/nics/d/home/reagan/builds/OpenSees/SRC/tcl'
/usr/bin/ld: skipping incompatible /usr/lib/libdl.a when searching for -ldl
/nics/d/home/reagan/builds/lib/libOpenSees.a(packages.o): In function `getLibraryFunction(char const*, char const*, void**, void**)':
packages.cpp:(.text+0xa7): warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: skipping incompatible /usr/lib/librt.a when searching for -lrt
/usr/bin/ld: skipping incompatible /usr/lib/libpthread.a when searching for -lpthread
/usr/bin/ld: skipping incompatible /usr/lib/libm.a when searching for -lm
/usr/bin/ld: skipping incompatible /usr/lib/libpthread.a when searching for -lpthread
/usr/bin/ld: skipping incompatible /usr/lib/libm.a when searching for -lm
/usr/bin/ld: skipping incompatible /usr/lib/libc.a when searching for -lc
/nics/d/home/reagan/builds/tcl_fmk/lib/libtcl8.5.a(tclUnixCompat.o): In function `TclpGetGrGid':
tclUnixCompat.c:(.text+0x41): warning: Using 'getgrgid' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/nics/d/home/reagan/builds/tcl_fmk/lib/libtcl8.5.a(tclUnixCompat.o): In function `TclpGetGrNam':
tclUnixCompat.c:(.text+0x51): warning: Using 'getgrnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/nics/d/home/reagan/builds/tcl_fmk/lib/libtcl8.5.a(tclUnixCompat.o): In function `TclpGetPwNam':
tclUnixCompat.c:(.text+0x71): warning: Using 'getpwnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/nics/d/home/reagan/builds/tcl_fmk/lib/libtcl8.5.a(tclUnixCompat.o): In function `TclpGetPwUid':
tclUnixCompat.c:(.text+0x61): warning: Using 'getpwuid' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/nics/d/home/reagan/builds/tcl_fmk/lib/libtcl8.5.a(tclUnixChan.o): In function `CreateSocketAddress':
tclUnixChan.c:(.text+0x1f17): warning: Using 'getaddrinfo' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/nics/d/home/reagan/builds/tcl_fmk/lib/libtcl8.5.a(tclUnixCompat.o): In function `TclpGetHostByAddr':
tclUnixCompat.c:(.text+0x21): warning: Using 'gethostbyaddr' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/nics/d/home/reagan/builds/lib/libOpenSees.a(HTTP.o): In function `establishHTTPConnection(char const*, unsigned int)':
HTTP.cpp:(.text+0x2d): warning: Using 'gethostbyname' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/nics/d/home/reagan/builds/tcl_fmk/lib/libtcl8.5.a(tclIOSock.o): In function `TclSockGetPort':
tclIOSock.c:(.text+0x143): warning: Using 'getservbyname' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: skipping incompatible /usr/lib/libdl.a when searching for -ldl
/usr/bin/ld: skipping incompatible /usr/lib/libc.a when searching for -lc
/nics/d/home/reagan/builds/OpenSees/SRC/tcl/commands.o: In function `specifySOE(void*, Tcl_Interp*, int, char const**)':
commands.cpp:(.text+0xdd99): undefined reference to `MPIDiagonalSolver::MPIDiagonalSolver(double)'
commands.cpp:(.text+0xddd9): undefined reference to `MPIDiagonalSOE::MPIDiagonalSOE(MPIDiagonalSolver&)'
/usr/bin/ld: link errors found, deleting executable `/nics/d/home/reagan/builds/bin/OpenSees'
gmake[1]: *** [tcl] Error 1
gmake[1]: Leaving directory `/nics/d/home/reagan/builds/OpenSees/SRC/modelbuilder/tcl'
make: *** [all] Error 2

I'm trying to build OpenSees revision 5184. I've left my home directory open on Kraken. Would be great if you could take a look: /nics/d/home/reagan/builds/OpenSees

Thanks a lot

Reagan

[fmk]

Reagan,

the file is built. in the bin directory of your build directory.

when running on Kraken 3 things to note (1 and 2 are different from when running on stampede)
1) the files must be in the TG_CLUSTER_LUSTRE directory for the compute nodes, cd $TG_CLUSTER_LUSTRE
- this includes both the .exe and all scripts and other files needed!
2) you will have to create a lib in this directory containing the tcl lib files:
a) in my tcl directory there is a lib.tar file copy this file to the lustre directory
b) uncompress it; tar xBf lib.tar
3) you need a pbs file - here is a sample one for the SmallMP ( i tested this with the .exe i created)
#!/bin/bash
#PBS -l size=12
#PBS -l walltime=00:10:00
#PBS -A TG-BCS130008
#PBS -N fmk
#PBS -o fmk.sysout
#PBS -e fmk.syserr
cd $PBS_O_WORKDIR
aprun -n 12 ./OpenSeesMP Example.tcl

[reagan]

It works great!

Thanks a lot.