The OpenSees Community

Hi Frank:

I have been trying to compile OpenSeesSP on an iMac with Mac OS X 10.6.6 for the last few weeks. I am able to compile OpenSees sequential without problem. However, even I can compile OpenSeesSP without any error, when I run anyexample (Example the Example6.1.tcl without the display part) using 2 or more nodes ( mpirun -np 2 (or 3 or 4) OpenSeesSP AnyExample.tcl) the program crash and give a segmentation fault (see error below), but if I run the same example with only one node (mpirun -np 1 OpenSeesSP AnyExample.tcl) the program works and give the same solution as the sequential version of OpenSees.

Error when I run : mpirun -np 2 ./OpenSeesSP ./AnyExample.tcl:
---------------------------------------------------------------------------------------------
Master Process Running OpenSees Interpreter 0


OpenSees -- Open System For Earthquake Engineering Simulation
Pacific Earthquake Engineering Research Center -- 2.2.2

(c) Copyright 1999,2000 The Regents of the University of California
All Rights Reserved
(Copyright and Disclaimer @ http://www.berkeley.edu/OpenSees/copyright.html)


Slave Process Running 1
[earthquake:90722] *** Process received signal ***
[earthquake:90722] Signal: Segmentation fault (11)
[earthquake:90722] Signal code: Address not mapped (1)
[earthquake:90722] Failing at address: 0x0
[earthquake:90722] [ 0] 2 libSystem.B.dylib 0x00007fff80f4367a _sigtramp + 26
[earthquake:90722] [ 1] 3 ??? 0x0000000000000008 0x0 + 8
[earthquake:90722] [ 2] 4 OpenSeesSP 0x00000001001150d0 _ZN14ActorSubdomain3runEv + 4208
[earthquake:90722] [ 3] 5 OpenSeesSP 0x000000010070b839 _ZN13MachineBroker9runActorsEv + 185
[earthquake:90722] [ 4] 6 OpenSeesSP 0x00000001000f3c9a main + 154
[earthquake:90722] [ 5] 7 OpenSeesSP 0x000000010000ba90 start + 52
[earthquake:90722] [ 6] 8 ??? 0x0000000000000002 0x0 + 2
[earthquake:90722] *** End of error message ***
--------------------------------------------------------------------------
mpirun noticed that process rank 1 with PID 90722 on node earthquake exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Here is how I compile OpenSeesSP:

I first install gcc 4.5.2 , tcl and tk 8.5.9 , and openssl (usign macports, everything that is installed with macports is in the path: /opt/local/), and after i set gcc 4.5.2 as my default gcc compiler, with this I compile openmpi 1.4.3 ( I check the tests that came with openmpi and they work Ok, and I check that openmpi is compiled with the gcc 4.5.2), with this I install scalapack_installer_0.96 ( I check also the tests in scalapack-1.8.0 and tests seen to work fine, no error)( the only think with scalapack_installer_0.96 is that I need to force in ./src/script/blacs.py to put in TRANSCOMM -DUseMpi2, because it is not doing this by itself) , Everything is working in 64bit, and with all this I am able to compile OpenSeesSP and i get the OpenSeesSP.exe using the following makefile.def:

#####################################################################
# %---------------------------------%
# | SECTION 1: PROGRAM |
# %---------------------------------%
#
# Specify the location and name of the OpenSees interpreter program
# that will be created (if this all works!)

PROGRAMMING_MODE = PARALLEL

OPERATING_SYSTEM = LINUX

DEBUG_MODE = NO_DEBUG
#DEBUG_MODE = DEBUG
RELIABILITY = NO_RELIABILITY

BASE = /opt/local
HOME = /Users/frojas/OpenSees
FE = $(HOME)/OpenSees/SRC

OpenSees_PROGRAM = $(HOME)/OpenSeesBin/OpenSeesSP

#NO GRAPHICS
GRAPHICS = NONE
GRAPHIC_FLAG = -D_NOGRAPHICS
GRAPHIC_LIBRARY =
AGL_OBJS =

AMDdir = $(HOME)/OpenSees/OTHER/AMD
LAPACKdir = $(HOME)/OpenSees/OTHER/LAPACK
SUPERLUdir = $(HOME)/OpenSees/OTHER/SuperLU_3.0/SRC
SUPERLU_DISTdir = $(HOME)/OpenSees/OTHER/SuperLU_DIST_2.0/SRC
ARPACKdir = $(HOME)/OpenSees/OTHER/ARPACK
UMFPACKdir = $(HOME)/OpenSees/OTHER/UMFPACK
METISdir = $(HOME)/OpenSees/OTHER/METIS
SRCdir = $(HOME)/OpenSees/SRC
BLASdir = $(HOME)/OpenSees/OTHER/BLAS
CBLASdir = $(HOME)/OpenSees/OTHER/CBLAS

DIRS = $(SUPERLUdir) \
$(SUPERLU_DISTdir) \
$(ARPACKdir) \
$(AMDdir) \
$(LAPACKdir) \
$(UMFPACKdir) \
$(METISdir) \
$(BLASdir) \
$(CBLASdir) \
$(SRCdir)


PROGRAMMING_FLAG = -D_PARALLEL_PROCESSING



# %-------------------------------------------------------%
# | SECTION 3: LIBRARIES |
# | |
# | The following section defines the libraries that will |
# | be created and/or linked with when the libraries are |
# | being created or linked with. |
# %-------------------------------------------------------%
#
# Note: if vendor supplied BLAS and LAPACK libraries leave the
# libraries blank. You have to get your own copy of the tcl/tk
# library!!
#
# Note: For libraries that will be created (any in DIRS above)
# make sure the directory exsists where you want the library to go!

FE_LIBRARY = $(HOME)/OpenSeesLib/libOpenSees.a
NDARRAY_LIBRARY = $(HOME)/OpenSeesLib/libndarray.a # BJ_UCD jeremic@ucdavis.edu
MATMOD_LIBRARY = $(HOME)/OpenSeesLib/libmatmod.a # BJ_UCD jeremic@ucdavis.edu
BJMISC_LIBRARY = $(HOME)/OpenSeesLib/libBJmisc.a # BJ_UCD jeremic@ucdavis.edu
LAPACK_LIBRARY = $(HOME)/OpenSeesLib/libLapack.a
CLAPACK_LIBRARY =
CLBLAS_LIBRARY =
LIBF2C_LIBRARY =
SUPERLU_LIBRARY = $(HOME)/OpenSeesLib/libSuperLU.a
ARPACK_LIBRARY = $(HOME)/OpenSeesLib/libArpack.a
AMD_LIBRARY = $(HOME)/OpenSeesLib/libAMD.a
UMFPACK_LIBRARY = $(HOME)/OpenSeesLib/libUmfpack.a
METIS_LIBRARY = $(HOME)/OpenSeesLib/libMetis.a
BLAS_LIBRARY = $(HOME)/OpenSeesLib/libBlas.a
CBLAS_LIBRARY = $(HOME)/OpenSeesLib/libCBlas.a
DISTRIBUTED_SUPERLU_LIBRARY = $(HOME)/OpenSeesLib/libDistributedSuperLU.a
TCL_LIBRARY = -L/opt/local/lib -ltcl -L/opt/local/lib -ltk

# WATCH OUT .. These libraries are removed when 'make wipe' is invoked.
WIPE_LIBS = $(FE_LIBRARY) \
$(NDARRAY_LIBRARY) \
$(MATMOD_LIBRARY) \
$(LAPACK_LIBRARY) \
$(AMD_LIBRARY) \
$(BLAS_LIBRARY) \
$(CLBLAS_LIBRARY) \
$(SUPERLU_LIBRARY) \
$(ARPACK_LIBRARY) \
$(UMFPACK_LIBRARY) \
$(METIS_LIBRARY)


# %---------------------------------------------------------%
# | SECTION 4: COMPILERS |
# | |
# | The following macros specify compilers, linker/loaders, |
# | the archiver, and their options. You need to make sure |
# | these are correct for your system. |
# %---------------------------------------------------------%

# Compilers
OMPIhome = $(HOME)/OpenSeesTools/openmpi-1.4.3/Darwin
CC++ = $(OMPIhome)/bin/mpic++
CC = $(OMPIhome)/bin/mpicc
FC = $(OMPIhome)/bin/mpif77
LINKER = $(CC++)
LINKFLAGS =

AR = ar
ARFLAGS = cqls
RANLIB = ranlib
RANLIBFLAGS =

# Compiler Flags
#
# NOTES:
# C++ FLAGS TAKE need _UNIX or _WIN32 for preprocessor dircetives
# - the _WIN32 for the Windows95/98 or NT operating system.
# C FLAGS used -DUSE_VENDOR_BLAS (needed in SuperLU) if UNIX in C++ FLAGS
#

OS_FLAG = -D_MACOSX

# modified as optimizaton currently causing problems with Steeln01 code
ifeq ($(DEBUG_MODE), DEBUG)

C++FLAGS = -m64 -Wall -D_LINUX -D_UNIX -D_TCL85 -D_AMDn $(OS_FLAG) -D_HTTPS \
$(GRAPHIC_FLAG) $(RELIABILITY_FLAG) $(DEBUG_FLAG) $(MUMPS_FLAG) \
$(PROGRAMMING_FLAG) -g -O0 -ffloat-store
CFLAGS = -Wall -O0 -g -m64
FFLAGS = -Wall -O -m64 -DCUBLAS -x f77-cpp-input --no-second-underscore


# Linker
LINKER = $(CC++)
LINKFLAGS = -g -pg -m64

else

C++FLAGS = -Wall -D_LINUX -D_UNIX -D_TCL85 -D_AMDn -D_MACOSX -D_HTTPS \
$(GRAPHIC_FLAG) $(RELIABILITY_FLAG) $(DEBUG_FLAG) $(MUMPS_FLAG) \
$(PROGRAMMING_FLAG) -O3 -ffloat-store -D_NO_PARALLEL_FILESYSTEM -m64
CFLAGS = -Wall -O2 -m64
FFLAGS = -Wall -O -DCUBLAS -m64 -x f77-cpp-input --no-second-underscore

LINKFLAGS = -L$(OMPIhome)/lib $(OMPIhome)/lib/libmpi.a

endif

# Misc
MAKE = make
CD = cd
ECHO = echo
RM = rm
RMFLAGS = -f
SHELL = /bin/sh

# %---------------------------------------------------------%
# | SECTION 5: COMPILATION |
# | |
# | The following macros specify the macros used in |
# | to compile the source code into object code. |
# %---------------------------------------------------------%

.SUFFIXES:
.SUFFIXES: .C .c .f .f90 .cpp .o .cpp

#
# %------------------%
# | Default command. |
# %------------------%
#
.DEFAULT:
@$(ECHO) "Unknown target $@, try: make help"
#
# %-------------------------------------------%
# | Command to build .o files from .f files. |
# %-------------------------------------------%
#

.cpp.o:
@$(ECHO) Making $@ from $<
$(CC++) $(C++FLAGS) $(INCLUDES) -c $< -o $@

.C.o:
@$(ECHO) Making $@ from $<
$(CC++) $(C++FLAGS) $(INCLUDES) -c $< -o $@
.c.o:
@$(ECHO) Making $@ from $<
$(CC) $(CFLAGS) -c $< -o $@
.f.o:
@$(ECHO) Making $@ from $<
$(FC) $(FFLAGS) -c $< -o $@

# %---------------------------------------------------------%
# | SECTION 6: OTHER LIBRARIES |
# | |
# | The following macros specify other libraries that must |
# | be linked with when creating executables. These are |
# | platform specific and typically order does matter!! |
# %---------------------------------------------------------%

HAVE_SCALAPACK = YES
SCALAPACK_INCLUDE =$(HOME)/OpenSeesTools/scalapack_package/Darwin
SCALAPACK_LIB =$(HOME)/OpenSeesTools/scalapack_package/Darwin/lib

ifeq ($(HAVE_SCALAPACK), YES)

SCALAP_DIR = $(HOME)/OpenSeesTools/scalapack_package/Darwin
SCALAP = YES
SCALAP_FLAG = -D_SCALAP

SCALAP_LIB_DIR = $(SCALAP_DIR)/lib

SCALAP_LIB = $(SCALAP_LIB_DIR)/libscalapack.a \
$(SCALAP_LIB_DIR)/libreflapack.a \
$(SCALAP_LIB_DIR)/librefblas.a \
$(SCALAP_LIB_DIR)/blacs.a \
$(SCALAP_LIB_DIR)/blacsC.a \
$(SCALAP_LIB_DIR)/blacsF77.a

endif

HAVE_MUMPS = NO
MUMPS_INCLUDE =
MUMPS_LIB =

MACHINE_LINKLIBS = -L/opt/local/lib \
-L$(HOME)/OpenSeesLib

HAVE_CUDA = NO
CUDA_DIR =
CUDA_FLAG =
CUDA_LIB =


FORTO =

MACHINE_NUMERICAL_LIBS = \
$(ARPACK_LIBRARY) \
$(SUPERLU_LIBRARY) \
$(UMFPACK_LIBRARY) \
$(AMD_LIBRARY) \
$(LAPACK_LIBRARY) \
$(CLAPACK_LIBRARY) \
$(CLBLAS_LIBRARY) \
$(BLAS_LIBRARY) \
$(CBLAS_LIBRARY) \
$(GRAPHIC_LIBRARY)\
$(FORTO) \
$(CUDA_LIB) \
-L/opt/local/lib/gcc45 -lgfortran


MACHINE_SPECIFIC_LIBS = -L/opt/local/lib -lssl

PARALLEL_LIB = $(FE)/system_of_eqn/linearSOE/sparseGEN/DistributedSuperLU.o \
$(FE)/system_of_eqn/linearSOE/sparseGEN/DistributedSparseGenColLinSOE.o \
$(FE)/system_of_eqn/linearSOE/sparseGEN/SparseGenColLinSOE.o \
$(LAPACK_LIB) $(DISTRIBUTED_SUPERLU_LIBRARY) $(MUMPS_LIB) $(SCALAP_LIB) $(METIS_LIBRARY) \
-L$(OMPIhome)/lib -lmpi_cxx -lmpi_f77 -lmpi -lopen-rte -lopen-pal


# %---------------------------------------------------------%
# | SECTION 7: INCLUDE FILES |
# | |
# | The following macros specify include files needed for |
# | compilation. |
# %---------------------------------------------------------%

MACHINE_INCLUDES = -I$(OMPIhome)/include \
-I$(FE)/../OTHER/SuperLU_DIST_2.0/SRC \
-I/opt/local/include -I$(HOME)/blitz $(MUMPS_INCLUDE) -I$(CUDA_DIR)/include

# this file contains all the OpenSees/SRC includes
include $(FE)/Makefile.incl

TCL_INCLUDES = -I/opt/local/include

INCLUDES = $(TCL_INCLUDES) $(FE_INCLUDES) $(MACHINE_INCLUDES)

#####################################################################

I have also try using different version of gcc (4.3.5 and 4.4.5) and with different combination of openmpi (1.2.6, 1.2.9, 1.3.4, 1.4.3) , I also try to install only gfortran and used the gcc 4.2.1 that came with mac os 10.6.6 to compile openmpi and after OpenSeesSP, but I always get the same segmentation fault error. I think is something with the message passing in the opensees classes. I am out of Ideas of how to fix this

Any idea of what is happening, and how to fix the problem?

Thanks

you are going to have to debug it to find where the problem is .. the example run just fine when i test them on my setup.