Question on the best practice to run multiple OpenSeesPy sessions in parallel

[kangsungku]

Hello,

I would like to know the best practice to utilize OpenSeesPy to analyze multiple models in parallel, such that I can fully utilize a multi-core CPU or a cluster with multiple nodes. Note: I would like to analyze multiple models in parallel, rather than utilizing multiple cores to speed up a single analysis. Here are the specific details on my issue.

-Goal: I would like to efficiently analyze a number of different structures to locate an optimal structure satisfying some constraints (e.g., safety constraint).

-What I did: To achieve the goal, I tried parallelizing a for-loop involving structural analysis using a library called Dask (https://dask.org/) where each iteration of the loop creates an instance importing OpenSeesPy and performing some analysis.

-Issue: Even though I parallelized the for-loop and it seems to be parallelized, the total CPU usage seems not to increase as the number of parallel workers increases. Also, sometimes I got errors like "node with tag xx already exists in model", which implies that OpenSeesPy interpreters called from different instances interfere with each other. Therefore, I believe even though multiple instances have been created, they are relying on the same OpenSees interpreter under the hood, which is not the intended situation. If possible, I would like to assign a separate OpenSees interpreter to each instance, such that they can analyze multiple models in parallel in a scalable manner.


The simplified version of my scripts is shown below. Script 1 defines a class (MasonryAnalysisModule) importing OpenSeesPy and performing some analysis, and Script 2 creates multiple instances of MasonryAnalysisModule and parallelize the analysis by using Dask with the for-loop. Any comments or suggestions (whether with Dask or other methods for parallelization) are really appreciated. Thank you very much for your time!

------------------------------------------------------------------------------------------------------
# Configuration:
Python 3.8.5
OpenSeesPy 3.2.2.5
Dask 2021.1.1

------------------------------------------------------------------------------------------------------
# Script 1: masonry_analysis_module.py

Code: Select all

import openseespy.opensees as ops  # import OpenSeesPy
import Get_Rendering as opsplt  # import OpenSeesPy plotting commands

class MasonryAnalysisModule:
    def test(self, input_seed):
        self.seed = input_seed

        # Initialize OpenSees Model
        # Clean previous model
        ops.wipe()
        ops.model('basic', '-ndm', 3, '-ndf', 3)    # For 'block', ndm and ndf should be 3
        print("This instance has seed of ", self.seed)

        for node_index in range(10):
            x = node_index * self.seed
            y = node_index * self.seed
            z = node_index * self.seed
            ops.node(node_index, x, y, z)

        # The result can be some analysis results, but for test, just return the node tags
        result = ops.getNodeTags()
        print("\t--> complete seed", self.seed)
        return result

----------------------------------------------------------------------------------------------------------
# Script 2: parallel_openseespy_test.py

Code: Select all

from dask.distributed import Client
from dask import delayed, compute

def ops_test(input_seed):
    import masonry_analysis_module as ma
    masonry_analysis_module = ma.MasonryAnalysisModule()
    analysis_result = masonry_analysis_module.test(input_seed)
    return analysis_result

if __name__ == "__main__":
    NUM_FEM_WORKERS = 4
    client = Client(n_workers=NUM_FEM_WORKERS)

    # For-loop using DASK-delayed
    list_result = []
    for input_seed in range(10):
        result = delayed(ops_test)(input_seed)
        list_result.append((input_seed, result))

    # Here, initialte the computation of for-loop and retrieve the results
    list_result = compute(*list_result)

    for result in list_result:
        print("X:", result)

----------------------------------------------------------------------------------------------------------

[mhscott]

This presentation by Dr. Minjie Zhu describes parallel computing with OpenSeesPy very well: https://youtu.be/vjGm2kM5Ihc

[kangsungku]

I appreciate your response.

I reviewed the examples on the video, and the examples seem to be utilizing MPI to 1) run multiple analysis for the same model (i.e., the analysis performed on the same nodes/elements but with different sets of material properties, such that each MPI worker analyzes with each set of properties), or 2) analyze a single model with multiple MPI workers (i.e., partition a model into several pieces, such that each worker analyzes each peace of the model).

However, what I would like to do is to analyze different models (i.e., different sets of nodes/elements) in parallel, so each MPI worker should have its own set of nodes/elements. The issue is that different MPI worker seems to share the nodes/elements, which leads to the error like "node with tag xx already exists in model" (e.g., a node created by worker 1 seems to be visible to other workers).

I believe technically it is able to create separate interpreters for each MPI worker, so would like to know how to do it. Any comments or suggestions will be appreciated!