[Trilinos-Users] Epetra slow down instead of speed up on local machine (OpenMPI?)

[Gyorgy Matyasfalvi]

Dear User Community:

I have a code that is using Epetra, this code runs as expected on TACC's
Stampede, by that I mean I get descent scaling, however when I am running
it on a local 16 core (Intel Xeon ES-2660 @ 2.20 GHz) machine with Ubuntu
12.04 I experience a slow down, instead of a speed up, when increasing the
number of cores.

To be more specific:
The code runs on Stampede with 1 core in 36 seconds with 2 cores in 18
seconds ... etc. On the local machine with 1 core I get 129 seconds with 2
cores 228 seconds ... etc.
Not only takes the 1 core job substantially more time on the local machine
it also increases running time as I increase the number of cores.

On the local machine I'm using OpenMPI and OpenBLAS. On Stampede MVAPICH2
and Intel MKL. I wonder if this could be the problem. Does anyone have
experience with OpenMPI and Epetra? It seems to me there is a communication
issue.

If I replace Epetra with a serial vector class the 1 core performance on
the local machine is comparable to the 1 core performance on Stampede.

Thanks in advance for any help!
Best,
George

My configure scripts are the following:


Local machine:
**************************************************************************************
#!/bin/bash

EXTRA_ARGS=$@

export CMAKE_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu

cmake \
-D CMAKE_BUILD_TYPE:STRING=DEBUG \
-DTrilinos_EXTRA_LINK_FLAGS:STRING="-lgfortran" \
\
-D TPL_ENABLE_BLAS:BOOL=ON \
-D BLAS_INCLUDE_DIRS:PATH="/usr/local/openblas/" \
-D BLAS_LIBRARY_DIRS:FILEPATH="/usr/local/openblas/lib" \
-D BLAS_LIBRARY_NAMES:STRING="libopenblas.a" \
\
-D TPL_ENABLE_LAPACK:BOOL=ON \
-D LAPACK_INCLUDE_DIRS:PATH="/usr/local/openblas/" \
-D LAPACK_LIBRARY_DIRS:FILEPATH="/usr/local/openblas/lib" \
-D LAPACK_LIBRARY_NAMES:STRING="libopenblas.a" \
\
-D TPL_ENABLE_MPI:BOOL=ON \
\
-D Trilinos_ENABLE_DEBUG:BOOL=ON \
-D Trilinos_ENABLE_TESTS:BOOL=ON \
-D Trilinos_ENABLE_OpenMP:BOOL=ON \
-D Trilinos_ENABLE_Zoltan:BOOL=ON \
-D Trilinos_ENABLE_Epetra:BOOL=ON \
-D Trilinos_ENABLE_EpetraExt=ON \
-D Trilinos_ENABLE_Kokkos:BOOL=ON \
-D Kokkos_ENABLE_EXAMPLES:BOOL=ON \
-D Trilinos_ENABLE_Teuchos:BOOL=ON \
-D Trilinos_ENABLE_TEUCHOS_TIME_MONITOR:BOOL=ON \
\
-D CMAKE_INSTALL_PREFIX:PATH="/usr/local/trilinos" \
-D Trilinos_INSTALL_INCLUDE_DIR:PATH="include" \
-D Trilinos_INSTALL_LIB_DIR:PATH="lib" \
-D Trilinos_INSTALL_RUNTIME_DIR:PATH="bin" \
\
$EXTRA_ARGS \
../trilinos-11.4.3-Source
**************************************************************************************


Stampede:
**************************************************************************************
#!/usr/bin/env bash

rm -f MakeCache.txt

EXTRA_ARGS=$@

cmake -D CMAKE_INSTALL_PREFIX:PATH=$HOME/trilinos/ \
  \
  -D MPI_BASE_DIR:PATH=$MPICH_HOME \
  \
  -D CMAKE_BUILD_TYPE:STRING=Release \
  -D CMAKE_Fortran_COMPILER:STRING="mpif90" \
  -D CMAKE_CXX_FLAGS:STRING="-O2 -Wall -ansi -Wno-long-long -fp-model
precise -fp-model source -ip -xHost" \
  -D BUILD_SHARED_LIBS:BOOL=ON \
  -D Trilinos_WARNINGS_AS_ERRORS_FLAGS:STRING="" \
  \
  -D Trilinos_ENABLE_ALL_PACKAGES:BOOL=OFF \
  -D Trilinos_ENABLE_ALL_OPTIONAL_PACKAGES:BOOL=OFF \
  -D Trilinos_ENABLE_ALL_FORWARD_DEP_PACKAGES:BOOL=OFF \
  -D Trilinos_ENABLE_Teuchos:BOOL=ON \
  -D Trilinos_ENABLE_Epetra:BOOL=ON \
  -D Trilinos_ENABLE_EpetraExt:BOOL=ON \
  -D Trilinos_ENABLE_AztecOO:BOOL=ON \
  -D Trilinos_ENABLE_Zoltan:BOOL=ON \
  -D Trilinos_ENABLE_SECONDARY_STABLE_CODE:BOOL=ON \
  \
  -D Trilinos_ENABLE_EXAMPLES:BOOL=OFF \
  -D Trilinos_ENABLE_TESTS:BOOL=OFF \
  \
  -D TPL_ENABLE_MATLAB:BOOL=OFF \
  -D TPL_ENABLE_Matio:BOOL=OFF \
  -D TPL_ENABLE_MPI:BOOL=ON \
  -D TPL_ENABLE_BLAS:BOOL=ON \
  -D TPL_BLAS_LIBRARIES:STRING="-L$MKLROOT/lib/intel64 -lmkl_intel_lp64
-lmkl_blas95_lp64 -lmkl_core -lmkl_sequential" \
  -D TPL_ENABLE_LAPACK:BOOL=ON \
  -D TPL_LAPACK_LIBRARIES:STRING="-L$MKLROOT/lib/intel64
-lmkl_lapack95_lp64" \
  -D TPL_ENABLE_Boost:BOOL=ON \
  -D TPL_Boost_INCLUDE_DIRS:PATH=$TACC_BOOST_INC \
  \
  -D CMAKE_VERBOSE_MAKEFILE:BOOL=OFF \
  \
  $EXTRA_ARGS \
  $HOME/trilinos/trilinos-11.4.3-Source

#make -j 8
#make install
**************************************************************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://software.sandia.gov/pipermail/trilinos-users/attachments/20140618/cce72e84/attachment.html>