[Trilinos-Users] Trilinos-Users Digest, Vol 106, Issue 17
Riccardo Rossi
rrossi at cimne.upc.edu
Wed Jun 18 12:37:17 MDT 2014
Hi,
try to do
export OMP_NUM_THREADS=1
before the run on your local machine
regards
Riccardo
On Wed, Jun 18, 2014 at 8:00 PM, <trilinos-users-request at software.sandia.gov
> wrote:
> Send Trilinos-Users mailing list submissions to
> trilinos-users at software.sandia.gov
>
> To subscribe or unsubscribe via the World Wide Web, visit
> https://software.sandia.gov/mailman/listinfo/trilinos-users
> or, via email, send a message with subject or body 'help' to
> trilinos-users-request at software.sandia.gov
>
> You can reach the person managing the list at
> trilinos-users-owner at software.sandia.gov
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Trilinos-Users digest..."
>
>
> Today's Topics:
>
> 1. Epetra slow down instead of speed up on local machine
> (OpenMPI?) (Gyorgy Matyasfalvi)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Wed, 18 Jun 2014 13:29:15 -0400
> From: Gyorgy Matyasfalvi <matyasfalvi at gmail.com>
> To: "trilinos-users at software.sandia.gov"
> <trilinos-users at software.sandia.gov>
> Subject: [Trilinos-Users] Epetra slow down instead of speed up on
> local machine (OpenMPI?)
> Message-ID:
> <CADtL5Si_WgQ7L-_8AHXRquBAqbvvc-CfAODcVh=
> UZYmA8jmgQg at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Dear User Community:
>
> I have a code that is using Epetra, this code runs as expected on TACC's
> Stampede, by that I mean I get descent scaling, however when I am running
> it on a local 16 core (Intel Xeon ES-2660 @ 2.20 GHz) machine with Ubuntu
> 12.04 I experience a slow down, instead of a speed up, when increasing the
> number of cores.
>
> To be more specific:
> The code runs on Stampede with 1 core in 36 seconds with 2 cores in 18
> seconds ... etc. On the local machine with 1 core I get 129 seconds with 2
> cores 228 seconds ... etc.
> Not only takes the 1 core job substantially more time on the local machine
> it also increases running time as I increase the number of cores.
>
> On the local machine I'm using OpenMPI and OpenBLAS. On Stampede MVAPICH2
> and Intel MKL. I wonder if this could be the problem. Does anyone have
> experience with OpenMPI and Epetra? It seems to me there is a communication
> issue.
>
> If I replace Epetra with a serial vector class the 1 core performance on
> the local machine is comparable to the 1 core performance on Stampede.
>
> Thanks in advance for any help!
> Best,
> George
>
> My configure scripts are the following:
>
>
> Local machine:
>
> **************************************************************************************
> #!/bin/bash
>
> EXTRA_ARGS=$@
>
> export CMAKE_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu
>
> cmake \
> -D CMAKE_BUILD_TYPE:STRING=DEBUG \
> -DTrilinos_EXTRA_LINK_FLAGS:STRING="-lgfortran" \
> \
> -D TPL_ENABLE_BLAS:BOOL=ON \
> -D BLAS_INCLUDE_DIRS:PATH="/usr/local/openblas/" \
> -D BLAS_LIBRARY_DIRS:FILEPATH="/usr/local/openblas/lib" \
> -D BLAS_LIBRARY_NAMES:STRING="libopenblas.a" \
> \
> -D TPL_ENABLE_LAPACK:BOOL=ON \
> -D LAPACK_INCLUDE_DIRS:PATH="/usr/local/openblas/" \
> -D LAPACK_LIBRARY_DIRS:FILEPATH="/usr/local/openblas/lib" \
> -D LAPACK_LIBRARY_NAMES:STRING="libopenblas.a" \
> \
> -D TPL_ENABLE_MPI:BOOL=ON \
> \
> -D Trilinos_ENABLE_DEBUG:BOOL=ON \
> -D Trilinos_ENABLE_TESTS:BOOL=ON \
> -D Trilinos_ENABLE_OpenMP:BOOL=ON \
> -D Trilinos_ENABLE_Zoltan:BOOL=ON \
> -D Trilinos_ENABLE_Epetra:BOOL=ON \
> -D Trilinos_ENABLE_EpetraExt=ON \
> -D Trilinos_ENABLE_Kokkos:BOOL=ON \
> -D Kokkos_ENABLE_EXAMPLES:BOOL=ON \
> -D Trilinos_ENABLE_Teuchos:BOOL=ON \
> -D Trilinos_ENABLE_TEUCHOS_TIME_MONITOR:BOOL=ON \
> \
> -D CMAKE_INSTALL_PREFIX:PATH="/usr/local/trilinos" \
> -D Trilinos_INSTALL_INCLUDE_DIR:PATH="include" \
> -D Trilinos_INSTALL_LIB_DIR:PATH="lib" \
> -D Trilinos_INSTALL_RUNTIME_DIR:PATH="bin" \
> \
> $EXTRA_ARGS \
> ../trilinos-11.4.3-Source
>
> **************************************************************************************
>
>
> Stampede:
>
> **************************************************************************************
> #!/usr/bin/env bash
>
> rm -f MakeCache.txt
>
> EXTRA_ARGS=$@
>
> cmake -D CMAKE_INSTALL_PREFIX:PATH=$HOME/trilinos/ \
> \
> -D MPI_BASE_DIR:PATH=$MPICH_HOME \
> \
> -D CMAKE_BUILD_TYPE:STRING=Release \
> -D CMAKE_Fortran_COMPILER:STRING="mpif90" \
> -D CMAKE_CXX_FLAGS:STRING="-O2 -Wall -ansi -Wno-long-long -fp-model
> precise -fp-model source -ip -xHost" \
> -D BUILD_SHARED_LIBS:BOOL=ON \
> -D Trilinos_WARNINGS_AS_ERRORS_FLAGS:STRING="" \
> \
> -D Trilinos_ENABLE_ALL_PACKAGES:BOOL=OFF \
> -D Trilinos_ENABLE_ALL_OPTIONAL_PACKAGES:BOOL=OFF \
> -D Trilinos_ENABLE_ALL_FORWARD_DEP_PACKAGES:BOOL=OFF \
> -D Trilinos_ENABLE_Teuchos:BOOL=ON \
> -D Trilinos_ENABLE_Epetra:BOOL=ON \
> -D Trilinos_ENABLE_EpetraExt:BOOL=ON \
> -D Trilinos_ENABLE_AztecOO:BOOL=ON \
> -D Trilinos_ENABLE_Zoltan:BOOL=ON \
> -D Trilinos_ENABLE_SECONDARY_STABLE_CODE:BOOL=ON \
> \
> -D Trilinos_ENABLE_EXAMPLES:BOOL=OFF \
> -D Trilinos_ENABLE_TESTS:BOOL=OFF \
> \
> -D TPL_ENABLE_MATLAB:BOOL=OFF \
> -D TPL_ENABLE_Matio:BOOL=OFF \
> -D TPL_ENABLE_MPI:BOOL=ON \
> -D TPL_ENABLE_BLAS:BOOL=ON \
> -D TPL_BLAS_LIBRARIES:STRING="-L$MKLROOT/lib/intel64 -lmkl_intel_lp64
> -lmkl_blas95_lp64 -lmkl_core -lmkl_sequential" \
> -D TPL_ENABLE_LAPACK:BOOL=ON \
> -D TPL_LAPACK_LIBRARIES:STRING="-L$MKLROOT/lib/intel64
> -lmkl_lapack95_lp64" \
> -D TPL_ENABLE_Boost:BOOL=ON \
> -D TPL_Boost_INCLUDE_DIRS:PATH=$TACC_BOOST_INC \
> \
> -D CMAKE_VERBOSE_MAKEFILE:BOOL=OFF \
> \
> $EXTRA_ARGS \
> $HOME/trilinos/trilinos-11.4.3-Source
>
> #make -j 8
> #make install
>
> **************************************************************************************
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://software.sandia.gov/pipermail/trilinos-users/attachments/20140618/cce72e84/attachment-0001.html
> >
>
> ------------------------------
>
> _______________________________________________
> Trilinos-Users mailing list
> Trilinos-Users at software.sandia.gov
> https://software.sandia.gov/mailman/listinfo/trilinos-users
>
>
> End of Trilinos-Users Digest, Vol 106, Issue 17
> ***********************************************
>
--
*Riccardo Rossi*
PhD, Civil Engineer
member of the Kratos Team: www.cimne.com/kratos
lecturer at Universitat Politècnica de Catalunya, BarcelonaTech (UPC)
Research fellow at International Center for Numerical Methods in
Engineering (CIMNE)
C/ Gran Capità, s/n, Campus Nord UPC, Ed. C1, Despatx C9
08034 – Barcelona – Spain – www.cimne.com -
T.(+34) 93 401 56 96 skype: *rougered4*
<http://www.cimne.com/>
<https://www.facebook.com/cimne> <http://blog.cimne.com/>
<http://vimeo.com/cimne> <http://www.youtube.com/user/CIMNEvideos>
<http://www.linkedin.com/company/cimne> <https://twitter.com/cimne>
Les dades personals contingudes en aquest missatge són tractades amb la
finalitat de mantenir el contacte professional entre CIMNE i voste. Podra
exercir els drets d'accés, rectificació, cancel·lació i oposició,
dirigint-se a cimne at cimne.upc.edu. La utilització de la seva adreça de
correu electronic per part de CIMNE queda subjecte a les disposicions de la
Llei 34/2002, de Serveis de la Societat de la Informació i el Comerç
Electronic.
Imprimiu aquest missatge, només si és estrictament necessari.
<http://www.cimne.com/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://software.sandia.gov/pipermail/trilinos-users/attachments/20140618/7aa3c5b0/attachment.html>
More information about the Trilinos-Users
mailing list