[Trilinos-Users] [EXTERNAL] Re: GMRES inconsistent when using multiple threads (Belos+Tpetra, since 12.12.1)

[Bartlett, Roscoe A]

Elek,

It is likely that CMAKE_CXX_COMPILER, CMAKE_C_COMPILER, and CMAKE_Fortran_COMPILER are getting set directly somehow.  In this case, it will ignore the values in the MPI_<LANG>_COMPILER variables.

This is mentioned at:

   https://trilinos.org/docs/files/TrilinosBuildReference.html#configuring-with-mpi-support

which says:

If both CMAKE_<LANG>_COMPILER and MPI_<LANG>_COMPILER are set, however, then CMAKE_<LANG>_COMPILER will be used and MPI_<LANG>_COMPILER will be ignored.

Cheers,

-Ross

Dr. Roscoe A. Bartlett, PhD
http://www.cs.sandia.gov/cr-rabartl
Sandia National Laboratories

From: Trilinos-Users [mailto:trilinos-users-bounces at trilinos.org] On Behalf Of Eligiusz Postek
Sent: Thursday, November 09, 2017 1:55 PM
To: Christopher Thiele <ct37 at rice.edu>
Cc: trilinos-users at trilinos.org
Subject: [EXTERNAL] Re: [Trilinos-Users] GMRES inconsistent when using multiple threads (Belos+Tpetra, since 12.12.1)

Hello,
just a remark.
When I set the variables for MPI compilers
I always get

CMake Warning:
  Manually-specified variables were not used by the project:

    MPI_CXX_COMPILER
    MPI_C_COMPILER
    MPI_Fortran_COMPILER
Are they "old" already not used variables
or anything else should be done?
Best regards,
Elek



On Thu, Nov 9, 2017 at 6:26 PM, Christopher Thiele <ct37 at rice.edu<mailto:ct37 at rice.edu>> wrote:
Hello,

I noticed some strange behavior in Belos' GMRES solver when using Tpetra and multiple threads. I wrote a little tool (solver_driver) that just reads in a matrix market file and solves a linear system to reproduce it. Here are some results with the rma10 matrix from the University of Florida collection (the choice is rather arbitrary):

./solver_driver_12.10.1 --threads=2 --matrix=rma10.mtx --rhs=rma10_b.mtx --tol=1.0e-1 --solver=GMRES --disable-preconditioner
---> Converges in just 3 iterations, independent of the number of threads etc. This seems reasonable given the tolerance of 1e-1.

./solver_driver_12.12.1 --threads=2 --matrix=rma10.mtx --rhs=rma10_b.mtx --tol=1.0e-1 --solver=GMRES --disable-preconditioner
---> Converges in either 61 or 62 iterations, but the final residual changes in the order of 1e-4 with each run.

OMP_PROC_BIND=true ./solver_driver_12.12.1 --threads=2 --matrix=rma10.mtx --rhs=rma10_b.mtx --tol=1.0e-1 --solver=GMRES --disable-preconditioner
---> Converges in 121 iterations, and the final residual changes in the order of 1e-3.

mpirun -n 1 ./solver_driver_12.12.1 --threads=2 --matrix=rma10.mtx --rhs=rma10_b.mtx --tol=1.0e-1 --solver=GMRES --disable-preconditioner
---> Seems to solve the problem, consistently reproduces the result obtained with 12.10.1

I also tested the current master branch from GitHub, and it shows the same behavior as the 12.12.1 release. The problem does not occur with other solvers like BiCGStab or CG. I added some information about my system and the Trilinos configuration below. I can also share the code (~200 loc), but it is really just a driver for the solvers.

Best regards,
Christopher



System:

OS: RHEL 7 (x84_64)
Compiler: GCC 4.8.5
MPI: Open MPI 1.10.6
CPU: Xeon E3-1220 V2 (4 cores)
Memory: 8 GB



Trilinos configuration script:

#!/bin/bash

set -e

SOURCE_BASE=/path/to/source
INSTALL_DIR=/path/to/installation

TRILINOS_PKG_CONF=" \
  -DTrilinos_ENABLE_Teuchos=ON \
  -DTrilinos_ENABLE_TeuchosParser=ON \
  -DTrilinos_ENABLE_Belos=ON \
  -DTrilinos_ENABLE_Ifpack2=ON \
  -DTrilinos_ENABLE_MueLu=OFF \
  -DTrilinos_ENABLE_Tpetra=ON \
  -DTrilinos_ENABLE_Teko=OFF \
  -DTrilinos_ENABLE_Zoltan2=OFF \
  -DTrilinos_ASSERT_MISSING_PACKAGES=OFF \
  "

TP_PKG_CONF=" \
  -DTPL_ENABLE_MPI=ON \
  -DTPL_ENABLE_CUDA=OFF \
  -DTPL_ENABLE_MKL=ON \
  -DTPL_MKL_LIBRARIES=/path/to/mkl/lib/intel64/libmkl_rt.so \
  -DTPL_MKL_INCLUDE_DIRS=/path/to/mkl/include \
    "

cmake \
  -D MPI_C_COMPILER=mpicc \
  -D MPI_CXX_COMPILER=mpicxx \
  -D MPI_Fortran_COMPILER=mpifort \
  -D CMAKE_C_COMPILER=mpicc \
  -D CMAKE_CXX_COMPILER=mpicxx \
  -D CMAKE_Fortran_COMPILER=mpifort \
  -D CMAKE_CXX_FLAGS="-g -O3 -march=native" \
  -D CMAKE_BUILD_TYPE=RELEASE \
  -D Trilinos_ENABLE_TESTS=OFF \
  -D Trilinos_ENABLE_EXPLICIT_INSTANTIATION=ON \
  -D Trilinos_CXX11_FLAGS="-std=c++11" \
  -D Kokkos_ENABLE_Pthread=OFF \
  -D Kokkos_ENABLE_OpenMP=ON \
  -D Kokkos_ENABLE_Cuda=OFF \
  -D Kokkos_ENABLE_Cuda_UVM=OFF \
  -D Trilinos_ENABLE_OpenMP=ON \
  -D Tpetra_INST_SERIAL=ON \
  -D Tpetra_INST_CUDA=OFF \
  -D Tpetra_INST_OPENMP=ON \
  -D Tpetra_INST_INT_INT=ON \
  -D Tpetra_INST_INT_LONG=OFF \
  -D Tpetra_INST_COMPLEX_DOUBLE=OFF \
  -D Teuchos_ENABLE_LONG_LONG_INT=OFF \
  -D CMAKE_INSTALL_PREFIX=${INSTALL_DIR} \
  ${TRILINOS_PKG_CONF} \
  ${TP_PKG_CONF} \
  ${SOURCE_BASE}
_______________________________________________
Trilinos-Users mailing list
Trilinos-Users at trilinos.org<mailto:Trilinos-Users at trilinos.org>
https://trilinos.org/mailman/listinfo/trilinos-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://trilinos.org/pipermail/trilinos-users/attachments/20171109/862254e2/attachment-0001.html>