[Trilinos-Users] [EXTERNAL] Re: GMRES inconsistent when using multiple threads (Belos+Tpetra, since 12.12.1)

[Eligiusz Postek]

Thank you Ross, Cheers, Elek

On Fri, Nov 10, 2017 at 12:55 AM, Bartlett, Roscoe A <rabartl at sandia.gov>
wrote:

> Elek,
>
>
>
> It is likely that CMAKE_CXX_COMPILER, CMAKE_C_COMPILER, and
> CMAKE_Fortran_COMPILER are getting set directly somehow.  In this case, it
> will ignore the values in the MPI_<LANG>_COMPILER variables.
>
>
>
> This is mentioned at:
>
>
>
>    https://trilinos.org/docs/files/TrilinosBuildReference.
> html#configuring-with-mpi-support
>
>
>
> which says:
>
>
>
> If both CMAKE_<LANG>_COMPILER and MPI_<LANG>_COMPILER are set, however,
> then CMAKE_<LANG>_COMPILER will be used and MPI_<LANG>_COMPILER will be
> ignored.
>
>
>
> Cheers,
>
>
>
> -Ross
>
>
>
> Dr. Roscoe A. Bartlett, PhD
>
> http://www.cs.sandia.gov/cr-rabartl
>
> Sandia National Laboratories
>
>
>
> *From:* Trilinos-Users [mailto:trilinos-users-bounces at trilinos.org] *On
> Behalf Of *Eligiusz Postek
> *Sent:* Thursday, November 09, 2017 1:55 PM
> *To:* Christopher Thiele <ct37 at rice.edu>
> *Cc:* trilinos-users at trilinos.org
> *Subject:* [EXTERNAL] Re: [Trilinos-Users] GMRES inconsistent when using
> multiple threads (Belos+Tpetra, since 12.12.1)
>
>
>
> Hello,
>
> just a remark.
>
> When I set the variables for MPI compilers
>
> I always get
>
> CMake Warning:
>   Manually-specified variables were not used by the project:
>
>     MPI_CXX_COMPILER
>     MPI_C_COMPILER
>     MPI_Fortran_COMPILER
>
> Are they "old" already not used variables
>
> or anything else should be done?
>
> Best regards,
>
> Elek
>
>
>
>
>
> On Thu, Nov 9, 2017 at 6:26 PM, Christopher Thiele <ct37 at rice.edu> wrote:
>
> Hello,
>
> I noticed some strange behavior in Belos' GMRES solver when using Tpetra
> and multiple threads. I wrote a little tool (solver_driver) that just reads
> in a matrix market file and solves a linear system to reproduce it. Here
> are some results with the rma10 matrix from the University of Florida
> collection (the choice is rather arbitrary):
>
> ./solver_driver_12.10.1 --threads=2 --matrix=rma10.mtx --rhs=rma10_b.mtx
> --tol=1.0e-1 --solver=GMRES --disable-preconditioner
> ---> Converges in just 3 iterations, independent of the number of threads
> etc. This seems reasonable given the tolerance of 1e-1.
>
> ./solver_driver_12.12.1 --threads=2 --matrix=rma10.mtx --rhs=rma10_b.mtx
> --tol=1.0e-1 --solver=GMRES --disable-preconditioner
> ---> Converges in either 61 or 62 iterations, but the final residual
> changes in the order of 1e-4 with each run.
>
> OMP_PROC_BIND=true ./solver_driver_12.12.1 --threads=2 --matrix=rma10.mtx
> --rhs=rma10_b.mtx --tol=1.0e-1 --solver=GMRES --disable-preconditioner
> ---> Converges in 121 iterations, and the final residual changes in the
> order of 1e-3.
>
> mpirun -n 1 ./solver_driver_12.12.1 --threads=2 --matrix=rma10.mtx
> --rhs=rma10_b.mtx --tol=1.0e-1 --solver=GMRES --disable-preconditioner
> ---> Seems to solve the problem, consistently reproduces the result
> obtained with 12.10.1
>
> I also tested the current master branch from GitHub, and it shows the same
> behavior as the 12.12.1 release. The problem does not occur with other
> solvers like BiCGStab or CG. I added some information about my system and
> the Trilinos configuration below. I can also share the code (~200 loc), but
> it is really just a driver for the solvers.
>
> Best regards,
> Christopher
>
>
>
> System:
>
> OS: RHEL 7 (x84_64)
> Compiler: GCC 4.8.5
> MPI: Open MPI 1.10.6
> CPU: Xeon E3-1220 V2 (4 cores)
> Memory: 8 GB
>
>
>
> Trilinos configuration script:
>
> #!/bin/bash
>
> set -e
>
> SOURCE_BASE=/path/to/source
> INSTALL_DIR=/path/to/installation
>
> TRILINOS_PKG_CONF=" \
>   -DTrilinos_ENABLE_Teuchos=ON \
>   -DTrilinos_ENABLE_TeuchosParser=ON \
>   -DTrilinos_ENABLE_Belos=ON \
>   -DTrilinos_ENABLE_Ifpack2=ON \
>   -DTrilinos_ENABLE_MueLu=OFF \
>   -DTrilinos_ENABLE_Tpetra=ON \
>   -DTrilinos_ENABLE_Teko=OFF \
>   -DTrilinos_ENABLE_Zoltan2=OFF \
>   -DTrilinos_ASSERT_MISSING_PACKAGES=OFF \
>   "
>
> TP_PKG_CONF=" \
>   -DTPL_ENABLE_MPI=ON \
>   -DTPL_ENABLE_CUDA=OFF \
>   -DTPL_ENABLE_MKL=ON \
>   -DTPL_MKL_LIBRARIES=/path/to/mkl/lib/intel64/libmkl_rt.so \
>   -DTPL_MKL_INCLUDE_DIRS=/path/to/mkl/include \
>     "
>
> cmake \
>   -D MPI_C_COMPILER=mpicc \
>   -D MPI_CXX_COMPILER=mpicxx \
>   -D MPI_Fortran_COMPILER=mpifort \
>   -D CMAKE_C_COMPILER=mpicc \
>   -D CMAKE_CXX_COMPILER=mpicxx \
>   -D CMAKE_Fortran_COMPILER=mpifort \
>   -D CMAKE_CXX_FLAGS="-g -O3 -march=native" \
>   -D CMAKE_BUILD_TYPE=RELEASE \
>   -D Trilinos_ENABLE_TESTS=OFF \
>   -D Trilinos_ENABLE_EXPLICIT_INSTANTIATION=ON \
>   -D Trilinos_CXX11_FLAGS="-std=c++11" \
>   -D Kokkos_ENABLE_Pthread=OFF \
>   -D Kokkos_ENABLE_OpenMP=ON \
>   -D Kokkos_ENABLE_Cuda=OFF \
>   -D Kokkos_ENABLE_Cuda_UVM=OFF \
>   -D Trilinos_ENABLE_OpenMP=ON \
>   -D Tpetra_INST_SERIAL=ON \
>   -D Tpetra_INST_CUDA=OFF \
>   -D Tpetra_INST_OPENMP=ON \
>   -D Tpetra_INST_INT_INT=ON \
>   -D Tpetra_INST_INT_LONG=OFF \
>   -D Tpetra_INST_COMPLEX_DOUBLE=OFF \
>   -D Teuchos_ENABLE_LONG_LONG_INT=OFF \
>   -D CMAKE_INSTALL_PREFIX=${INSTALL_DIR} \
>   ${TRILINOS_PKG_CONF} \
>   ${TP_PKG_CONF} \
>   ${SOURCE_BASE}
> _______________________________________________
> Trilinos-Users mailing list
> Trilinos-Users at trilinos.org
> https://trilinos.org/mailman/listinfo/trilinos-users
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://trilinos.org/pipermail/trilinos-users/attachments/20171110/cb49995d/attachment.html>