[Trilinos-Users] [EXTERNAL] Hybrid parallelism in Trilinos

[Max Grossman]

Eric,

Thanks for the tip! You were exactly right, I was using OpenMPI and adding ‘--bind-to socket’ to the mpirun arguments fixed the issue.

Max

> On Jan 23, 2016, at 6:16 AM, Phipps, Eric T <etphipp at sandia.gov> wrote:
> 
> Which MPI library and version are you using?  Your MPI ranks are probably being bound to cores, in which case Kokkos only sees one thread.  You want to turn that off (or even better bind to sockets instead and do one MPI rank per socket).  The MPI options to do that depend on which library you are using.
> 
> -Eric
> 
>> On Jan 22, 2016, at 8:21 PM, Max Grossman <jmg3 at rice.edu> wrote:
>> 
>> Hi all,
>> 
>> I’m having trouble getting hybrid (MPI + OpenMP) parallelism working with the Trilinos/Tpetra SpMV kernel and was hoping someone here might be able to help. My approach is to use the MpiComm communication and omp_set_num_threads to try to control the parallelism in my SpMV kernel when I call Tpetra::CrsMatrix::Apply. While I can confirm that running with SerialComm and setting omp_num_threads results in the correct number of threads being used, when I change that to MpiComm Kokkos seems to be ignoring the set number of OpenMP threads and hard-coding it to one thread. I’ve verified this behavior by observing the output of top during a run.
>> 
>> So my question is, how can I achieve hybrid parallelism in Tpetra/Kokkos? Or is distributed and multi-threaded programming not currently supported in the same Trilinos program?
>> 
>> Thanks for any help,
>> 
>> Max
>> _______________________________________________
>> Trilinos-Users mailing list
>> Trilinos-Users at trilinos.org
>> https://trilinos.org/mailman/listinfo/trilinos-users
>