[Trilinos-Users] Hybrid parallelism in Trilinos

[Max Grossman]

Hi all,

I’m having trouble getting hybrid (MPI + OpenMP) parallelism working with the Trilinos/Tpetra SpMV kernel and was hoping someone here might be able to help. My approach is to use the MpiComm communication and omp_set_num_threads to try to control the parallelism in my SpMV kernel when I call Tpetra::CrsMatrix::Apply. While I can confirm that running with SerialComm and setting omp_num_threads results in the correct number of threads being used, when I change that to MpiComm Kokkos seems to be ignoring the set number of OpenMP threads and hard-coding it to one thread. I’ve verified this behavior by observing the output of top during a run.

So my question is, how can I achieve hybrid parallelism in Tpetra/Kokkos? Or is distributed and multi-threaded programming not currently supported in the same Trilinos program?

Thanks for any help,

Max