[Trilinos-Users] Tpetra threading questions

[Hoemmen, Mark]

On 4/14/14 12:00 PM, "trilinos-users-request at software.sandia.gov"
<trilinos-users-request at software.sandia.gov> wrote:
>Message: 1
>Date: Mon, 14 Apr 2014 09:03:11 -0700
>From: David Hysom <hysom1 at llnl.gov>
>To: trilinos-users at software.sandia.gov
>Subject: [Trilinos-Users] Tpetra threading questions
>Message-ID: <534C06BF.5090708 at llnl.gov>
>Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
>Hi,
>
>When constructing a map we pass the number of threads
>in a parameter list. I had thought that meant that each map
>(and subsequently CrsMatrix) had an independent number of
>threads. I've just figured out that, for OpenMP nodes, Tpetra
>calls set_num_threads(), which is globally scoped.
>
>This causes difficulties for our scenario: we have a pool of
>matrices, and we periodically add matrices to the pool, or
>extract a matrix for use (e.g, LOBPCG). The matrices can
>vary greatly in size, and we've been constructing a Map for
>each, with a number of threads intended to optimize performance.
>Now we've learned that the number of threads will be whatever
>was used in the most recently constructed matrix, due to the
>call to set_num_threads().
>
>My question is: do the TBB and TPI nodes act similarly, e.g, the
>number of threads is a global attribute?

Yes.  The number of threads is global for all Kokkos Node types.  The
number of threads parameter is meant to go to the Kokkos Node instance at
the time of its construction, but that only helps you if you pass the Node
instance around and don't rely on Map's constructor to create a default
one.

Some questions for you:

  1. Do you intend to run multiple simultaneous LOBPCG solves using
threads?  This may not work correctly, since Teuchos::RCP is not currently
thread safe.

  2. We are currently refactoring Tpetra's support for threads, to support
thread-parallel fill into data structures.  If you really want each Map to
have a different number of threads, we should talk now about interfaces
and expectations.

Please feel free to reply to me privately if you would prefer not to
discuss these things on the list.

Thanks!
mfh