[Trilinos-Users] Trilinos-Users Digest, Vol 103, Issue 2

[Hoemmen, Mark]

On Mar 5, 2014, at 12:00 PM, <trilinos-users-request at software.sandia.gov>
 <trilinos-users-request at software.sandia.gov> wrote:
> Message: 2
> Date: Wed, 5 Mar 2014 14:30:14 +0000
> From: <Jonas.Thies at dlr.de>
> Subject: [Trilinos-Users] experiences with Trilinos and OpenMP
> To: <trilinos-users at software.sandia.gov>
> Message-ID:
> 	<DC79F9D2C597AF428838F0B5DE2957CF0A3349B2 at dlrexmbx02.intra.dlr.de>
> Content-Type: text/plain; charset="iso-8859-1"
> 
> Hi,
> 
> we're getting a bit frustrated trying to combine Epetra and Teuchos with
> OpenMP. A few things just seem to be wrong, for instance CrsMatrix::SumIntoGlobalValues
> has a line
> 
> #ifdef EPETRA_HAVE_OMP_NONASSOCIATIVE
> #pragma omp atomic
> #endif
> 
> I think this should always be done atomic because FECrsMatrix uses this function
> and several threads thus might sum into the same row simultaneously. At least some
> synchronization is needed here.
> 
> Also it seems that Teuchos::RCP does not work correctly if used in an OpenMP section,
> but I can't give a simple example of that right now. Is this a known problem?
> 
> Does anyone have similar experience with OpenMP and Epetra?

If you really want to use Epetra with OpenMP, it would be wise to enable the "nonassociative" option.  There are actually two OpenMP options with Epetra.  Just turning on OpenMP makes Epetra thread-parallel only where rearranging operations would not affect the result in floating-point arithmetic.  For example, sparse matrix-vector multiply is parallel over matrix rows in that case, as is vector update (e.g., y := alpha*x + y).  However, dot products and norms are _NOT_ parallel unless you enable the nonassociative option.  Radu Popescu (who implemented the above atomic update functionality) understood that atomic updates only promise atomicity, but may change the order of operations.  Thus, it is appropriate to require the nonassociative option in order to get atomic updates.

mfh