[Trilinos-Users] Matvec-operation for distributed memory machines/MPI and dense matrix/vector

[Heroux, Michael A]

Roland,

What are the details of your dense matrix use?  

If you are looking for a function to work with a square or nearly-square dense matrix, Trilinos doesn't have a ready-to-use approach.  For MPI-only execution (no GPU or shared memory parallel execution under MPI), we do have a package called Pliris that provides dense LU factorization for square matrices with an optimal distribution.  There is surely a function in Pliris for computing a dense matvec, but I don't think that it is exposed as a separate function.  If your matrix is square or nearly square, a sqrt(p) by sqrt(p) distribution of the matrix is typically optimal, where p is the number of MPI ranks.  We do not have easy-to-use functionality for this kind of distribution.  Outside of Pliris, which is holistically focused on dense LU, we don't expose basic kernels for sqrt(p) by sqrt(p).

If you are looking for support for dense matvec in the context of block Krylov methods or something similar, the Epetra package provides this with good performance (when linked with optimized BLAS).  In this case, the dense matrix is strongly rectangular with many more rows than columns, or columns than rows.  The matrix has a 1D distribution over the number of MPI ranks, with typically m/p rows per MPI rank, where m is the number of rows and p is the number of MPI ranks (or the equivalent when the number of columns >> number of rows).

The class that provides this functionality is Epetra_MultiVector and the key method is called Multiply().  If the BLAS you link to have shared memory parallel thread support, e.g., MKL blas from Intel, you can also see multicore performance improvements under MPI.

If you want explicit support for threads or GPUs, the Trilinos package Tpetra provides a MultiVector class with this capability in combination with Kokkos, but this support requires a fairly significant ramp up for most users.

I hope this is helpful.

Mike

On 8/7/19, 6:09 AM, "Trilinos-Users on behalf of Roland Richter" <trilinos-users-bounces at trilinos.org on behalf of roland.richter at ntnu.no> wrote:

    Hei,
    
    I would like to use Trilinos for MatVec-operations (i.e. A*x = b), with
    A and x a distributed dense matrix/vector. The whole operation should
    run using MPI threads. I found possible approaches for sparse matrices,
    but none so far for dense matrices. Are there options of doing that in
    Trilinos (and maybe some examples)?
    
    Thanks!
    
    
    _______________________________________________
    Trilinos-Users mailing list
    Trilinos-Users at trilinos.org
    https://trilinos.org/mailman/listinfo/trilinos-users