[Trilinos-Users] Matrix-matrix product with large CrsMatrices

[Christopher Thiele]

Hello,

I have a code that involves a matrix-matrix multiplication (A*B) using the
Tpetra::MatrixMatrix::Multiply() function. As far as I understand, this
function first transfers remote parts of one of the matrices via MPI and
then computes the actual matrix-matrix product. However, for large matrices
this does not seem to work for two reasons:

1. If only the rows of the matrices are distributed across MPI processes,
i.e. if a process always owns all or none of the entries of a given row,
then basically the full matrix B must be present on each process to compute
the matrix-matrix product, which somehow makes it pointless to distribute
it in the first place. Especially the global number of non-zeros is
restricted by the memory available to one process, despite distributing the
matrix.

2. In fact, the non-zeros per process seem to be restricted to 2 GB anyway,
because they are communicated with MPI functions that take a 32 bit integer
as byte count. One cannot circumvent this using e.g. a
Teuchos::MpiComm<int64_t> communicator, because the Tpetra::Map class
expects a Teuchos::Comm<int> communicator. At least this is what I observed
when I tried larger matrices.

Do you see a way to get around these issues without re-writing the
Multiply() function, or am I maybe missing something here? I appreciate any
suggestions.

Regards,
Christopher
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://trilinos.org/pipermail/trilinos-users/attachments/20151012/60120d3d/attachment.html>