[Trilinos-Users] Epetra, Isorropia (Zoltan) - 2D matrix partition

[Gyorgy Matyasfalvi]

Hello C.S. and Erik,
Thanks for the responses!

So to be specific, say we are dealing with a sparse matrix A, where the
number of rows is 400,000 and number of columns is 20,000,000.

I want to do matrix vector multiplies with that matrix. So my vectors are
20,000,000 long.

Then say on 1000 cores Epetra_Map will map 400 rows to each core, right?
First processor has rows 0 to 399, second has rows 400 to 799 etc. And if I
feed that map into Epetra_CrsMatrix then processor 1 owns the non-zeros
from the first 400 rows, processor 2 from the second 400 etc.

If I feed the Epetra_CrsMatrix into Isorropia it will rebalance it so that
the number of non-zeros is about the same on each core, right?

Then, once my Epetra_CrsMatrix is filled I could extract an Epetra_Map for
the columns from it, which I could use as map for my right hand side
vector, right?

And then I'd be set to do sparse matrix vector multiplication.


But let's assume for example I have a row in A, where the number of
non-zeros is 1,000,000. In that case I would have one processor that is
doing arithmetic with a vector of 1,000,000, no matter how much I increase
the number of cores. So to me this seems like a possible bottleneck in
terms of scaling.

So my question is: Is it possible to force a different mapping of the
matrix? E.g. one, where you gather the non-zeros from say blocks of 40,000
x 200,000, instead of 400 x 20,000,000. This way you can guarantee that
your right hand side vector cannot be longer than 200,000. And this way you
would avoid bottlenecks created by a "dense" row. (I'm aware this would
require more communication)

Not sure if that makes, sense. Please let me know if there is any further
clarifications required.

Thanks a lot! Best,
Gyorgy



On Wed, Aug 20, 2014 at 6:12 PM, Erik Boman <egboman at sandia.gov> wrote:

> Gyorgy,
>
> I suppose by "2D partitioning" you refer to the partitioning of the
> nonzeros within a matrix? In that case, rows are actually split among
> processes and this breaks some packages in Trilinos. The short answer is
> that Isorropia does not currently support 1D-to-2D rebalancing (but it is
> something we are considering for the future). However, Epetra works with
> matrices distributed in 2D fashion, in particular, matvecs work well.
> Unfortunately, most preconditioners do not support this.
>
> I am curious what your application is and why you think 2D would be better
> (more scalable) than 1D distribution.
>
> Thanks,
> Erik
>
> PS: Partitioning of matrix rows based on 2D geometry is a completely
> different meaning of "2D partitioning".
>
> Gyorgy Matyasfalvi wrote:
>
>> Dear Users:
>> I was wondering if it is possible to create a 2D partitioned Epetra
>> sparse matrix and then feed that to Isorropia for rebalancing?
>> To me it seems that all the Epetra sparse matrices are 1D row
>> partitioned. But the matrix I'm dealing with is too big for 1D partitioning
>> to be enough.
>> Thanks for the help!
>> Best,
>> Gyorgy ------------------------------------------------------------
>> ------------
>>
>> _______________________________________________
>> Trilinos-Users mailing list
>> Trilinos-Users at software.sandia.gov
>> https://software.sandia.gov/mailman/listinfo/trilinos-users
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://software.sandia.gov/pipermail/trilinos-users/attachments/20140820/6b02e5f5/attachment.html>