[Trilinos-Users] VbrMatrix RAM usage

[Williams, Alan B]

Andrea,

Sorry for the delay in getting back to you about this.

The VbrMatrix storage format definitely incurs some storage overhead, as compared to the simple number of double-precision values being stored.

In your program, you are storing 5,000,000 X 1 X 3 == 15 million doubles. With 8 bytes per double, that is 120 million bytes.

The matrix class Epetra_VbrMatrix (which is the base class for Epetra_FEVbrMatrix) internally stores a Epetra_CrsGraph object, which represents the sparsity structure. This requires a couple of integers per block-row, and 1 integer per block-nonzero. (Your case has 5 million block-rows with 1 block-nonzero per row, so at least 15 million integers in total.)

Additionally, the Epetra_VbrMatrix class stores a Epetra_SerialDenseMatrix object for each block-nonzero. This adds a couple of integers, plus a bool and a pointer, for each block-nonzero. In your case, since your block-nonzeros are small (1x3 doubles), this is a substantial overhead. The VbrMatrix format has proportionally less overhead the bigger your block-nonzeros are. But in your case, with 1x3 blocks, the VbrMatrix is indeed occupying several times more memory than is required for the 15million doubles.

One final note: the VbrMatrix class has a method called OptimizeStorage which condenses storage somewhat, but I just noticed that it has a bug which will seg-fault in certain cases. I will commit a fix for the bug right away, which will be in the next release. If you want that bug-fix, let me know and I can send you the code snippet.

Alan



> -----Original Message-----
> From: trilinos-users-bounces at software.sandia.gov [mailto:trilinos-
> users-bounces at software.sandia.gov] On Behalf Of Andrea Zonca
> Sent: Friday, March 04, 2011 12:20 PM
> To: trilinos-users at software.sandia.gov
> Subject: Re: [Trilinos-Users] VbrMatrix RAM usage
> 
> hi,
> I am really sorry to bug you again but this is really a showstopper for
> my work.
> I continued my debugging on the Carver [1] cluster, with a different
> compiler (PGI),
> with different trilinos version, 10.4.0, getting the same results, a
> VBR matrix storing 1.5 million doubles (about 115 MB)
> has a memory usage of about 500MB per processor when using 2
> processors.
> Could somebody please spend 10 minutes testing my very simple code in
> [2] and
> at least tell me if it is just me or you get the same results?
> I was wondering whether I should try Tpetra, however I need the
> functionality of FEVbrMatrix (setting elements on different
> processors) that is not available in Tpetra.
> thanks,
> andrea zonca
> 
> [1] http://www.nersc.gov/nusers/systems/carver/
> [2] https://gist.github.com/848310
> 
> _______________________________________________
> Trilinos-Users mailing list
> Trilinos-Users at software.sandia.gov
> http://software.sandia.gov/mailman/listinfo/trilinos-users