[Trilinos-Users] integrate out-of-core matrix implementation with Trilinos

[Hoemmen, Mark]

On 3/17/15, 5:33 PM, "trilinos-users-request at software.sandia.gov"
<trilinos-users-request at software.sandia.gov> wrote:
>Message: 1
>Date: Tue, 17 Mar 2015 14:01:03 -0400
>From: Zheng Da <dzheng5 at jhu.edu>
>To: trilinos-users at software.sandia.gov, maherou at sandia.gov,
>	hkthorn at sandia.gov,	rblehou at sandia.gov
>Cc: joshua vogelstein <jovo at jhu.edu>, Randal Burns
>	<randal at cs.jhu.edu>,	Disa Mhembere <dmhembe1 at jhu.edu>
>Subject: [Trilinos-Users] integrate out-of-core matrix implementation
>	with	Trilinos
>Message-ID:
>	<CAFLer831az4AJ1K_mOj1U=F-V_69za57LxvG1GJ1QQR6jdMrzA at mail.gmail.com>
>Content-Type: text/plain; charset="utf-8"
>
>Hello,
>
>We are a team from Johns Hopkins University. We are implementing SSD-based
>matrix operations and especially interested in optimizing sparse matrix
>multiplication on a large SSD array. Our goal is to have matrix operations
>run at the speed comparable to the in-memory implementation as we did for
>graph analysis using SSDs (https://github.com/icoming/FlashGraph).

Hi!  This is a VERY interesting project.  As Rich mentioned, you have many
different implementation options for using Anasazi's iterative
eigensolvers with your matrices.  Here are some possibilities:

  1. Write your own matrix and vector classes.  Implement
Anasazi::MultiVecTraits and Anasazi::OperatorTraits for these classes.
(This does not require Epetra or Tpetra integration.)

  2. If only the matrix lives in SSD, not the vectors, then you may
implement Tpetra::Operator or Epetra_Operator using your matrix, and use
Tpetra's or Epetra's vectors.

  3. If you want to explore tighter integration of SSD storage into linear
algebra classes, let's talk.  Trilinos' Kokkos package (which Tpetra uses)
has some new facilities that could help you do that, as long as it is
possible to memory-map SSD (so that we can access it using pointers,
rather than the file system or some library).  This could be especially
helpful if you plan to distribute your SSD matrices and vectors over
multiple MPI processes, because it would save you the trouble of needing
to implement MPI communication and redistribution.

NONE of these options require reimplementing Epetra or Tpetra.  I have
ranked the options in increasing order of interesting-ness, and increasing
order of software development time.  I am especially interested in Option
3. 


>Another matrix decomposition we like to have is QR decomposition. I'm not
>sure what Trilinos package implements it.

If your matrices are dense and have many more rows relative to columns, we
might be able to help.  Tpetra has a "tall skinny QR" (TSQR) subpackage
that implements a parallel QR factorization for this case.  It is
optimized for <= 20 columns, because that is the case that occurs most
often in block iterative eigensolvers.  We don't provide a general dense
parallel QR decomposition.

>Another question is about maintenance. Reimplementing Epetra or Tpetra
>will
>be a lot of work. If we do implement it, we also hope our implementation
>will work with Trilinos' future release. What are your suggestions of
>making our work more maintainable?

  1. Don't reimplement Epetra or Tpetra (see above) :-)
  2. Test often (at least weekly) against the development branch of
Trilinos, using the public git repository
  3. Test-driven development (write tests before you write code)

This is probably a lot less work than you might think :-)

mfh