[Trilinos-Users] [EXTERNAL] Re: Problem inserting non-local data in Epetra_FECrsMatrix

Eric Marttila eric.marttila at thermoanalytics.com
Tue Feb 21 09:05:02 MST 2012


All,

I've been using the workaround suggested earlier (making sure I call the 
derived class version of InsertGlobalValues), and my code executes correctly 
on multiple processors.  The problem I've seen is that the insertions of non-
local data are extremely slow.  I've attached a small bit of code which runs 
with a matrix size of 500,000.  When run with 2 processors the total insertion 
time is about 4 minutes (most of which is spent doing the non-local data).  
The insertion time seems to increase exponentially with the matrix size; a 
matrix size of 1 million takes about 20 minutes.  If the code is run with 1 
processor it runs in less than 1 second.

I saw the notes in the Epetra documentation that the GlobalAssemble() is an 
expensive process, so I was expecting somewhat of a slowdown when filling my 
matrix from the master process, but wasn't expecting a slowdown of this 
extent.  Also, in my case it seems that the expensive process is the 
InsertGlobalValues.  (the time for GlobalAssemble is very small in 
comparison).

I have two questions regarding this:
1. Is the performance that I see (with non-local insertions) the expected 
behavior for Epetra_FECrsMatrix ?

2. Is there a different class that I should use instead of Epetra_FECrsMatrix?  
I switched from Epetra_CrsMatrix to Epetra_FECrsMatrix to get the non-local 
data functionality.  But in my case the master process is filling the entire 
matrix, so there is no chance that multiple processes would be setting values 
for a given matrix element.  I'm wondering if FECrsMatrix is overkill for my 
situation.

Thanks.
--Eric


On Wednesday, February 15, 2012 04:04:02 pm Williams, Alan B wrote:
> Thanks Dave,
> That may indeed be the issue. It is an oversight that only one of those
> base-class methods is over-ridden be FECrsMatrix.
> 
> I'm out of town this week but I'll try to get that taken care of soon after
> I get back.
> 
> Alan
> 
> 
> From: Littlewood, David John
> Sent: Wednesday, February 15, 2012 01:59 PM
> To: Williams, Alan B; 'turane at gmail.com' <turane at gmail.com>;
> 'eric.marttila at thermoanalytics.com' <eric.marttila at thermoanalytics.com>
> Cc: 'trilinos-users at software.sandia.gov'
> <trilinos-users at software.sandia.gov> Subject: Re: [Trilinos-Users]
> [EXTERNAL] Re: Problem inserting non-local data in Epetra_FECrsMatrix
> 
> Folks,
> 
> I ran into a similar-sounding issue a few months ago.  I copy-and-pasted
> the e-mail snippet below in the hope that it might help you here…
> 
> Dave
> --
> David Littlewood
> Multiphysics Simulation Technology, Dept. 1444
> 505-284-0830
> 
> 
> ***************************
> 
> I've been working with Epetra_FECrsMatrix and ran into a peculiarity.  In
> the base class Epetra_CrsMatrix, there are two functions called
> InsertGlobalValues:
> 
> virtual int InsertGlobalValues (int GlobalRow, int NumEntries, const double
> *Values, const int *Indices)
> 
> virtual int InsertGlobalValues (int GlobalRow, int NumEntries, double
> *Values, int *Indices)
> 
> The only difference here is "const" on the last two arguments.
> 
> It turns out that the first function is also implemented in the derived
> class Epetra_FECrsMatrix, but the second one is not.  Thus if one calls
> 
> Epetra_FECrsMatrix::InsertGlobalValues (int GlobalRow, int NumEntries,
> const double *Values, const int *Indices)
> 
> you get the FE functionality (ability to insert nonlocal data), and if one
> calls
> 
> Epetra_FECrsMatrix::InsertGlobalValues (int GlobalRow, int NumEntries,
> double *Values, int *Indices)
> 
> you get the base class function, which does not handle nonlocal data.
> 
> ***************************
> 
> 
> From: "Williams, Alan B" <william at sandia.gov<mailto:william at sandia.gov>>
> Date: Wed, 15 Feb 2012 18:55:13 +0000
> To: "'turane at gmail.com<mailto:'turane at gmail.com>'"
> <turane at gmail.com<mailto:turane at gmail.com>>,
> "'eric.marttila at thermoanalytics.com<mailto:'eric.marttila at thermoanalytics.
> com>'"
> <eric.marttila at thermoanalytics.com<mailto:eric.marttila at thermoanalytics.co
> m>> Cc:
> "'trilinos-users at software.sandia.gov<mailto:'trilinos-users at software.sandi
> a.gov>'"
> <trilinos-users at software.sandia.gov<mailto:trilinos-users at software.sandia.
> gov>> Subject: Re: [Trilinos-Users] [EXTERNAL] Re: Problem inserting
> non-local data in Epetra_FECrsMatrix
> 
> Hello,
> The usage of insertGlobalValues that Eric reported trying, should work. I
> believe he has encountered a bug in the implementation of FECrsMatrix.
> I'll look into that.
> 
> Alan
> 
> 
> 
> From: erhan turan [mailto:turane at gmail.com]
> Sent: Wednesday, February 15, 2012 11:28 AM
> To: Eric Marttila
> <eric.marttila at thermoanalytics.com<mailto:eric.marttila at thermoanalytics.co
> m>> Cc:
> trilinos-users at software.sandia.gov<mailto:trilinos-users at software.sandia.g
> ov>
> <trilinos-users at software.sandia.gov<mailto:trilinos-users at software.sandia.
> gov>> Subject: [EXTERNAL] Re: [Trilinos-Users] Problem inserting non-local
> data in Epetra_FECrsMatrix
> 
> Hello,
> 
> I had a similar problem before. My colleague Cyril Flaig found a
> workaround. For non-local entries, you can try SumIntoGlobalValues rather
> than InsertGlobalValues. I mean, If A.LRID()== -1 , use
> SumIntoGlobalValues with global row index and use InsertGlobalValues
> otherwise. That might work.
> 
> For that purpose, what is defined was two pointers say AF and A using
> Epetra_FECrsMatrix and Epetra_CrsMatrix, respectively. Here, we first
> declared AF using the map and later just saying A=AF. Later all methods
> are used for A: A->InsertGlobalValues() and A->SumIntoGlobalValues() .
> Fill is completed using AF->GlobalAssemble(). If you try,
> A->FillComplete() it wont fill non-local entries. That might be not the
> best solution. Maybe there is a simpler way. But it will probably work.
> 
> Since you mentioned this issue, I want to make a reference to
> Epetra_FECrsMatrix documentation. There it reads:
> 
> InsertGlobalValues()<http://trilinos.sandia.gov/packages/docs/r10.8/package
> s/epetra/doc/html/classEpetra__FECrsMatrix.html#a54820e91fb84c7c594c4e6f3ea
> 290fe4> inserts values into the matrix only if the graph has not yet been
> finalized
> (FillComplete()<http://trilinos.sandia.gov/packages/docs/r10.8/packages/ep
> etra/doc/html/classEpetra__CrsMatrix.html#a355b4e8d1e28b16bacf37ab637ac0c3f
> > has not yet been called). For non-local values, the call to
> InsertGlobalValues()<http://trilinos.sandia.gov/packages/docs/r10.8/packag
> es/epetra/doc/html/classEpetra__FECrsMatrix.html#a54820e91fb84c7c594c4e6f3e
> a290fe4> may succeed but
> theGlobalAssemble()<http://trilinos.sandia.gov/packages/docs/r10.8/package
> s/epetra/doc/html/classEpetra__FECrsMatrix.html#aca5191dcdfb99eb6316b86fee4
> 59b243> method may then fail because the non-local data is not actually
> inserted in the underlying matrix until
> GlobalAssemble()<http://trilinos.sandia.gov/packages/docs/r10.8/packages/e
> petra/doc/html/classEpetra__FECrsMatrix.html#aca5191dcdfb99eb6316b86fee459b
> 243> is called.
> 
> This might be the reason that FillComplete() alone does not work but those
> overloaded SumInto methods over GlobalAssemble() work. However I still
> believe that this sentence may be vague :)
> 
> Erhan Turan
> Department of Computer Science
> ETH Zurich
> 
> 
> On Wed, Feb 15, 2012 at 4:19 PM, Eric Marttila
> <eric.marttila at thermoanalytics.com<mailto:eric.marttila at thermoanalytics.co
> m>> wrote: Dear all,
> 
> I've been using Epetra_CrsMatrix to store a linear system, but recently
> encountered a situation where I need to fill the entire matrix from the
> master process.  From reading the Epetra documentation I believe I need to
> switch to a EpetraFECrs_Matrix to do this.
> 
> I changed my code to use the EpetraFECrs_Matrix, but my call to
> InsertGlobalValues() fails on the first row of the matrix that is non-local
> to the master process.
> 
> I've attached a small section of code that reproduces the behavior.  It
> runs fine with 1 process, but fails when inserting row index 2 when run
> with 2 processors.
> 
> I would appreciate any thoughts on what I'm missing with my usage of
> EpetraFECrs_Matrix.
> 
> Thanks.
> --Eric
> 
> p.s. I'm running Trilinos 10.8.5 on linux x86_64, using mpich2.
> 
> --
> Eric A. Marttila
> ThermoAnalytics, Inc.
> 23440 Airpark Blvd.
> Calumet, MI 49913
> 
> email:
> Eric.Marttila at ThermoAnalytics.com<mailto:Eric.Marttila at ThermoAnalytics.com
> > phone: 810-636-2443
> fax:   906-482-9755
> web: http://www.thermoanalytics.com
> 
> _______________________________________________
> Trilinos-Users mailing list
> Trilinos-Users at software.sandia.gov<mailto:Trilinos-Users at software.sandia.go
> v> http://software.sandia.gov/mailman/listinfo/trilinos-users
> 
> 
> _______________________________________________ Trilinos-Users mailing list
> Trilinos-Users at software.sandia.gov<mailto:Trilinos-Users at software.sandia.g
> ov> http://software.sandia.gov/mailman/listinfo/trilinos-users

-- 
Eric A. Marttila
ThermoAnalytics, Inc.
23440 Airpark Blvd.
Calumet, MI 49913

email: Eric.Marttila at ThermoAnalytics.com
phone: 810-636-2443
fax:   906-482-9755
web: http://www.thermoanalytics.com
-------------- next part --------------
/*--------------------------------------------------------------------*/
#include <time.h>
#include "mpi.h"
#include "Epetra_FECrsMatrix.h"
#include "Epetra_Map.h"
#include "Epetra_MpiComm.h"

int main(int argCount, char **argValue)
{
  int ierr;
  MPI_Init(&argCount,&argValue);
  Epetra_MpiComm Comm(MPI_COMM_WORLD);
  const int rank = Comm.MyPID();

  // Construct a Map 
  int nGlobalElements = 500000;
  Epetra_Map Map(nGlobalElements, 0, Comm);

  // Create a matrix
  Epetra_FECrsMatrix A(Copy, Map, 1);

  time_t startTime = 0;
  if (rank == 0) {
    startTime = time(0);
  }

  // Fill matrix on the master process
  if (rank == 0) {
    double values[1];
    int    indices[1];
    const int numEntries = 1;

    for (int globalRowIdx=0; globalRowIdx<nGlobalElements; ++globalRowIdx) {
      indices[0] = globalRowIdx;
      values[0] = 3.2 + globalRowIdx*0.01;

      if (globalRowIdx % 5000 == 0) {
	cerr << "About to insert row " << globalRowIdx << "\n";
      }

      ierr = A.InsertGlobalValues( globalRowIdx, numEntries,
				   (const double *)&values[0], 
				   (const int *)&indices[0] );

      assert(ierr==0);
    }
  }

  double insertionTime = 0;
  if (rank == 0) {
    time_t endTime = time(0);
    insertionTime = difftime(endTime, startTime);
  }

  // Finish up
  ierr = A.GlobalAssemble();
  assert(ierr==0);

  if (rank == 0) {
    cerr << "insertion time = " << insertionTime << " (seconds)\n";
  }


  MPI_Finalize();

  return 0;
}
/*--------------------------------------------------------------------*/


More information about the Trilinos-Users mailing list