[Trilinos-Users] [EXTERNAL] Re: Problem inserting non-local data in Epetra_FECrsMatrix
Eric Marttila
eric.marttila at thermoanalytics.com
Tue Feb 21 09:05:02 MST 2012
All,
I've been using the workaround suggested earlier (making sure I call the
derived class version of InsertGlobalValues), and my code executes correctly
on multiple processors. The problem I've seen is that the insertions of non-
local data are extremely slow. I've attached a small bit of code which runs
with a matrix size of 500,000. When run with 2 processors the total insertion
time is about 4 minutes (most of which is spent doing the non-local data).
The insertion time seems to increase exponentially with the matrix size; a
matrix size of 1 million takes about 20 minutes. If the code is run with 1
processor it runs in less than 1 second.
I saw the notes in the Epetra documentation that the GlobalAssemble() is an
expensive process, so I was expecting somewhat of a slowdown when filling my
matrix from the master process, but wasn't expecting a slowdown of this
extent. Also, in my case it seems that the expensive process is the
InsertGlobalValues. (the time for GlobalAssemble is very small in
comparison).
I have two questions regarding this:
1. Is the performance that I see (with non-local insertions) the expected
behavior for Epetra_FECrsMatrix ?
2. Is there a different class that I should use instead of Epetra_FECrsMatrix?
I switched from Epetra_CrsMatrix to Epetra_FECrsMatrix to get the non-local
data functionality. But in my case the master process is filling the entire
matrix, so there is no chance that multiple processes would be setting values
for a given matrix element. I'm wondering if FECrsMatrix is overkill for my
situation.
Thanks.
--Eric
On Wednesday, February 15, 2012 04:04:02 pm Williams, Alan B wrote:
> Thanks Dave,
> That may indeed be the issue. It is an oversight that only one of those
> base-class methods is over-ridden be FECrsMatrix.
>
> I'm out of town this week but I'll try to get that taken care of soon after
> I get back.
>
> Alan
>
>
> From: Littlewood, David John
> Sent: Wednesday, February 15, 2012 01:59 PM
> To: Williams, Alan B; 'turane at gmail.com' <turane at gmail.com>;
> 'eric.marttila at thermoanalytics.com' <eric.marttila at thermoanalytics.com>
> Cc: 'trilinos-users at software.sandia.gov'
> <trilinos-users at software.sandia.gov> Subject: Re: [Trilinos-Users]
> [EXTERNAL] Re: Problem inserting non-local data in Epetra_FECrsMatrix
>
> Folks,
>
> I ran into a similar-sounding issue a few months ago. I copy-and-pasted
> the e-mail snippet below in the hope that it might help you here…
>
> Dave
> --
> David Littlewood
> Multiphysics Simulation Technology, Dept. 1444
> 505-284-0830
>
>
> ***************************
>
> I've been working with Epetra_FECrsMatrix and ran into a peculiarity. In
> the base class Epetra_CrsMatrix, there are two functions called
> InsertGlobalValues:
>
> virtual int InsertGlobalValues (int GlobalRow, int NumEntries, const double
> *Values, const int *Indices)
>
> virtual int InsertGlobalValues (int GlobalRow, int NumEntries, double
> *Values, int *Indices)
>
> The only difference here is "const" on the last two arguments.
>
> It turns out that the first function is also implemented in the derived
> class Epetra_FECrsMatrix, but the second one is not. Thus if one calls
>
> Epetra_FECrsMatrix::InsertGlobalValues (int GlobalRow, int NumEntries,
> const double *Values, const int *Indices)
>
> you get the FE functionality (ability to insert nonlocal data), and if one
> calls
>
> Epetra_FECrsMatrix::InsertGlobalValues (int GlobalRow, int NumEntries,
> double *Values, int *Indices)
>
> you get the base class function, which does not handle nonlocal data.
>
> ***************************
>
>
> From: "Williams, Alan B" <william at sandia.gov<mailto:william at sandia.gov>>
> Date: Wed, 15 Feb 2012 18:55:13 +0000
> To: "'turane at gmail.com<mailto:'turane at gmail.com>'"
> <turane at gmail.com<mailto:turane at gmail.com>>,
> "'eric.marttila at thermoanalytics.com<mailto:'eric.marttila at thermoanalytics.
> com>'"
> <eric.marttila at thermoanalytics.com<mailto:eric.marttila at thermoanalytics.co
> m>> Cc:
> "'trilinos-users at software.sandia.gov<mailto:'trilinos-users at software.sandi
> a.gov>'"
> <trilinos-users at software.sandia.gov<mailto:trilinos-users at software.sandia.
> gov>> Subject: Re: [Trilinos-Users] [EXTERNAL] Re: Problem inserting
> non-local data in Epetra_FECrsMatrix
>
> Hello,
> The usage of insertGlobalValues that Eric reported trying, should work. I
> believe he has encountered a bug in the implementation of FECrsMatrix.
> I'll look into that.
>
> Alan
>
>
>
> From: erhan turan [mailto:turane at gmail.com]
> Sent: Wednesday, February 15, 2012 11:28 AM
> To: Eric Marttila
> <eric.marttila at thermoanalytics.com<mailto:eric.marttila at thermoanalytics.co
> m>> Cc:
> trilinos-users at software.sandia.gov<mailto:trilinos-users at software.sandia.g
> ov>
> <trilinos-users at software.sandia.gov<mailto:trilinos-users at software.sandia.
> gov>> Subject: [EXTERNAL] Re: [Trilinos-Users] Problem inserting non-local
> data in Epetra_FECrsMatrix
>
> Hello,
>
> I had a similar problem before. My colleague Cyril Flaig found a
> workaround. For non-local entries, you can try SumIntoGlobalValues rather
> than InsertGlobalValues. I mean, If A.LRID()== -1 , use
> SumIntoGlobalValues with global row index and use InsertGlobalValues
> otherwise. That might work.
>
> For that purpose, what is defined was two pointers say AF and A using
> Epetra_FECrsMatrix and Epetra_CrsMatrix, respectively. Here, we first
> declared AF using the map and later just saying A=AF. Later all methods
> are used for A: A->InsertGlobalValues() and A->SumIntoGlobalValues() .
> Fill is completed using AF->GlobalAssemble(). If you try,
> A->FillComplete() it wont fill non-local entries. That might be not the
> best solution. Maybe there is a simpler way. But it will probably work.
>
> Since you mentioned this issue, I want to make a reference to
> Epetra_FECrsMatrix documentation. There it reads:
>
> InsertGlobalValues()<http://trilinos.sandia.gov/packages/docs/r10.8/package
> s/epetra/doc/html/classEpetra__FECrsMatrix.html#a54820e91fb84c7c594c4e6f3ea
> 290fe4> inserts values into the matrix only if the graph has not yet been
> finalized
> (FillComplete()<http://trilinos.sandia.gov/packages/docs/r10.8/packages/ep
> etra/doc/html/classEpetra__CrsMatrix.html#a355b4e8d1e28b16bacf37ab637ac0c3f
> > has not yet been called). For non-local values, the call to
> InsertGlobalValues()<http://trilinos.sandia.gov/packages/docs/r10.8/packag
> es/epetra/doc/html/classEpetra__FECrsMatrix.html#a54820e91fb84c7c594c4e6f3e
> a290fe4> may succeed but
> theGlobalAssemble()<http://trilinos.sandia.gov/packages/docs/r10.8/package
> s/epetra/doc/html/classEpetra__FECrsMatrix.html#aca5191dcdfb99eb6316b86fee4
> 59b243> method may then fail because the non-local data is not actually
> inserted in the underlying matrix until
> GlobalAssemble()<http://trilinos.sandia.gov/packages/docs/r10.8/packages/e
> petra/doc/html/classEpetra__FECrsMatrix.html#aca5191dcdfb99eb6316b86fee459b
> 243> is called.
>
> This might be the reason that FillComplete() alone does not work but those
> overloaded SumInto methods over GlobalAssemble() work. However I still
> believe that this sentence may be vague :)
>
> Erhan Turan
> Department of Computer Science
> ETH Zurich
>
>
> On Wed, Feb 15, 2012 at 4:19 PM, Eric Marttila
> <eric.marttila at thermoanalytics.com<mailto:eric.marttila at thermoanalytics.co
> m>> wrote: Dear all,
>
> I've been using Epetra_CrsMatrix to store a linear system, but recently
> encountered a situation where I need to fill the entire matrix from the
> master process. From reading the Epetra documentation I believe I need to
> switch to a EpetraFECrs_Matrix to do this.
>
> I changed my code to use the EpetraFECrs_Matrix, but my call to
> InsertGlobalValues() fails on the first row of the matrix that is non-local
> to the master process.
>
> I've attached a small section of code that reproduces the behavior. It
> runs fine with 1 process, but fails when inserting row index 2 when run
> with 2 processors.
>
> I would appreciate any thoughts on what I'm missing with my usage of
> EpetraFECrs_Matrix.
>
> Thanks.
> --Eric
>
> p.s. I'm running Trilinos 10.8.5 on linux x86_64, using mpich2.
>
> --
> Eric A. Marttila
> ThermoAnalytics, Inc.
> 23440 Airpark Blvd.
> Calumet, MI 49913
>
> email:
> Eric.Marttila at ThermoAnalytics.com<mailto:Eric.Marttila at ThermoAnalytics.com
> > phone: 810-636-2443
> fax: 906-482-9755
> web: http://www.thermoanalytics.com
>
> _______________________________________________
> Trilinos-Users mailing list
> Trilinos-Users at software.sandia.gov<mailto:Trilinos-Users at software.sandia.go
> v> http://software.sandia.gov/mailman/listinfo/trilinos-users
>
>
> _______________________________________________ Trilinos-Users mailing list
> Trilinos-Users at software.sandia.gov<mailto:Trilinos-Users at software.sandia.g
> ov> http://software.sandia.gov/mailman/listinfo/trilinos-users
--
Eric A. Marttila
ThermoAnalytics, Inc.
23440 Airpark Blvd.
Calumet, MI 49913
email: Eric.Marttila at ThermoAnalytics.com
phone: 810-636-2443
fax: 906-482-9755
web: http://www.thermoanalytics.com
-------------- next part --------------
/*--------------------------------------------------------------------*/
#include <time.h>
#include "mpi.h"
#include "Epetra_FECrsMatrix.h"
#include "Epetra_Map.h"
#include "Epetra_MpiComm.h"
int main(int argCount, char **argValue)
{
int ierr;
MPI_Init(&argCount,&argValue);
Epetra_MpiComm Comm(MPI_COMM_WORLD);
const int rank = Comm.MyPID();
// Construct a Map
int nGlobalElements = 500000;
Epetra_Map Map(nGlobalElements, 0, Comm);
// Create a matrix
Epetra_FECrsMatrix A(Copy, Map, 1);
time_t startTime = 0;
if (rank == 0) {
startTime = time(0);
}
// Fill matrix on the master process
if (rank == 0) {
double values[1];
int indices[1];
const int numEntries = 1;
for (int globalRowIdx=0; globalRowIdx<nGlobalElements; ++globalRowIdx) {
indices[0] = globalRowIdx;
values[0] = 3.2 + globalRowIdx*0.01;
if (globalRowIdx % 5000 == 0) {
cerr << "About to insert row " << globalRowIdx << "\n";
}
ierr = A.InsertGlobalValues( globalRowIdx, numEntries,
(const double *)&values[0],
(const int *)&indices[0] );
assert(ierr==0);
}
}
double insertionTime = 0;
if (rank == 0) {
time_t endTime = time(0);
insertionTime = difftime(endTime, startTime);
}
// Finish up
ierr = A.GlobalAssemble();
assert(ierr==0);
if (rank == 0) {
cerr << "insertion time = " << insertionTime << " (seconds)\n";
}
MPI_Finalize();
return 0;
}
/*--------------------------------------------------------------------*/
More information about the Trilinos-Users
mailing list