[Trilinos-Users] [EXTERNAL] Re: Problem inserting non-local data in Epetra_FECrsMatrix

Williams, Alan B william at sandia.gov
Tue Feb 21 11:47:21 MST 2012 

Eric,

I'm not very familiar with Epetra_LinearProblemRedistor.

I ran your test program and profiled it, and then replaced some data-structures and insert operations in FECrsMatrix with better STL-based alternatives. The insertions and associated data-sliding for non-local data was really killing it.

Now, setting the number of elements to 1 million and running it on 2 procs, it completes in 3 or 4 seconds (on my linux workstation).

I'll check these changes into the votd branch today. If you don't object, I'll keep your test program in the test directory for future reference.

Thanks,
Alan


> -----Original Message-----
> From: Eric Marttila [mailto:eric.marttila at thermoanalytics.com]
> Sent: Tuesday, February 21, 2012 9:41 AM
> To: Williams, Alan B
> Cc: Littlewood, David John; 'turane at gmail.com'; 'trilinos-
> users at software.sandia.gov'; Heroux, Michael A
> Subject: Re: [Trilinos-Users] [EXTERNAL] Re: Problem inserting non-local data
> in Epetra_FECrsMatrix
> 
> Hi Alan,
> 
> Thanks a lot for the quick response.  That all sounds very promising.
> I was just looking at Epetra_LinearProblemRedistor.  Could that also be used
> to do the redistribution for my case?
> 
> --Eric
> 
> 
> On Tuesday, February 21, 2012 11:21:17 am Williams, Alan B wrote:
> > Hi Eric,
> >
> > Yesterday I put a fix into the trilinos votd branch to add the over-ride of
> > the missing InsertGlobalValues method in FECrsMatrix. I believe an update
> > will be released soon that will include that fix.
> >
> > For the performance question, the extreme slow-down you are seeing is
> not
> > expected. Thanks for sending the test program, I will use that to
> > reproduce the behavior and figure out what's going on. There is probably a
> > silly hot-spot in there that I can easily fix. I'll let you know what I
> > find out.
> >
> > As for other ways of doing what you want to do, you could probably do it
> by
> > creating a strictly local CrsMatrix and then using import/export
> > functionality to redistribute it. If I improve the performance of
> > FECrsMatrix, then it should be just as good as the alternative approach,
> > and hopefully simpler to use.
> >
> > Alan
> >
> > > -----Original Message-----
> > > From: Eric Marttila [mailto:eric.marttila at thermoanalytics.com]
> > > Sent: Tuesday, February 21, 2012 9:05 AM
> > > To: Williams, Alan B
> > > Cc: Littlewood, David John; 'turane at gmail.com'; 'trilinos-
> > > users at software.sandia.gov'; Heroux, Michael A
> > > Subject: Re: [Trilinos-Users] [EXTERNAL] Re: Problem inserting non-local
> > > data in Epetra_FECrsMatrix
> > >
> > > All,
> > >
> > > I've been using the workaround suggested earlier (making sure I call the
> > > derived class version of InsertGlobalValues), and my code executes
> > > correctly on multiple processors.  The problem I've seen is that the
> > > insertions of non- local data are extremely slow.  I've attached a small
> > > bit of code which runs with a matrix size of 500,000.  When run with 2
> > > processors the total insertion time is about 4 minutes (most of which is
> > > spent doing the non-local data). The insertion time seems to increase
> > > exponentially with the matrix size; a matrix size of 1 million takes
> > > about 20 minutes.  If the code is run with 1 processor it runs in less
> > > than 1 second.
> > >
> > > I saw the notes in the Epetra documentation that the GlobalAssemble() is
> > > an expensive process, so I was expecting somewhat of a slowdown when
> > > filling my
> > > matrix from the master process, but wasn't expecting a slowdown of this
> > > extent.  Also, in my case it seems that the expensive process is the
> > > InsertGlobalValues.  (the time for GlobalAssemble is very small in
> > > comparison).
> > >
> > > I have two questions regarding this:
> > > 1. Is the performance that I see (with non-local insertions) the expected
> > > behavior for Epetra_FECrsMatrix ?
> > >
> > > 2. Is there a different class that I should use instead of
> > > Epetra_FECrsMatrix? I switched from Epetra_CrsMatrix to
> > > Epetra_FECrsMatrix to get the non-local data functionality.  But in my
> > > case the master process is filling the entire matrix, so there is no
> > > chance that multiple processes would be setting values for a given
> > > matrix element.  I'm wondering if FECrsMatrix is overkill for my
> > > situation.
> > >
> > > Thanks.
> > > --Eric
> > >
> > > On Wednesday, February 15, 2012 04:04:02 pm Williams, Alan B wrote:
> > > > Thanks Dave,
> > > > That may indeed be the issue. It is an oversight that only one of those
> > > > base-class methods is over-ridden be FECrsMatrix.
> > > >
> > > > I'm out of town this week but I'll try to get that taken care of soon
> > > > after I get back.
> > > >
> > > > Alan
> > > >
> > > >
> > > > From: Littlewood, David John
> > > > Sent: Wednesday, February 15, 2012 01:59 PM
> > > > To: Williams, Alan B; 'turane at gmail.com' <turane at gmail.com>;
> > > > 'eric.marttila at thermoanalytics.com'
> <eric.marttila at thermoanalytics.com>
> > > > Cc: 'trilinos-users at software.sandia.gov'
> > > > <trilinos-users at software.sandia.gov> Subject: Re: [Trilinos-Users]
> > > > [EXTERNAL] Re: Problem inserting non-local data in Epetra_FECrsMatrix
> > > >
> > > > Folks,
> > > >
> > > > I ran into a similar-sounding issue a few months ago.  I
> > > > copy-and-pasted the e-mail snippet below in the hope that it might
> > > > help you here…
> > > >
> > > > Dave
> > > > --
> > > > David Littlewood
> > > > Multiphysics Simulation Technology, Dept. 1444
> > > > 505-284-0830
> > > >
> > > >
> > > > ***************************
> > > >
> > > > I've been working with Epetra_FECrsMatrix and ran into a peculiarity.
> > > > In the base class Epetra_CrsMatrix, there are two functions called
> > > > InsertGlobalValues:
> > > >
> > > > virtual int InsertGlobalValues (int GlobalRow, int NumEntries, const
> > > > double *Values, const int *Indices)
> > > >
> > > > virtual int InsertGlobalValues (int GlobalRow, int NumEntries, double
> > > > *Values, int *Indices)
> > > >
> > > > The only difference here is "const" on the last two arguments.
> > > >
> > > > It turns out that the first function is also implemented in the derived
> > > > class Epetra_FECrsMatrix, but the second one is not.  Thus if one calls
> > > >
> > > > Epetra_FECrsMatrix::InsertGlobalValues (int GlobalRow, int
> NumEntries,
> > > > const double *Values, const int *Indices)
> > > >
> > > > you get the FE functionality (ability to insert nonlocal data), and if
> > > > one calls
> > > >
> > > > Epetra_FECrsMatrix::InsertGlobalValues (int GlobalRow, int
> NumEntries,
> > > > double *Values, int *Indices)
> > > >
> > > > you get the base class function, which does not handle nonlocal data.
> > > >
> > > > ***************************
> > > >
> > > >
> > > > From: "Williams, Alan B"
> > >
> > > <william at sandia.gov<mailto:william at sandia.gov>>
> > >
> > > > Date: Wed, 15 Feb 2012 18:55:13 +0000
> > > > To: "'turane at gmail.com<mailto:'turane at gmail.com>'"
> > > > <turane at gmail.com<mailto:turane at gmail.com>>,
> > >
> > >
> "'eric.marttila at thermoanalytics.com<mailto:'eric.marttila at thermoanalytics
> > > .
> > >
> > > > com>'"
> > >
> > >
> <eric.marttila at thermoanalytics.com<mailto:eric.marttila at thermoanalytics.c
> > > o
> > >
> > > > m>> Cc:
> > > > "'trilinos-users at software.sandia.gov<mailto:'trilinos-
> > >
> > > users at software.sandi
> > >
> > > > a.gov>'"
> > > > <trilinos-users at software.sandia.gov<mailto:trilinos-
> > >
> > > users at software.sandia.
> > >
> > > > gov>> Subject: Re: [Trilinos-Users] [EXTERNAL] Re: Problem inserting
> > > > non-local data in Epetra_FECrsMatrix
> > > >
> > > > Hello,
> > > > The usage of insertGlobalValues that Eric reported trying, should work.
> > > > I believe he has encountered a bug in the implementation of
> > > > FECrsMatrix. I'll look into that.
> > > >
> > > > Alan
> > > >
> > > >
> > > >
> > > > From: erhan turan [mailto:turane at gmail.com]
> > > > Sent: Wednesday, February 15, 2012 11:28 AM
> > > > To: Eric Marttila
> > >
> > >
> <eric.marttila at thermoanalytics.com<mailto:eric.marttila at thermoanalytics.c
> > > o
> > >
> > > > m>> Cc:
> > > > trilinos-users at software.sandia.gov<mailto:trilinos-
> > >
> > > users at software.sandia.g
> > >
> > > > ov>
> > > > <trilinos-users at software.sandia.gov<mailto:trilinos-
> > >
> > > users at software.sandia.
> > >
> > > > gov>> Subject: [EXTERNAL] Re: [Trilinos-Users] Problem inserting
> > > > non-local data in Epetra_FECrsMatrix
> > > >
> > > > Hello,
> > > >
> > > > I had a similar problem before. My colleague Cyril Flaig found a
> > > > workaround. For non-local entries, you can try SumIntoGlobalValues
> > > > rather than InsertGlobalValues. I mean, If A.LRID()== -1 , use
> > > > SumIntoGlobalValues with global row index and use InsertGlobalValues
> > > > otherwise. That might work.
> > > >
> > > > For that purpose, what is defined was two pointers say AF and A using
> > > > Epetra_FECrsMatrix and Epetra_CrsMatrix, respectively. Here, we first
> > > > declared AF using the map and later just saying A=AF. Later all methods
> > > > are used for A: A->InsertGlobalValues() and A->SumIntoGlobalValues()
> .
> > > > Fill is completed using AF->GlobalAssemble(). If you try,
> > > > A->FillComplete() it wont fill non-local entries. That might be not the
> > > > best solution. Maybe there is a simpler way. But it will probably work.
> > > >
> > > > Since you mentioned this issue, I want to make a reference to
> > >
> > > > Epetra_FECrsMatrix documentation. There it reads:
> > >
> InsertGlobalValues()<http://trilinos.sandia.gov/packages/docs/r10.8/packa
> > > g e
> > >
> > >
> s/epetra/doc/html/classEpetra__FECrsMatrix.html#a54820e91fb84c7c594c4e
> > > 6f3ea
> > >
> > > > 290fe4> inserts values into the matrix only if the graph has not yet
> > > > been finalized
> > >
> > >
> (FillComplete()<http://trilinos.sandia.gov/packages/docs/r10.8/packages/e
> > > p
> > >
> > >
> etra/doc/html/classEpetra__CrsMatrix.html#a355b4e8d1e28b16bacf37ab637
> > > ac0c3f
> > >
> > > > > has not yet been called). For non-local values, the call to
> > >
> > >
> InsertGlobalValues()<http://trilinos.sandia.gov/packages/docs/r10.8/packa
> > > g
> > >
> > >
> es/epetra/doc/html/classEpetra__FECrsMatrix.html#a54820e91fb84c7c594c4
> > > e6f3e
> > >
> > > > a290fe4> may succeed but
> > >
> > >
> theGlobalAssemble()<http://trilinos.sandia.gov/packages/docs/r10.8/packag
> > > e
> > >
> > >
> s/epetra/doc/html/classEpetra__FECrsMatrix.html#aca5191dcdfb99eb6316b
> > > 86fee4
> > >
> > > > 59b243> method may then fail because the non-local data is not actually
> > > > inserted in the underlying matrix until
> > >
> > >
> GlobalAssemble()<http://trilinos.sandia.gov/packages/docs/r10.8/packages/
> > > e
> > >
> > >
> petra/doc/html/classEpetra__FECrsMatrix.html#aca5191dcdfb99eb6316b86f
> > > ee459b
> > >
> > > > 243> is called.
> > > >
> > > > This might be the reason that FillComplete() alone does not work but
> > > > those overloaded SumInto methods over GlobalAssemble() work.
> However I
> > > > still believe that this sentence may be vague :)
> > > >
> > > > Erhan Turan
> > > > Department of Computer Science
> > > > ETH Zurich
> > > >
> > > >
> > > > On Wed, Feb 15, 2012 at 4:19 PM, Eric Marttila
> > >
> > >
> <eric.marttila at thermoanalytics.com<mailto:eric.marttila at thermoanalytics.c
> > > o
> > >
> > > > m>> wrote: Dear all,
> > > >
> > > > I've been using Epetra_CrsMatrix to store a linear system, but recently
> > > > encountered a situation where I need to fill the entire matrix from the
> > > > master process.  From reading the Epetra documentation I believe I
> need
> > >
> > > to
> > >
> > > > switch to a EpetraFECrs_Matrix to do this.
> > > >
> > > > I changed my code to use the EpetraFECrs_Matrix, but my call to
> > > > InsertGlobalValues() fails on the first row of the matrix that is
> > > > non-local to the master process.
> > > >
> > > > I've attached a small section of code that reproduces the behavior.  It
> > > > runs fine with 1 process, but fails when inserting row index 2 when run
> > > > with 2 processors.
> > > >
> > > > I would appreciate any thoughts on what I'm missing with my usage of
> > > > EpetraFECrs_Matrix.
> > > >
> > > > Thanks.
> > > > --Eric
> > > >
> > > > p.s. I'm running Trilinos 10.8.5 on linux x86_64, using mpich2.
> > > >
> > > > --
> > > > Eric A. Marttila
> > > > ThermoAnalytics, Inc.
> > > > 23440 Airpark Blvd.
> > > > Calumet, MI 49913
> > >
> > > > email:
> > >
> Eric.Marttila at ThermoAnalytics.com<mailto:Eric.Marttila at ThermoAnalytics.c
> > > om
> > >
> > > > > phone: 810-636-2443
> > > >
> > > > fax:   906-482-9755
> > > > web: http://www.thermoanalytics.com
> > > >
> > > > _______________________________________________
> > > > Trilinos-Users mailing list
> > > > Trilinos-Users at software.sandia.gov<mailto:Trilinos-
> > >
> > > Users at software.sandia.go
> > >
> > > > v> http://software.sandia.gov/mailman/listinfo/trilinos-users
> > > >
> > > >
> > > > _______________________________________________ Trilinos-
> > >
> > > Users mailing list
> > >
> > > > Trilinos-Users at software.sandia.gov<mailto:Trilinos-
> > >
> > > Users at software.sandia.g
> > >
> > > > ov> http://software.sandia.gov/mailman/listinfo/trilinos-users
> > >
> > > --
> > > Eric A. Marttila
> > > ThermoAnalytics, Inc.
> > > 23440 Airpark Blvd.
> > > Calumet, MI 49913
> > >
> > > email: Eric.Marttila at ThermoAnalytics.com
> > > phone: 810-636-2443
> > > fax:   906-482-9755
> > > web: http://www.thermoanalytics.com
> 
> --
> Eric A. Marttila
> ThermoAnalytics, Inc.
> 23440 Airpark Blvd.
> Calumet, MI 49913
> 
> email: Eric.Marttila at ThermoAnalytics.com
> phone: 810-636-2443
> fax:   906-482-9755
> web: http://www.thermoanalytics.com

More information about the Trilinos-Users mailing list