[Trilinos-Users] [EXTERNAL] Re: Problem inserting non-local data in Epetra_FECrsMatrix

Eric Marttila eric.marttila at thermoanalytics.com
Tue Feb 21 09:41:12 MST 2012 

Hi Alan,

Thanks a lot for the quick response.  That all sounds very promising.
I was just looking at Epetra_LinearProblemRedistor.  Could that also be used 
to do the redistribution for my case?

--Eric


On Tuesday, February 21, 2012 11:21:17 am Williams, Alan B wrote:
> Hi Eric,
> 
> Yesterday I put a fix into the trilinos votd branch to add the over-ride of
> the missing InsertGlobalValues method in FECrsMatrix. I believe an update
> will be released soon that will include that fix.
> 
> For the performance question, the extreme slow-down you are seeing is not
> expected. Thanks for sending the test program, I will use that to
> reproduce the behavior and figure out what's going on. There is probably a
> silly hot-spot in there that I can easily fix. I'll let you know what I
> find out.
> 
> As for other ways of doing what you want to do, you could probably do it by
> creating a strictly local CrsMatrix and then using import/export
> functionality to redistribute it. If I improve the performance of
> FECrsMatrix, then it should be just as good as the alternative approach,
> and hopefully simpler to use.
> 
> Alan
> 
> > -----Original Message-----
> > From: Eric Marttila [mailto:eric.marttila at thermoanalytics.com]
> > Sent: Tuesday, February 21, 2012 9:05 AM
> > To: Williams, Alan B
> > Cc: Littlewood, David John; 'turane at gmail.com'; 'trilinos-
> > users at software.sandia.gov'; Heroux, Michael A
> > Subject: Re: [Trilinos-Users] [EXTERNAL] Re: Problem inserting non-local
> > data in Epetra_FECrsMatrix
> > 
> > All,
> > 
> > I've been using the workaround suggested earlier (making sure I call the
> > derived class version of InsertGlobalValues), and my code executes
> > correctly on multiple processors.  The problem I've seen is that the
> > insertions of non- local data are extremely slow.  I've attached a small
> > bit of code which runs with a matrix size of 500,000.  When run with 2
> > processors the total insertion time is about 4 minutes (most of which is
> > spent doing the non-local data). The insertion time seems to increase
> > exponentially with the matrix size; a matrix size of 1 million takes
> > about 20 minutes.  If the code is run with 1 processor it runs in less
> > than 1 second.
> > 
> > I saw the notes in the Epetra documentation that the GlobalAssemble() is
> > an expensive process, so I was expecting somewhat of a slowdown when
> > filling my
> > matrix from the master process, but wasn't expecting a slowdown of this
> > extent.  Also, in my case it seems that the expensive process is the
> > InsertGlobalValues.  (the time for GlobalAssemble is very small in
> > comparison).
> > 
> > I have two questions regarding this:
> > 1. Is the performance that I see (with non-local insertions) the expected
> > behavior for Epetra_FECrsMatrix ?
> > 
> > 2. Is there a different class that I should use instead of
> > Epetra_FECrsMatrix? I switched from Epetra_CrsMatrix to
> > Epetra_FECrsMatrix to get the non-local data functionality.  But in my
> > case the master process is filling the entire matrix, so there is no
> > chance that multiple processes would be setting values for a given
> > matrix element.  I'm wondering if FECrsMatrix is overkill for my
> > situation.
> > 
> > Thanks.
> > --Eric
> > 
> > On Wednesday, February 15, 2012 04:04:02 pm Williams, Alan B wrote:
> > > Thanks Dave,
> > > That may indeed be the issue. It is an oversight that only one of those
> > > base-class methods is over-ridden be FECrsMatrix.
> > > 
> > > I'm out of town this week but I'll try to get that taken care of soon
> > > after I get back.
> > > 
> > > Alan
> > > 
> > > 
> > > From: Littlewood, David John
> > > Sent: Wednesday, February 15, 2012 01:59 PM
> > > To: Williams, Alan B; 'turane at gmail.com' <turane at gmail.com>;
> > > 'eric.marttila at thermoanalytics.com' <eric.marttila at thermoanalytics.com>
> > > Cc: 'trilinos-users at software.sandia.gov'
> > > <trilinos-users at software.sandia.gov> Subject: Re: [Trilinos-Users]
> > > [EXTERNAL] Re: Problem inserting non-local data in Epetra_FECrsMatrix
> > > 
> > > Folks,
> > > 
> > > I ran into a similar-sounding issue a few months ago.  I
> > > copy-and-pasted the e-mail snippet below in the hope that it might
> > > help you here…
> > > 
> > > Dave
> > > --
> > > David Littlewood
> > > Multiphysics Simulation Technology, Dept. 1444
> > > 505-284-0830
> > > 
> > > 
> > > ***************************
> > > 
> > > I've been working with Epetra_FECrsMatrix and ran into a peculiarity. 
> > > In the base class Epetra_CrsMatrix, there are two functions called
> > > InsertGlobalValues:
> > > 
> > > virtual int InsertGlobalValues (int GlobalRow, int NumEntries, const
> > > double *Values, const int *Indices)
> > > 
> > > virtual int InsertGlobalValues (int GlobalRow, int NumEntries, double
> > > *Values, int *Indices)
> > > 
> > > The only difference here is "const" on the last two arguments.
> > > 
> > > It turns out that the first function is also implemented in the derived
> > > class Epetra_FECrsMatrix, but the second one is not.  Thus if one calls
> > > 
> > > Epetra_FECrsMatrix::InsertGlobalValues (int GlobalRow, int NumEntries,
> > > const double *Values, const int *Indices)
> > > 
> > > you get the FE functionality (ability to insert nonlocal data), and if
> > > one calls
> > > 
> > > Epetra_FECrsMatrix::InsertGlobalValues (int GlobalRow, int NumEntries,
> > > double *Values, int *Indices)
> > > 
> > > you get the base class function, which does not handle nonlocal data.
> > > 
> > > ***************************
> > > 
> > > 
> > > From: "Williams, Alan B"
> > 
> > <william at sandia.gov<mailto:william at sandia.gov>>
> > 
> > > Date: Wed, 15 Feb 2012 18:55:13 +0000
> > > To: "'turane at gmail.com<mailto:'turane at gmail.com>'"
> > > <turane at gmail.com<mailto:turane at gmail.com>>,
> > 
> > "'eric.marttila at thermoanalytics.com<mailto:'eric.marttila at thermoanalytics
> > .
> > 
> > > com>'"
> > 
> > <eric.marttila at thermoanalytics.com<mailto:eric.marttila at thermoanalytics.c
> > o
> > 
> > > m>> Cc:
> > > "'trilinos-users at software.sandia.gov<mailto:'trilinos-
> > 
> > users at software.sandi
> > 
> > > a.gov>'"
> > > <trilinos-users at software.sandia.gov<mailto:trilinos-
> > 
> > users at software.sandia.
> > 
> > > gov>> Subject: Re: [Trilinos-Users] [EXTERNAL] Re: Problem inserting
> > > non-local data in Epetra_FECrsMatrix
> > > 
> > > Hello,
> > > The usage of insertGlobalValues that Eric reported trying, should work.
> > > I believe he has encountered a bug in the implementation of
> > > FECrsMatrix. I'll look into that.
> > > 
> > > Alan
> > > 
> > > 
> > > 
> > > From: erhan turan [mailto:turane at gmail.com]
> > > Sent: Wednesday, February 15, 2012 11:28 AM
> > > To: Eric Marttila
> > 
> > <eric.marttila at thermoanalytics.com<mailto:eric.marttila at thermoanalytics.c
> > o
> > 
> > > m>> Cc:
> > > trilinos-users at software.sandia.gov<mailto:trilinos-
> > 
> > users at software.sandia.g
> > 
> > > ov>
> > > <trilinos-users at software.sandia.gov<mailto:trilinos-
> > 
> > users at software.sandia.
> > 
> > > gov>> Subject: [EXTERNAL] Re: [Trilinos-Users] Problem inserting
> > > non-local data in Epetra_FECrsMatrix
> > > 
> > > Hello,
> > > 
> > > I had a similar problem before. My colleague Cyril Flaig found a
> > > workaround. For non-local entries, you can try SumIntoGlobalValues
> > > rather than InsertGlobalValues. I mean, If A.LRID()== -1 , use
> > > SumIntoGlobalValues with global row index and use InsertGlobalValues
> > > otherwise. That might work.
> > > 
> > > For that purpose, what is defined was two pointers say AF and A using
> > > Epetra_FECrsMatrix and Epetra_CrsMatrix, respectively. Here, we first
> > > declared AF using the map and later just saying A=AF. Later all methods
> > > are used for A: A->InsertGlobalValues() and A->SumIntoGlobalValues() .
> > > Fill is completed using AF->GlobalAssemble(). If you try,
> > > A->FillComplete() it wont fill non-local entries. That might be not the
> > > best solution. Maybe there is a simpler way. But it will probably work.
> > > 
> > > Since you mentioned this issue, I want to make a reference to
> > 
> > > Epetra_FECrsMatrix documentation. There it reads:
> > InsertGlobalValues()<http://trilinos.sandia.gov/packages/docs/r10.8/packa
> > g e
> > 
> > s/epetra/doc/html/classEpetra__FECrsMatrix.html#a54820e91fb84c7c594c4e
> > 6f3ea
> > 
> > > 290fe4> inserts values into the matrix only if the graph has not yet
> > > been finalized
> > 
> > (FillComplete()<http://trilinos.sandia.gov/packages/docs/r10.8/packages/e
> > p
> > 
> > etra/doc/html/classEpetra__CrsMatrix.html#a355b4e8d1e28b16bacf37ab637
> > ac0c3f
> > 
> > > > has not yet been called). For non-local values, the call to
> > 
> > InsertGlobalValues()<http://trilinos.sandia.gov/packages/docs/r10.8/packa
> > g
> > 
> > es/epetra/doc/html/classEpetra__FECrsMatrix.html#a54820e91fb84c7c594c4
> > e6f3e
> > 
> > > a290fe4> may succeed but
> > 
> > theGlobalAssemble()<http://trilinos.sandia.gov/packages/docs/r10.8/packag
> > e
> > 
> > s/epetra/doc/html/classEpetra__FECrsMatrix.html#aca5191dcdfb99eb6316b
> > 86fee4
> > 
> > > 59b243> method may then fail because the non-local data is not actually
> > > inserted in the underlying matrix until
> > 
> > GlobalAssemble()<http://trilinos.sandia.gov/packages/docs/r10.8/packages/
> > e
> > 
> > petra/doc/html/classEpetra__FECrsMatrix.html#aca5191dcdfb99eb6316b86f
> > ee459b
> > 
> > > 243> is called.
> > > 
> > > This might be the reason that FillComplete() alone does not work but
> > > those overloaded SumInto methods over GlobalAssemble() work. However I
> > > still believe that this sentence may be vague :)
> > > 
> > > Erhan Turan
> > > Department of Computer Science
> > > ETH Zurich
> > > 
> > > 
> > > On Wed, Feb 15, 2012 at 4:19 PM, Eric Marttila
> > 
> > <eric.marttila at thermoanalytics.com<mailto:eric.marttila at thermoanalytics.c
> > o
> > 
> > > m>> wrote: Dear all,
> > > 
> > > I've been using Epetra_CrsMatrix to store a linear system, but recently
> > > encountered a situation where I need to fill the entire matrix from the
> > > master process.  From reading the Epetra documentation I believe I need
> > 
> > to
> > 
> > > switch to a EpetraFECrs_Matrix to do this.
> > > 
> > > I changed my code to use the EpetraFECrs_Matrix, but my call to
> > > InsertGlobalValues() fails on the first row of the matrix that is
> > > non-local to the master process.
> > > 
> > > I've attached a small section of code that reproduces the behavior.  It
> > > runs fine with 1 process, but fails when inserting row index 2 when run
> > > with 2 processors.
> > > 
> > > I would appreciate any thoughts on what I'm missing with my usage of
> > > EpetraFECrs_Matrix.
> > > 
> > > Thanks.
> > > --Eric
> > > 
> > > p.s. I'm running Trilinos 10.8.5 on linux x86_64, using mpich2.
> > > 
> > > --
> > > Eric A. Marttila
> > > ThermoAnalytics, Inc.
> > > 23440 Airpark Blvd.
> > > Calumet, MI 49913
> > 
> > > email:
> > Eric.Marttila at ThermoAnalytics.com<mailto:Eric.Marttila at ThermoAnalytics.c
> > om
> > 
> > > > phone: 810-636-2443
> > > 
> > > fax:   906-482-9755
> > > web: http://www.thermoanalytics.com
> > > 
> > > _______________________________________________
> > > Trilinos-Users mailing list
> > > Trilinos-Users at software.sandia.gov<mailto:Trilinos-
> > 
> > Users at software.sandia.go
> > 
> > > v> http://software.sandia.gov/mailman/listinfo/trilinos-users
> > > 
> > > 
> > > _______________________________________________ Trilinos-
> > 
> > Users mailing list
> > 
> > > Trilinos-Users at software.sandia.gov<mailto:Trilinos-
> > 
> > Users at software.sandia.g
> > 
> > > ov> http://software.sandia.gov/mailman/listinfo/trilinos-users
> > 
> > --
> > Eric A. Marttila
> > ThermoAnalytics, Inc.
> > 23440 Airpark Blvd.
> > Calumet, MI 49913
> > 
> > email: Eric.Marttila at ThermoAnalytics.com
> > phone: 810-636-2443
> > fax:   906-482-9755
> > web: http://www.thermoanalytics.com

-- 
Eric A. Marttila
ThermoAnalytics, Inc.
23440 Airpark Blvd.
Calumet, MI 49913

email: Eric.Marttila at ThermoAnalytics.com
phone: 810-636-2443
fax:   906-482-9755
web: http://www.thermoanalytics.com

More information about the Trilinos-Users mailing list