[Trilinos-Users] [EXTERNAL] Re: Problem inserting non-local data in Epetra_FECrsMatrix

Eric Marttila eric.marttila at thermoanalytics.com
Tue Feb 21 12:01:59 MST 2012 

Alan,
Glad to hear you have fixed the performance problem so quickly.
I have no problem with you keeping my test program for future use.
Thanks for all  your help.
--Eric

On Tuesday, February 21, 2012 01:47:21 pm Williams, Alan B wrote:
> Eric,
> 
> I'm not very familiar with Epetra_LinearProblemRedistor.
> 
> I ran your test program and profiled it, and then replaced some
> data-structures and insert operations in FECrsMatrix with better STL-based
> alternatives. The insertions and associated data-sliding for non-local
> data was really killing it.
> 
> Now, setting the number of elements to 1 million and running it on 2 procs,
> it completes in 3 or 4 seconds (on my linux workstation).
> 
> I'll check these changes into the votd branch today. If you don't object,
> I'll keep your test program in the test directory for future reference.
> 
> Thanks,
> Alan
> 
> > -----Original Message-----
> > From: Eric Marttila [mailto:eric.marttila at thermoanalytics.com]
> > Sent: Tuesday, February 21, 2012 9:41 AM
> > To: Williams, Alan B
> > Cc: Littlewood, David John; 'turane at gmail.com'; 'trilinos-
> > users at software.sandia.gov'; Heroux, Michael A
> > Subject: Re: [Trilinos-Users] [EXTERNAL] Re: Problem inserting non-local
> > data in Epetra_FECrsMatrix
> > 
> > Hi Alan,
> > 
> > Thanks a lot for the quick response.  That all sounds very promising.
> > I was just looking at Epetra_LinearProblemRedistor.  Could that also be
> > used to do the redistribution for my case?
> > 
> > --Eric
> > 
> > On Tuesday, February 21, 2012 11:21:17 am Williams, Alan B wrote:
> > > Hi Eric,
> > > 
> > > Yesterday I put a fix into the trilinos votd branch to add the
> > > over-ride of the missing InsertGlobalValues method in FECrsMatrix. I
> > > believe an update will be released soon that will include that fix.
> > > 
> > > For the performance question, the extreme slow-down you are seeing is
> > 
> > not
> > 
> > > expected. Thanks for sending the test program, I will use that to
> > > reproduce the behavior and figure out what's going on. There is
> > > probably a silly hot-spot in there that I can easily fix. I'll let you
> > > know what I find out.
> > > 
> > > As for other ways of doing what you want to do, you could probably do
> > > it
> > 
> > by
> > 
> > > creating a strictly local CrsMatrix and then using import/export
> > > functionality to redistribute it. If I improve the performance of
> > > FECrsMatrix, then it should be just as good as the alternative
> > > approach, and hopefully simpler to use.
> > > 
> > > Alan
> > > 
> > > > -----Original Message-----
> > > > From: Eric Marttila [mailto:eric.marttila at thermoanalytics.com]
> > > > Sent: Tuesday, February 21, 2012 9:05 AM
> > > > To: Williams, Alan B
> > > > Cc: Littlewood, David John; 'turane at gmail.com'; 'trilinos-
> > > > users at software.sandia.gov'; Heroux, Michael A
> > > > Subject: Re: [Trilinos-Users] [EXTERNAL] Re: Problem inserting
> > > > non-local data in Epetra_FECrsMatrix
> > > > 
> > > > All,
> > > > 
> > > > I've been using the workaround suggested earlier (making sure I call
> > > > the derived class version of InsertGlobalValues), and my code
> > > > executes correctly on multiple processors.  The problem I've seen is
> > > > that the insertions of non- local data are extremely slow.  I've
> > > > attached a small bit of code which runs with a matrix size of
> > > > 500,000.  When run with 2 processors the total insertion time is
> > > > about 4 minutes (most of which is spent doing the non-local data).
> > > > The insertion time seems to increase exponentially with the matrix
> > > > size; a matrix size of 1 million takes about 20 minutes.  If the
> > > > code is run with 1 processor it runs in less than 1 second.
> > > > 
> > > > I saw the notes in the Epetra documentation that the GlobalAssemble()
> > > > is an expensive process, so I was expecting somewhat of a slowdown
> > > > when filling my
> > > > matrix from the master process, but wasn't expecting a slowdown of
> > > > this extent.  Also, in my case it seems that the expensive process
> > > > is the InsertGlobalValues.  (the time for GlobalAssemble is very
> > > > small in comparison).
> > > > 
> > > > I have two questions regarding this:
> > > > 1. Is the performance that I see (with non-local insertions) the
> > > > expected behavior for Epetra_FECrsMatrix ?
> > > > 
> > > > 2. Is there a different class that I should use instead of
> > > > Epetra_FECrsMatrix? I switched from Epetra_CrsMatrix to
> > > > Epetra_FECrsMatrix to get the non-local data functionality.  But in
> > > > my case the master process is filling the entire matrix, so there is
> > > > no chance that multiple processes would be setting values for a
> > > > given matrix element.  I'm wondering if FECrsMatrix is overkill for
> > > > my situation.
> > > > 
> > > > Thanks.
> > > > --Eric
> > > > 
> > > > On Wednesday, February 15, 2012 04:04:02 pm Williams, Alan B wrote:
> > > > > Thanks Dave,
> > > > > That may indeed be the issue. It is an oversight that only one of
> > > > > those base-class methods is over-ridden be FECrsMatrix.
> > > > > 
> > > > > I'm out of town this week but I'll try to get that taken care of
> > > > > soon after I get back.
> > > > > 
> > > > > Alan
> > > > > 
> > > > > 
> > > > > From: Littlewood, David John
> > > > > Sent: Wednesday, February 15, 2012 01:59 PM
> > > > > To: Williams, Alan B; 'turane at gmail.com' <turane at gmail.com>;
> > > > > 'eric.marttila at thermoanalytics.com'
> > 
> > <eric.marttila at thermoanalytics.com>
> > 
> > > > > Cc: 'trilinos-users at software.sandia.gov'
> > > > > <trilinos-users at software.sandia.gov> Subject: Re: [Trilinos-Users]
> > > > > [EXTERNAL] Re: Problem inserting non-local data in
> > > > > Epetra_FECrsMatrix
> > > > > 
> > > > > Folks,
> > > > > 
> > > > > I ran into a similar-sounding issue a few months ago.  I
> > > > > copy-and-pasted the e-mail snippet below in the hope that it might
> > > > > help you here…
> > > > > 
> > > > > Dave
> > > > > --
> > > > > David Littlewood
> > > > > Multiphysics Simulation Technology, Dept. 1444
> > > > > 505-284-0830
> > > > > 
> > > > > 
> > > > > ***************************
> > > > > 
> > > > > I've been working with Epetra_FECrsMatrix and ran into a
> > > > > peculiarity. In the base class Epetra_CrsMatrix, there are two
> > > > > functions called InsertGlobalValues:
> > > > > 
> > > > > virtual int InsertGlobalValues (int GlobalRow, int NumEntries,
> > > > > const double *Values, const int *Indices)
> > > > > 
> > > > > virtual int InsertGlobalValues (int GlobalRow, int NumEntries,
> > > > > double *Values, int *Indices)
> > > > > 
> > > > > The only difference here is "const" on the last two arguments.
> > > > > 
> > > > > It turns out that the first function is also implemented in the
> > > > > derived class Epetra_FECrsMatrix, but the second one is not.  Thus
> > > > > if one calls
> > > > > 
> > > > > Epetra_FECrsMatrix::InsertGlobalValues (int GlobalRow, int
> > 
> > NumEntries,
> > 
> > > > > const double *Values, const int *Indices)
> > > > > 
> > > > > you get the FE functionality (ability to insert nonlocal data), and
> > > > > if one calls
> > > > > 
> > > > > Epetra_FECrsMatrix::InsertGlobalValues (int GlobalRow, int
> > 
> > NumEntries,
> > 
> > > > > double *Values, int *Indices)
> > > > > 
> > > > > you get the base class function, which does not handle nonlocal
> > > > > data.
> > > > > 
> > > > > ***************************
> > > > > 
> > > > > 
> > > > > From: "Williams, Alan B"
> > > > 
> > > > <william at sandia.gov<mailto:william at sandia.gov>>
> > > > 
> > > > > Date: Wed, 15 Feb 2012 18:55:13 +0000
> > > > > To: "'turane at gmail.com<mailto:'turane at gmail.com>'"
> > > > > <turane at gmail.com<mailto:turane at gmail.com>>,
> > 
> > "'eric.marttila at thermoanalytics.com<mailto:'eric.marttila at thermoanalytics
> > 
> > > > .
> > > > 
> > > > > com>'"
> > 
> > <eric.marttila at thermoanalytics.com<mailto:eric.marttila at thermoanalytics.c
> > 
> > > > o
> > > > 
> > > > > m>> Cc:
> > > > > "'trilinos-users at software.sandia.gov<mailto:'trilinos-
> > > > 
> > > > users at software.sandi
> > > > 
> > > > > a.gov>'"
> > > > > <trilinos-users at software.sandia.gov<mailto:trilinos-
> > > > 
> > > > users at software.sandia.
> > > > 
> > > > > gov>> Subject: Re: [Trilinos-Users] [EXTERNAL] Re: Problem
> > > > > inserting non-local data in Epetra_FECrsMatrix
> > > > > 
> > > > > Hello,
> > > > > The usage of insertGlobalValues that Eric reported trying, should
> > > > > work. I believe he has encountered a bug in the implementation of
> > > > > FECrsMatrix. I'll look into that.
> > > > > 
> > > > > Alan
> > > > > 
> > > > > 
> > > > > 
> > > > > From: erhan turan [mailto:turane at gmail.com]
> > > > > Sent: Wednesday, February 15, 2012 11:28 AM
> > > > > To: Eric Marttila
> > 
> > <eric.marttila at thermoanalytics.com<mailto:eric.marttila at thermoanalytics.c
> > 
> > > > o
> > > > 
> > > > > m>> Cc:
> > > > > trilinos-users at software.sandia.gov<mailto:trilinos-
> > > > 
> > > > users at software.sandia.g
> > > > 
> > > > > ov>
> > > > > <trilinos-users at software.sandia.gov<mailto:trilinos-
> > > > 
> > > > users at software.sandia.
> > > > 
> > > > > gov>> Subject: [EXTERNAL] Re: [Trilinos-Users] Problem inserting
> > > > > non-local data in Epetra_FECrsMatrix
> > > > > 
> > > > > Hello,
> > > > > 
> > > > > I had a similar problem before. My colleague Cyril Flaig found a
> > > > > workaround. For non-local entries, you can try SumIntoGlobalValues
> > > > > rather than InsertGlobalValues. I mean, If A.LRID()== -1 , use
> > > > > SumIntoGlobalValues with global row index and use
> > > > > InsertGlobalValues otherwise. That might work.
> > > > > 
> > > > > For that purpose, what is defined was two pointers say AF and A
> > > > > using Epetra_FECrsMatrix and Epetra_CrsMatrix, respectively. Here,
> > > > > we first declared AF using the map and later just saying A=AF.
> > > > > Later all methods are used for A: A->InsertGlobalValues() and
> > > > > A->SumIntoGlobalValues()
> > 
> > .
> > 
> > > > > Fill is completed using AF->GlobalAssemble(). If you try,
> > > > > A->FillComplete() it wont fill non-local entries. That might be not
> > > > > the best solution. Maybe there is a simpler way. But it will
> > > > > probably work.
> > > > > 
> > > > > Since you mentioned this issue, I want to make a reference to
> > 
> > > > > Epetra_FECrsMatrix documentation. There it reads:
> > InsertGlobalValues()<http://trilinos.sandia.gov/packages/docs/r10.8/packa
> > 
> > > > g e
> > 
> > s/epetra/doc/html/classEpetra__FECrsMatrix.html#a54820e91fb84c7c594c4e
> > 
> > > > 6f3ea
> > > > 
> > > > > 290fe4> inserts values into the matrix only if the graph has not
> > > > > yet been finalized
> > 
> > (FillComplete()<http://trilinos.sandia.gov/packages/docs/r10.8/packages/e
> > 
> > > > p
> > 
> > etra/doc/html/classEpetra__CrsMatrix.html#a355b4e8d1e28b16bacf37ab637
> > 
> > > > ac0c3f
> > > > 
> > > > > > has not yet been called). For non-local values, the call to
> > 
> > InsertGlobalValues()<http://trilinos.sandia.gov/packages/docs/r10.8/packa
> > 
> > > > g
> > 
> > es/epetra/doc/html/classEpetra__FECrsMatrix.html#a54820e91fb84c7c594c4
> > 
> > > > e6f3e
> > > > 
> > > > > a290fe4> may succeed but
> > 
> > theGlobalAssemble()<http://trilinos.sandia.gov/packages/docs/r10.8/packag
> > 
> > > > e
> > 
> > s/epetra/doc/html/classEpetra__FECrsMatrix.html#aca5191dcdfb99eb6316b
> > 
> > > > 86fee4
> > > > 
> > > > > 59b243> method may then fail because the non-local data is not
> > > > > actually inserted in the underlying matrix until
> > 
> > GlobalAssemble()<http://trilinos.sandia.gov/packages/docs/r10.8/packages/
> > 
> > > > e
> > 
> > petra/doc/html/classEpetra__FECrsMatrix.html#aca5191dcdfb99eb6316b86f
> > 
> > > > ee459b
> > > > 
> > > > > 243> is called.
> > > > > 
> > > > > This might be the reason that FillComplete() alone does not work
> > > > > but those overloaded SumInto methods over GlobalAssemble() work.
> > 
> > However I
> > 
> > > > > still believe that this sentence may be vague :)
> > > > > 
> > > > > Erhan Turan
> > > > > Department of Computer Science
> > > > > ETH Zurich
> > > > > 
> > > > > 
> > > > > On Wed, Feb 15, 2012 at 4:19 PM, Eric Marttila
> > 
> > <eric.marttila at thermoanalytics.com<mailto:eric.marttila at thermoanalytics.c
> > 
> > > > o
> > > > 
> > > > > m>> wrote: Dear all,
> > > > > 
> > > > > I've been using Epetra_CrsMatrix to store a linear system, but
> > > > > recently encountered a situation where I need to fill the entire
> > > > > matrix from the master process.  From reading the Epetra
> > > > > documentation I believe I
> > 
> > need
> > 
> > > > to
> > > > 
> > > > > switch to a EpetraFECrs_Matrix to do this.
> > > > > 
> > > > > I changed my code to use the EpetraFECrs_Matrix, but my call to
> > > > > InsertGlobalValues() fails on the first row of the matrix that is
> > > > > non-local to the master process.
> > > > > 
> > > > > I've attached a small section of code that reproduces the behavior.
> > > > >  It runs fine with 1 process, but fails when inserting row index 2
> > > > > when run with 2 processors.
> > > > > 
> > > > > I would appreciate any thoughts on what I'm missing with my usage
> > > > > of EpetraFECrs_Matrix.
> > > > > 
> > > > > Thanks.
> > > > > --Eric
> > > > > 
> > > > > p.s. I'm running Trilinos 10.8.5 on linux x86_64, using mpich2.
> > > > > 
> > > > > --
> > > > > Eric A. Marttila
> > > > > ThermoAnalytics, Inc.
> > > > > 23440 Airpark Blvd.
> > > > > Calumet, MI 49913
> > 
> > > > > email:
> > Eric.Marttila at ThermoAnalytics.com<mailto:Eric.Marttila at ThermoAnalytics.c
> > 
> > > > om
> > > > 
> > > > > > phone: 810-636-2443
> > > > > 
> > > > > fax:   906-482-9755
> > > > > web: http://www.thermoanalytics.com
> > > > > 
> > > > > _______________________________________________
> > > > > Trilinos-Users mailing list
> > > > > Trilinos-Users at software.sandia.gov<mailto:Trilinos-
> > > > 
> > > > Users at software.sandia.go
> > > > 
> > > > > v> http://software.sandia.gov/mailman/listinfo/trilinos-users
> > > > > 
> > > > > 
> > > > > _______________________________________________ Trilinos-
> > > > 
> > > > Users mailing list
> > > > 
> > > > > Trilinos-Users at software.sandia.gov<mailto:Trilinos-
> > > > 
> > > > Users at software.sandia.g
> > > > 
> > > > > ov> http://software.sandia.gov/mailman/listinfo/trilinos-users
> > > > 
> > > > --
> > > > Eric A. Marttila
> > > > ThermoAnalytics, Inc.
> > > > 23440 Airpark Blvd.
> > > > Calumet, MI 49913
> > > > 
> > > > email: Eric.Marttila at ThermoAnalytics.com
> > > > phone: 810-636-2443
> > > > fax:   906-482-9755
> > > > web: http://www.thermoanalytics.com
> > 
> > --
> > Eric A. Marttila
> > ThermoAnalytics, Inc.
> > 23440 Airpark Blvd.
> > Calumet, MI 49913
> > 
> > email: Eric.Marttila at ThermoAnalytics.com
> > phone: 810-636-2443
> > fax:   906-482-9755
> > web: http://www.thermoanalytics.com

-- 
Eric A. Marttila
ThermoAnalytics, Inc.
23440 Airpark Blvd.
Calumet, MI 49913

email: Eric.Marttila at ThermoAnalytics.com
phone: 810-636-2443
fax:   906-482-9755
web: http://www.thermoanalytics.com

More information about the Trilinos-Users mailing list