[Trilinos-Users] Question about inserting a row not owned by a processor.
Williams, Alan B
william at sandia.gov
Wed Jun 16 14:08:45 MDT 2010
When working with on-processor values, it should be fine if you use Epetra_MultiVector methods, since Epetra_MultiVector is the base-class for Epetra_FEVector.
Are you using the '[]' operator? Note that Epetra_Vector and Epetra_MultiVector both have '[]' and they are not the same. Note that Epetra_Vector is not a base-class for Epetra_FEVector.
Alan
> -----Original Message-----
> From: trilinos-users-bounces at software.sandia.gov [mailto:trilinos-
> users-bounces at software.sandia.gov] On Behalf Of M. Scot Breitenfeld
> Sent: Wednesday, June 16, 2010 2:00 PM
> To: trilinos-users at software.sandia.gov
> Subject: Re: [Trilinos-Users] Question about inserting a row not owned
> by a processor.
>
> Is this only for off-processor values that you have to use
> Epetra_FEVector member functions, or do you always have to use them?
> The
> case that I'm running now seg faults when run on one processor and has
> no off-processor contributions. It also worked correctly with
> Epetra_vector and Epetra_crsMatrix since it did not have an issue with
> off-processor values, so I'm replacing them with the *FE* classes and
> testing to make sure it still works.
>
>
> On 06/16/2010 02:43 PM, Williams, Alan B wrote:
> > When adding off-processor values to an Epetra_FEVector, you have to
> use one of the Epetra_FEVector member functions, such as
> SumIntoGlobalValues or ReplaceGlobalValues. Otherwise you're accessing
> memory or methods of the base-class (Epetra_MultiVector) which doesn't
> support off-processor contributions.
> >
> > Alan
> >
> >
> >
> >> -----Original Message-----
> >> From: M. Scot Breitenfeld [mailto:brtnfld at uiuc.edu]
> >> Sent: Wednesday, June 16, 2010 1:38 PM
> >> To: Williams, Alan B
> >> Cc: trilinos-users at software.sandia.gov
> >> Subject: Re: [Trilinos-Users] Question about inserting a row not
> owned
> >> by a processor.
> >>
> >> Alan, thanks that does seem to work for the K matrix, but I'm having
> >> problems getting the rhs vector to work.
> >>
> >> I changed: globalRhs_ = Teuchos::rcp(new
> >> Epetra_Vector(*globalRowMap_));
> >> to : globalRhsFE_ = Teuchos::rcp(new
> >> Epetra_FEVector(*globalRowMap_));
> >>
> >> where
> >>
> >> globalRowMap_ = Teuchos::rcp(new Epetra_Map(-1, numUnks,
> >> globalGIDList.Values(), 1, comm_));
> >>
> >> but when I try:
> >>
> >> int BasicLinProbMgr::insertRhsValue(int GID, double value) {
> >> int rhsLID = globalRowMap_->LID(GID); // Get solver LID
> >> if (rhsLID==-1) {
> >> ierr += rhsLID;
> >> }
> >> else
> >>
> >> I changed: *globalRhs_)[rhsLID] += value
> >> to: *((*globalRhsFE_)[rhsLID]) += value;
> >> return(ierr);
> >> }
> >>
> >> It seg faults at: *((*globalRhsFE_)[rhsLID]) += value. It loops
> twice,
> >> rhsLID = 0,1 then on 2 it seg faults.
> >>
> >> For rhsLID=2 it passes the tests in:
> >>
> >> T& RCP<T>::operator*() const
> >> {
> >> debug_assert_not_null();
> >> debug_assert_valid_ptr();
> >> return *ptr_;
> >> }
> >>
> >> returns to *((*globalRhsFE_)[rhsLID]) += value and then crashes.
> >>
> >> Is there something additional I need to do to use Epetra_FEVector as
> >> opposed to Epetra_Vector
> >>
> >> Here is the state of *globalRhsFE before it segfaults:
> >>
> >> *globalRhsFE_ = class Epetra_FEVector
> >> myFirstID_ 1
> >> myNumIDs_ 126
> >> nonlocalIDs_ NULL
> >> nonlocalElementSize_ NULL
> >> numNonlocalIDs_ 0
> >> numNonlocalIDsAlloc_ 0
> >> nonlocalCoefs_ 0xa382058
> >> *((*globalRhsFE_).nonlocalCoefs_) NULL
> >> numNonlocalCoefs_ 0
> >> numNonlocalCoefsAlloc_ 0
> >> ignoreNonLocalEntries_ false
> >> Values_ 0xa382638
> >> *((*globalRhsFE_).Values_) 0
> >> Pointers_ 0xa382048
> >> *((*globalRhsFE_).Pointers_) 0xa382638
> >> *(*((*globalRhsFE_).Pointers_)) 0
> >> MyLength_ 126
> >> GlobalLength_ 126
> >> NumVectors_ 1
> >> UserAllocated_ false
> >> ConstantStride_ true
> >> Stride_ 126
> >> Allocated_ true
> >> DoubleTemp_ NULL
> >> Vectors_ NULL
> >> Util_ = class Epetra_Util
> >> Seed_ 1714636915
> >> static const double chopVal_
> >> Map_ = class Epetra_BlockMap
> >> BlockMapData_ 0xa380788
> >> *(((*globalRhsFE_).Map_).BlockMapData_) = class Epetra_BlockMapData
> >> Comm_ 0xa38089c
> >> *((*(((*globalRhsFE_).Map_).BlockMapData_)).Comm_) = class
> Epetra_Comm
> >> Directory_ NULL
> >> LID_ = class Epetra_IntSerialDenseVector
> >> CV_ Copy
> >> A_Copied_ false
> >> M_ 0
> >> N_ 1
> >> LDA_ 0
> >> A_ NULL
> >> Label_ 0xa380010="Epetra::IntSerialDenseVector"
> >> *(((*(((*globalRhsFE_).Map_).BlockMapData_)).LID_).Label_) 'E'
> >> static int TracebackMode
> >> MyGlobalElements_ = class Epetra_IntSerialDenseVector
> >> CV_ Copy
> >> A_Copied_ true
> >> M_ 126
> >> N_ 1
> >> LDA_ 126
> >> A_ 0xa380bd0
> >> *(((*(((*globalRhsFE_).Map_).BlockMapData_)).MyGlobalElements_).A_)
> >> 1
> >> Label_ 0xa3808c0="Epetra::IntSerialDenseVector"
> >> static int TracebackMode
> >> FirstPointInElementList_ = class Epetra_IntSerialDenseVector
> >> CV_ Copy
> >> A_Copied_ false
> >> M_ 0
> >> N_ 1
> >> LDA_ 0
> >> A_ NULL
> >> Label_ 0xa3808e8="Epetra::IntSerialDenseVector"
> >>
> *(((*(((*globalRhsFE_).Map_).BlockMapData_)).FirstPointInElementList_).
> >> Label_)
> >> 'E'
> >> static int TracebackMode
> >> ElementSizeList_ = class Epetra_IntSerialDenseVector
> >> CV_ Copy
> >> A_Copied_ false
> >> M_ 0
> >> N_ 1
> >> LDA_ 0
> >> A_ NULL
> >> Label_ 0xa380910="Epetra::IntSerialDenseVector"
> >>
> *(((*(((*globalRhsFE_).Map_).BlockMapData_)).ElementSizeList_).Label_)
> >> 'E'
> >> static int TracebackMode
> >> PointToElementList_ = class Epetra_IntSerialDenseVector
> >> CV_ Copy
> >> A_Copied_ false
> >> M_ 0
> >> N_ 1
> >> LDA_ 0
> >> A_ NULL
> >> Label_ 0xa380938="Epetra::IntSerialDenseVector"
> >> static int TracebackMode
> >> NumGlobalElements_ 126
> >> NumMyElements_ 126
> >> IndexBase_ 1
> >> ElementSize_ 1
> >> MinMyElementSize_ 1
> >> MaxMyElementSize_ 1
> >> MinElementSize_ 1
> >> MaxElementSize_ 1
> >> MinAllGID_ 1
> >> MaxAllGID_ 126
> >> MinMyGID_ 1
> >> MaxMyGID_ 126
> >> MinLID_ 0
> >> MaxLID_ 125
> >> NumGlobalPoints_ 126
> >> NumMyPoints_ 126
> >> ConstantElementSize_ true
> >> LinearMap_ false
> >> DistributedGlobal_ false
> >> OneToOne_ true
> >> LastContiguousGID_ 126
> >> LastContiguousGIDLoc_ 125
> >> LIDHash_ NULL
> >> ReferenceCount_ 10
> >> Label_ 0xa3821a0="Epetra::Map"
> >> *(((*globalRhsFE_).Map_).Label_) 'E'
> >> static int TracebackMode
> >> Comm_ 0xa38089c
> >> DistributedGlobal_ false
> >> Exports_ NULL
> >> Imports_ NULL
> >> LenExports_ 0
> >> LenImports_ 0
> >> Sizes_ NULL
> >> Label_ 0xa382188="Epetra::MultiVector"
> >> *((*globalRhsFE_).Label_) 'E'
> >> static int TracebackMode
> >> FlopCounter_ NULL
> >>
> >>
> >>
> >> On 06/15/2010 01:32 PM, Williams, Alan B wrote:
> >>
> >>> Scot,
> >>> It's possible that the class Epetra_FECrsMatrix can do what you
> need.
> >>>
> >> It allows for inserting entries that are owned by another processor,
> >> and then communicates them to that processor during the
> >> globalAssemble() method.
> >>
> >>> Alan
> >>>
> >>>
> >>>
> >>>
> >>>> -----Original Message-----
> >>>> From: trilinos-users-bounces at software.sandia.gov [mailto:trilinos-
> >>>> users-bounces at software.sandia.gov] On Behalf Of M. Scot
> Breitenfeld
> >>>> Sent: Tuesday, June 15, 2010 10:52 AM
> >>>> To: trilinos-users at software.sandia.gov
> >>>> Subject: [Trilinos-Users] Question about inserting a row not owned
> >>>>
> >> by a
> >>
> >>>> processor.
> >>>>
> >>>> Greetings,
> >>>>
> >>>> I'll try to explain a simplified version of my situation the best
> >>>>
> >> that
> >>
> >>>> I
> >>>> can. Lets say I have a line discretized as follows:
> >>>>
> >>>> Proc 0 | Proc 1
> >>>> o------o------o---|---o-----o-----o
> >>>> 1 2 3 4 5 6 (global number)
> >>>>
> >>>> where o are nodes (there are no elements for this method) and
> there
> >>>>
> >> is
> >>
> >>>> only 1 d.o.f at each node
> >>>>
> >>>> For Processor 0 I register: as my global node list: 1, 2, 3, 4
> >>>> as the row list: 1, 2,
> 3,
> >>>>
> >> 4
> >>
> >>>> as my owned node list:
> 1,
> >>>>
> >> 2,
> >>
> >>>> 3
> >>>>
> >>>> Proc 0
> >>>> o------o------o---|---o
> >>>> 1 2 3 4 (global number)
> >>>> 1 2 3 4 (rows in global list)
> >>>>
> >>>> For Processor 1 I register: as my global node list: 4, 5, 6, 3
> >>>> as the row list: 4, 5,
> 6,
> >>>>
> >> 3
> >>
> >>>> as my owned node list:
> 4,
> >>>>
> >> 5,
> >>
> >>>> 6
> >>>>
> >>>> Proc 1
> >>>> o---|---o------o------o
> >>>> 3 4 5 6 (global number)
> >>>> 3 4 5 6 (rows in global list)
> >>>>
> >>>> Now up to this point it works fine in parallel. But for my method
> I
> >>>> have
> >>>> something like this:
> >>>>
> >>>> row 1: (T12-T21)*V2+(T13-T31)*V3+(T14-T41)*V4
> >>>> row 2: (T21-T12)*V2+(T23-T32)*V3+(T24-T42)*V4
> >>>> row 3: (T31-T13)*V2+(T34-T43)*V3
> >>>> row 4: (T41-T14)*V2+(T45-T54)*V3+(T34-T43)*V4
> >>>> ....
> >>>>
> >>>> T12, for example, means node 1 is influenced by node 2; meaning
> >>>> processor 0 will calculate T12, T13, T14 which is fine. But T14
> also
> >>>> needs to be subtracted from row 4. The problem is row 4 (node 4)
> is
> >>>>
> >> not
> >>
> >>>> owned by processor 0, it's on processor 1. When I use
> >>>> InsertGlobalValues
> >>>> to insert the values in row 4 on processor 0 it will return an
> >>>>
> >> error.
> >>
> >>>> Do
> >>>> I have to write mpi code to pass the T14 matrix to processor one
> to
> >>>> insert into row 4, or is there a routine that I can use instead,
> or
> >>>> maybe I'm missing something.
> >>>>
> >>>> Thanks,
> >>>> Scot
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> Trilinos-Users mailing list
> >>>> Trilinos-Users at software.sandia.gov
> >>>> http://software.sandia.gov/mailman/listinfo/trilinos-users
> >>>>
> >>>>
> >>>
> >>>
> >>
> >
> >
> >
>
>
> _______________________________________________
> Trilinos-Users mailing list
> Trilinos-Users at software.sandia.gov
> http://software.sandia.gov/mailman/listinfo/trilinos-users
More information about the Trilinos-Users
mailing list