[Trilinos-Users] Question about inserting a row not owned by a processor.

M. Scot Breitenfeld brtnfld at uiuc.edu
Thu Jun 17 08:50:10 MDT 2010


oh I think I understand my mistake, for Epetra_FEVector I used the
Epetra_MultiVector  operator,  so I changed

(*globalRhsFE_)[rhsLID] += value;

 which for Epertra_Vector was: double & operator[]
<http://trilinos.sandia.gov/packages/docs/r10.2/packages/epetra/doc/html/classEpetra__Vector.html#44ab8096e05da1662db0693cc885610f>
(int index))

to *((*globalRhsFE_)[rhsLID]) += value; for Epetra_MultiVector double
*& operator[]
<http://trilinos.sandia.gov/packages/docs/r10.2/packages/epetra/doc/html/classEpetra__MultiVector.html#134683eb593b2bfece80a7aa6cce2985>
(int i)

but that operator is for the ith vector in the multi-vector, not for the
ith element in the vector.  I see I need to use, as you said,
SumIntoGlobalValue to assign a value to an element in the rhs vector
when using the MultiVector operator:

int ierr = globalRhsFE_->SumIntoGlobalValues(rhsLID, 1, value);

On that note, I don't see a function that accesses an element in a
multi-vector, when globalLhs was Epetra_vector I could just use,

value = (*globalLhs_)[i];

do I have to extract the whole vector using
Epetra_MultiVector::ExtractView to access an element of the vector?

Thanks.


On 06/16/2010 03:08 PM, Williams, Alan B wrote:
> When working with on-processor values, it should be fine if you use Epetra_MultiVector methods, since Epetra_MultiVector is the base-class for Epetra_FEVector.
>
> Are you using the '[]' operator? Note that Epetra_Vector and Epetra_MultiVector both have '[]' and they are not the same. Note that Epetra_Vector is not a base-class for Epetra_FEVector.
>
> Alan
>
>
>   
>> -----Original Message-----
>> From: trilinos-users-bounces at software.sandia.gov [mailto:trilinos-
>> users-bounces at software.sandia.gov] On Behalf Of M. Scot Breitenfeld
>> Sent: Wednesday, June 16, 2010 2:00 PM
>> To: trilinos-users at software.sandia.gov
>> Subject: Re: [Trilinos-Users] Question about inserting a row not owned
>> by a processor.
>>
>> Is this only for off-processor values that you have to use
>> Epetra_FEVector member functions, or do you always have to use them?
>> The
>> case that I'm running now seg faults when run on one processor and  has
>> no off-processor contributions. It also worked correctly with
>> Epetra_vector and Epetra_crsMatrix since it did not have an issue with
>> off-processor values, so I'm replacing them with the *FE* classes and
>> testing to make sure it still works.
>>
>>
>> On 06/16/2010 02:43 PM, Williams, Alan B wrote:
>>     
>>> When adding off-processor values to an Epetra_FEVector, you have to
>>>       
>> use one of the Epetra_FEVector member functions, such as
>> SumIntoGlobalValues or ReplaceGlobalValues. Otherwise you're accessing
>> memory or methods of the base-class (Epetra_MultiVector) which doesn't
>> support off-processor contributions.
>>     
>>> Alan
>>>
>>>
>>>
>>>       
>>>> -----Original Message-----
>>>> From: M. Scot Breitenfeld [mailto:brtnfld at uiuc.edu]
>>>> Sent: Wednesday, June 16, 2010 1:38 PM
>>>> To: Williams, Alan B
>>>> Cc: trilinos-users at software.sandia.gov
>>>> Subject: Re: [Trilinos-Users] Question about inserting a row not
>>>>         
>> owned
>>     
>>>> by a processor.
>>>>
>>>> Alan, thanks that does seem to work for the K matrix, but I'm having
>>>> problems getting the rhs vector to work.
>>>>
>>>>   I changed:   globalRhs_ = Teuchos::rcp(new
>>>> Epetra_Vector(*globalRowMap_));
>>>>   to           :   globalRhsFE_ = Teuchos::rcp(new
>>>> Epetra_FEVector(*globalRowMap_));
>>>>
>>>> where
>>>>
>>>>  globalRowMap_ = Teuchos::rcp(new Epetra_Map(-1, numUnks,
>>>> globalGIDList.Values(), 1, comm_));
>>>>
>>>> but when I try:
>>>>
>>>> int BasicLinProbMgr::insertRhsValue(int GID, double value) {
>>>>   int rhsLID = globalRowMap_->LID(GID); // Get solver LID
>>>>   if (rhsLID==-1) {
>>>>     ierr += rhsLID;
>>>>   }
>>>>   else
>>>>
>>>>   I changed: *globalRhs_)[rhsLID] += value
>>>>              to: *((*globalRhsFE_)[rhsLID]) += value;
>>>>   return(ierr);
>>>> }
>>>>
>>>> It seg faults at: *((*globalRhsFE_)[rhsLID]) += value. It loops
>>>>         
>> twice,
>>     
>>>> rhsLID = 0,1 then on 2 it seg faults.
>>>>
>>>> For rhsLID=2 it passes the tests in:
>>>>
>>>> T& RCP<T>::operator*() const
>>>> {
>>>>   debug_assert_not_null();
>>>>   debug_assert_valid_ptr();
>>>>   return *ptr_;
>>>> }
>>>>
>>>> returns to *((*globalRhsFE_)[rhsLID]) += value and then crashes.
>>>>
>>>> Is there something additional I need to do to use Epetra_FEVector as
>>>> opposed to Epetra_Vector
>>>>
>>>> Here is the state of *globalRhsFE before it segfaults:
>>>>
>>>> *globalRhsFE_ = class Epetra_FEVector
>>>> myFirstID_    1
>>>> myNumIDs_    126
>>>> nonlocalIDs_    NULL
>>>> nonlocalElementSize_    NULL
>>>> numNonlocalIDs_    0
>>>> numNonlocalIDsAlloc_    0
>>>> nonlocalCoefs_    0xa382058
>>>> *((*globalRhsFE_).nonlocalCoefs_)    NULL
>>>> numNonlocalCoefs_    0
>>>> numNonlocalCoefsAlloc_    0
>>>> ignoreNonLocalEntries_    false
>>>> Values_    0xa382638
>>>> *((*globalRhsFE_).Values_)    0
>>>> Pointers_    0xa382048
>>>> *((*globalRhsFE_).Pointers_)    0xa382638
>>>> *(*((*globalRhsFE_).Pointers_))    0
>>>> MyLength_    126
>>>> GlobalLength_    126
>>>> NumVectors_    1
>>>> UserAllocated_    false
>>>> ConstantStride_    true
>>>> Stride_    126
>>>> Allocated_    true
>>>> DoubleTemp_    NULL
>>>> Vectors_    NULL
>>>> Util_ = class Epetra_Util
>>>> Seed_    1714636915
>>>> static const double chopVal_
>>>> Map_ = class Epetra_BlockMap
>>>> BlockMapData_    0xa380788
>>>> *(((*globalRhsFE_).Map_).BlockMapData_) = class Epetra_BlockMapData
>>>> Comm_    0xa38089c
>>>> *((*(((*globalRhsFE_).Map_).BlockMapData_)).Comm_) = class
>>>>         
>> Epetra_Comm
>>     
>>>> Directory_    NULL
>>>> LID_ = class Epetra_IntSerialDenseVector
>>>> CV_    Copy
>>>> A_Copied_    false
>>>> M_    0
>>>> N_    1
>>>> LDA_    0
>>>> A_    NULL
>>>> Label_    0xa380010="Epetra::IntSerialDenseVector"
>>>> *(((*(((*globalRhsFE_).Map_).BlockMapData_)).LID_).Label_)    'E'
>>>> static int TracebackMode
>>>> MyGlobalElements_ = class Epetra_IntSerialDenseVector
>>>> CV_    Copy
>>>> A_Copied_    true
>>>> M_    126
>>>> N_    1
>>>> LDA_    126
>>>> A_    0xa380bd0
>>>> *(((*(((*globalRhsFE_).Map_).BlockMapData_)).MyGlobalElements_).A_)
>>>> 1
>>>> Label_    0xa3808c0="Epetra::IntSerialDenseVector"
>>>> static int TracebackMode
>>>> FirstPointInElementList_ = class Epetra_IntSerialDenseVector
>>>> CV_    Copy
>>>> A_Copied_    false
>>>> M_    0
>>>> N_    1
>>>> LDA_    0
>>>> A_    NULL
>>>> Label_    0xa3808e8="Epetra::IntSerialDenseVector"
>>>>
>>>>         
>> *(((*(((*globalRhsFE_).Map_).BlockMapData_)).FirstPointInElementList_).
>>     
>>>> Label_)
>>>> 'E'
>>>> static int TracebackMode
>>>> ElementSizeList_ = class Epetra_IntSerialDenseVector
>>>> CV_    Copy
>>>> A_Copied_    false
>>>> M_    0
>>>> N_    1
>>>> LDA_    0
>>>> A_    NULL
>>>> Label_    0xa380910="Epetra::IntSerialDenseVector"
>>>>
>>>>         
>> *(((*(((*globalRhsFE_).Map_).BlockMapData_)).ElementSizeList_).Label_)
>>     
>>>> 'E'
>>>> static int TracebackMode
>>>> PointToElementList_ = class Epetra_IntSerialDenseVector
>>>> CV_    Copy
>>>> A_Copied_    false
>>>> M_    0
>>>> N_    1
>>>> LDA_    0
>>>> A_    NULL
>>>> Label_    0xa380938="Epetra::IntSerialDenseVector"
>>>> static int TracebackMode
>>>> NumGlobalElements_    126
>>>> NumMyElements_    126
>>>> IndexBase_    1
>>>> ElementSize_    1
>>>> MinMyElementSize_    1
>>>> MaxMyElementSize_    1
>>>> MinElementSize_    1
>>>> MaxElementSize_    1
>>>> MinAllGID_    1
>>>> MaxAllGID_    126
>>>> MinMyGID_    1
>>>> MaxMyGID_    126
>>>> MinLID_    0
>>>> MaxLID_    125
>>>> NumGlobalPoints_    126
>>>> NumMyPoints_    126
>>>> ConstantElementSize_    true
>>>> LinearMap_    false
>>>> DistributedGlobal_    false
>>>> OneToOne_    true
>>>> LastContiguousGID_    126
>>>> LastContiguousGIDLoc_    125
>>>> LIDHash_    NULL
>>>> ReferenceCount_    10
>>>> Label_    0xa3821a0="Epetra::Map"
>>>> *(((*globalRhsFE_).Map_).Label_)    'E'
>>>> static int TracebackMode
>>>> Comm_    0xa38089c
>>>> DistributedGlobal_    false
>>>> Exports_    NULL
>>>> Imports_    NULL
>>>> LenExports_    0
>>>> LenImports_    0
>>>> Sizes_    NULL
>>>> Label_    0xa382188="Epetra::MultiVector"
>>>> *((*globalRhsFE_).Label_)    'E'
>>>> static int TracebackMode
>>>> FlopCounter_    NULL
>>>>
>>>>
>>>>
>>>> On 06/15/2010 01:32 PM, Williams, Alan B wrote:
>>>>
>>>>         
>>>>> Scot,
>>>>> It's possible that the class Epetra_FECrsMatrix can do what you
>>>>>           
>> need.
>>     
>>>>>           
>>>> It allows for inserting entries that are owned by another processor,
>>>> and then communicates them to that processor during the
>>>> globalAssemble() method.
>>>>
>>>>         
>>>>> Alan
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>           
>>>>>> -----Original Message-----
>>>>>> From: trilinos-users-bounces at software.sandia.gov [mailto:trilinos-
>>>>>> users-bounces at software.sandia.gov] On Behalf Of M. Scot
>>>>>>             
>> Breitenfeld
>>     
>>>>>> Sent: Tuesday, June 15, 2010 10:52 AM
>>>>>> To: trilinos-users at software.sandia.gov
>>>>>> Subject: [Trilinos-Users] Question about inserting a row not owned
>>>>>>
>>>>>>             
>>>> by a
>>>>
>>>>         
>>>>>> processor.
>>>>>>
>>>>>> Greetings,
>>>>>>
>>>>>> I'll try to explain a simplified version of my situation the best
>>>>>>
>>>>>>             
>>>> that
>>>>
>>>>         
>>>>>> I
>>>>>> can. Lets say I have a line discretized as follows:
>>>>>>
>>>>>>        Proc 0            |    Proc 1
>>>>>>       o------o------o---|---o-----o-----o
>>>>>>       1       2      3          4     5      6  (global number)
>>>>>>
>>>>>> where o are nodes (there are no elements for this method) and
>>>>>>             
>> there
>>     
>>>>>>             
>>>> is
>>>>
>>>>         
>>>>>> only 1 d.o.f at each node
>>>>>>
>>>>>> For Processor 0 I register: as my global node list: 1, 2, 3, 4
>>>>>>                                            as the row list: 1, 2,
>>>>>>             
>> 3,
>>     
>>>>>>             
>>>> 4
>>>>
>>>>         
>>>>>>                                            as my owned node list:
>>>>>>             
>> 1,
>>     
>>>>>>             
>>>> 2,
>>>>
>>>>         
>>>>>> 3
>>>>>>
>>>>>>        Proc 0
>>>>>>       o------o------o---|---o
>>>>>>       1       2        3       4 (global number)
>>>>>>         1         2          3         4   (rows in global list)
>>>>>>
>>>>>> For Processor 1 I register: as my global node list: 4, 5, 6, 3
>>>>>>                                            as the row list: 4, 5,
>>>>>>             
>> 6,
>>     
>>>>>>             
>>>> 3
>>>>
>>>>         
>>>>>>                                            as my owned node list:
>>>>>>             
>> 4,
>>     
>>>>>>             
>>>> 5,
>>>>
>>>>         
>>>>>> 6
>>>>>>
>>>>>>       Proc 1
>>>>>>       o---|---o------o------o
>>>>>>       3        4       5       6 (global number)
>>>>>>         3         4          5         6   (rows in global list)
>>>>>>
>>>>>> Now up to this point it works fine in parallel. But for my method
>>>>>>             
>> I
>>     
>>>>>> have
>>>>>> something like this:
>>>>>>
>>>>>> row 1:  (T12-T21)*V2+(T13-T31)*V3+(T14-T41)*V4
>>>>>> row 2:  (T21-T12)*V2+(T23-T32)*V3+(T24-T42)*V4
>>>>>> row 3:  (T31-T13)*V2+(T34-T43)*V3
>>>>>> row 4:  (T41-T14)*V2+(T45-T54)*V3+(T34-T43)*V4
>>>>>> ....
>>>>>>
>>>>>> T12, for example, means node 1 is influenced by node 2; meaning
>>>>>> processor 0 will calculate T12, T13, T14 which is fine. But T14
>>>>>>             
>> also
>>     
>>>>>> needs to be subtracted from row 4. The problem is row 4 (node 4)
>>>>>>             
>> is
>>     
>>>>>>             
>>>> not
>>>>
>>>>         
>>>>>> owned by processor 0, it's on processor 1. When I use
>>>>>> InsertGlobalValues
>>>>>> to insert the values in row 4 on processor 0 it will return an
>>>>>>
>>>>>>             
>>>> error.
>>>>
>>>>         
>>>>>> Do
>>>>>> I have to write mpi code to pass the T14 matrix to processor one
>>>>>>             
>> to
>>     
>>>>>> insert into row 4, or is there a routine that I can use instead,
>>>>>>             
>> or
>>     
>>>>>> maybe I'm missing something.
>>>>>>
>>>>>> Thanks,
>>>>>> Scot
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Trilinos-Users mailing list
>>>>>> Trilinos-Users at software.sandia.gov
>>>>>> http://software.sandia.gov/mailman/listinfo/trilinos-users
>>>>>>
>>>>>>
>>>>>>             
>>>>>           
>>>>         
>>>
>>>       
>> _______________________________________________
>> Trilinos-Users mailing list
>> Trilinos-Users at software.sandia.gov
>> http://software.sandia.gov/mailman/listinfo/trilinos-users
>>     
>   




More information about the Trilinos-Users mailing list