[Trilinos-Users] [EXTERNAL] question about using ParMETIS with Trilinos

Alicia M Klinvex aklinvex at purdue.edu
Fri May 18 13:55:17 MDT 2012

Hello Karen,

Yes, Heidi Thornquist and Siva Rajamanickam pointed out the link line problem, and I can now use MUMPS with Trilinos.  I have another question that perhaps you can help me with.

I am trying to use LOBPCG to solve a generalized eigenvalue problem with Amesos_Mumps as a preconditioner.  I got Ifpack_ex_Amesos.cpp to run, but if I try to change the amesos solver type from Klu to Mumps, I get the output below.  I ran Amesos_compare_solvers.exe, and it was able to find and use Mumps, so I don't know what is causing this problem.  Do you have any idea why Ifpack will accept Klu but not Mumps?


                ***** Problem: Epetra::CrsMatrix
                ***** Preconditioned GMRES solution
                ***** Ifpack_AdditiveSchwarz, ov = 0, local solver =
                ***** `Amesos_Mumps'
                ***** Condition number estimate = 70.6153
                ***** No scaling

                iter:    0           residual = 1.000000e+00
                iter:    1           residual = 1.568841e-15

                Solution time: 0.001528 (sec.)
                total iterations: 1
Attempting to use an MPI routine after finalizing MPICH

*** Warning! The following Teuchos::RCPNode objects were created but have
*** not been destroyed yet.  A memory checking tool may complain that these
*** objects are not destroyed correctly.
*** There can be many possible reasons that this might occur including:
***   a) The program called abort() or exit() before main() was finished.
***      All of the objects that would have been freed through destructors
***      are not freed but some compilers (e.g. GCC) will still call the
***      destructors on static objects (which is what causes this message
***      to be printed).
***   b) The program is using raw new/delete to manage some objects and
***      delete was not called correctly and the objects not deleted hold
***      other objects through reference-counted pointers.
***   c) This may be an indication that these objects may be involved in
***      a circular dependency of reference-counted managed objects.

  0: RCPNode (map_key_void_ptr=0x266ac50)
       Information = {T=Epetra_Map, ConcreteT=Epetra_Map, p=0x266ac50, has_ownership=1}
       RCPNode address = 0x2669e30
       insertionNumber = 0
  1: RCPNode (map_key_void_ptr=0x2669f30)
       Information = {T=Epetra_CrsMatrix, ConcreteT=Epetra_CrsMatrix, p=0x2669f30, has_ownership=1}
       RCPNode address = 0x266a110
       insertionNumber = 1
  2: RCPNode (map_key_void_ptr=0x266a068)
       Information = {T=Epetra_CrsMatrix, ConcreteT=Epetra_CrsMatrix, p=0x266a068, has_ownership=0}
       RCPNode address = 0x266a6e0
       insertionNumber = 2
  3: RCPNode (map_key_void_ptr=0x266a550)
       Information = {T=Ifpack_AdditiveSchwarz<Ifpack_Amesos>, ConcreteT=Ifpack_AdditiveSchwarz<Ifpack_Amesos>, p=0x266a550, has_ownership=1}
       RCPNode address = 0x266a730
       insertionNumber = 3
  4: RCPNode (map_key_void_ptr=0x26911f0)
       Information = {T=Epetra_Time, ConcreteT=Epetra_Time, p=0x26911f0, has_ownership=1}
       RCPNode address = 0x2674b90
       insertionNumber = 4
  5: RCPNode (map_key_void_ptr=0x2691190)
       Information = {T=Epetra_MpiComm, ConcreteT=Epetra_MpiComm, p=0x2691190, has_ownership=1}
       RCPNode address = 0x2674be0
       insertionNumber = 5
  6: RCPNode (map_key_void_ptr=0x26750e0)
       Information = {T=Epetra_Map, ConcreteT=Epetra_Map, p=0x26750e0, has_ownership=1}
       RCPNode address = 0x266a9d0
       insertionNumber = 6
  7: RCPNode (map_key_void_ptr=0x26765a0)
       Information = {T=Epetra_Vector, ConcreteT=Epetra_Vector, p=0x26765a0, has_ownership=1}
       RCPNode address = 0x26753d0
       insertionNumber = 7
  8: RCPNode (map_key_void_ptr=0x2674fa0)
       Information = {T=Ifpack_LocalFilter, ConcreteT=Ifpack_LocalFilter, p=0x2674fa0, has_ownership=1}
       RCPNode address = 0x26754e0
       insertionNumber = 8
  9: RCPNode (map_key_void_ptr=0x267e570)
       Information = {T=Epetra_LinearProblem, ConcreteT=Epetra_LinearProblem, p=0x267e570, has_ownership=1}
       RCPNode address = 0x267e5c0
       insertionNumber = 9
 10: RCPNode (map_key_void_ptr=0x26755f0)
       Information = {T=Ifpack_Amesos, ConcreteT=Ifpack_Amesos, p=0x26755f0, has_ownership=1}
       RCPNode address = 0x267e680
       insertionNumber = 10
 11: RCPNode (map_key_void_ptr=0x267e700)
       Information = {T=std::vector<Teuchos::RCP<Epetra_Time>, std::allocator<Teuchos::RCP<Epetra_Time> > >, ConcreteT=std::vector<Teuchos::RCP<Epetra_Time>, std::allocator<Teuchos::RCP<Epetra_Time> > >, p=0x267e700, has_ownership=1}
       RCPNode address = 0x267eb70
       insertionNumber = 12
 12: RCPNode (map_key_void_ptr=0x267eda0)
       Information = {T=std::vector<Amesos_Time_Data, std::allocator<Amesos_Time_Data> >, ConcreteT=std::vector<Amesos_Time_Data, std::allocator<Amesos_Time_Data> >, p=0x267eda0, has_ownership=1}
       RCPNode address = 0x267ebc0
       insertionNumber = 13
 13: RCPNode (map_key_void_ptr=0x267ee30)
       Information = {T=Amesos_Mumps, ConcreteT=Amesos_Mumps, p=0x267ee30, has_ownership=1}
       RCPNode address = 0x267ff00
       insertionNumber = 14
 14: RCPNode (map_key_void_ptr=0x267fea0)
       Information = {T=Epetra_Time, ConcreteT=Epetra_Time, p=0x267fea0, has_ownership=1}
       RCPNode address = 0x267ec10
       insertionNumber = 15

NOTE: To debug issues, open a debugger, and set a break point in the function where
the RCPNode object is first created to determine the context where the object first
gets created.  Each RCPNode object is given a unique insertionNumber to allow setting
breakpoints in the code.  For example, in GDB one can perform:

1) Open the debugger (GDB) and run the program again to get updated object addresses

2) Set a breakpoint in the RCPNode insertion routine when the desired RCPNode is first
inserted.  In GDB, to break when the RCPNode with insertionNumber==3 is added, do:

  (gdb) b 'Teuchos::RCPNodeTracer::addNewRCPNode( [TAB] ' [ENTER]
  (gdb) cond 1 insertionNumber==3 [ENTER]

3) Run the program in the debugger.  In GDB, do:

  (gdb) run [ENTER]

4) Examine the call stack when the program breaks in the function addNewRCPNode(...)

----- Original Message -----
From: "Karen D Devine" <kddevin at sandia.gov>
To: "Alicia M Klinvex" <aklinvex at purdue.edu>, trilinos-users at software.sandia.gov
Cc: "Michael A Heroux" <maherou at sandia.gov>, "Faisal Saied" <fsaied at purdue.edu>
Sent: Friday, May 18, 2012 2:43:52 PM
Subject: Re: [EXTERNAL] [Trilinos-Users] question about using ParMETIS with Trilinos

Alicia:  Have you resolved this problem?  I know the link line needs to
list libmetis.a AFTER libparmetis.a.  It is not clear from your email what
order is used by TRIBITS.  Try running CMAKE with
Then check the link line for the unit test with the failing build.  If
libmetis.a is before libparmetis.a, there is a bug in TRIBITS that needs
to be fixed; we can report it to the appropriate person.

On 5/9/12 1:25 PM, "Alicia M Klinvex" <aklinvex at purdue.edu> wrote:

>I'm having a bit of trouble, and I was hoping you could help me.  I want
>to solve a generalized eigenvalue problem using block Krylov-Schur, which
>requires a linear solver in Amesos.  I would like to use a distributed
>memory solver such as Amesos_Superludist or Amesos_Mumps, both of which
>depend on ParMETIS.  I installed and tested ParMETIS, which has been
>working fine on its own.  In my configure script, I turned on ParMETIS
>and set the paths, and when I run the script, Trilinos seems to find
>everything necessary:
>-- PARSE_REQUIRED_LIBS_NAMES='parmetis;metis'
>-- ParMETIS_LIBRARY_DIRS='/u/slotnick_s2/aklinvex/ParMetis-3.0'
>-- ParMETIS_LIBRARY_NAMES='libmetis.a; libparmetis.a'
>-- PARSE_REQUIRED_LIBS_NAMES='libmetis.a; libparmetis.a'
>-- ParMETIS_LIBRARY_DIRS='/u/slotnick_s2/aklinvex/ParMetis-3.0'
>-- ParMETIS_INCLUDE_DIRS='/u/slotnick_s2/aklinvex/ParMetis-3.0'
>-- LIBNAME_SET='libmetis.a'
>-- LIBNAME_SET=' libparmetis.a'
>-- INCLUDE_FILE='parmetis.h'
>-- _ParMETIS_parmetis.h_PATH='/u/slotnick_s2/aklinvex/ParMetis-3.0'
>However, when I type "make", I get many errors of the following form:
>[ 60%] Built target
>/u/slotnick_s2/aklinvex/ParMetis-3.0/libparmetis.a(initpart.o): In
>function `Moc_InitPartition_RB__':
>initpart.c:(.text+0x840): undefined reference to
>initpart.c:(.text+0x12f5): undefined reference to
>initpart.c:(.text+0x17da): undefined reference to `METIS_WPartGraphKway2'
>initpart.c:(.text+0x1887): undefined reference to `METIS_WPartGraphKway2'
>/u/slotnick_s2/aklinvex/ParMetis-3.0/libparmetis.a(initbalance.o): In
>function `Balance_Partition__':
>initbalance.c:(.text+0x1184): undefined reference to
>initbalance.c:(.text+0x1381): undefined reference to
>initbalance.c:(.text+0x1a55): undefined reference to
>initbalance.c:(.text+0x1afb): undefined reference to
>/u/slotnick_s2/aklinvex/ParMetis-3.0/libparmetis.a(initmsection.o): In
>function `InitMultisection__':
>initmsection.c:(.text+0xbd9): undefined reference to
>initmsection.c:(.text+0xc3c): undefined reference to
>Trilinos seems to forget where METIS is.  I have tried it with ParMETIS
>3.0 and 4.0.2, both of which behave the same way.  Do you know what could
>be causing this problem?
>Thank you,
>Alicia Klinvex
>aklinvex at purdue.edu
>Trilinos-Users mailing list
>Trilinos-Users at software.sandia.gov

More information about the Trilinos-Users mailing list