[Trilinos-Users] How to get lapack/blas library consistency?
John R Cary
cary at colorado.edu
Wed Jul 7 15:45:47 MDT 2010
Thanks, all, for your help. Curiously,
iter.cary$ echo $LD_LIBRARY_PATH
iter.cary$ ldd libifpack.so | egrep '(lapack|blas|atlas)'
liblapack.so.3 => /usr/lib64/atlas/liblapack.so.3 (0x00007f37f5637000)
libblas.so.3 => /usr/lib64/libblas.so.3 (0x00007f37f53e1000)
libf77blas.so.3 => /usr/lib64/atlas/libf77blas.so.3 (0x00007f37f43bd000)
libcblas.so.3 => /usr/lib64/atlas/libcblas.so.3 (0x00007f37f419c000)
libatlas.so.3 => /usr/lib64/atlas/libatlas.so.3 (0x00007f37f38e2000)
So it does not seem to be my LD_LIBRARY_PATH.
I relinked and found
/usr/lib64/ccache/g++ -fPIC -fPIC -O3 -shared
-Wl,-soname,libifpack.so -o libifpack.so
CMakeFiles/ifpack.dir/Ifpack_IlukGraph.cpp.o
CMakeFiles/ifpack.dir/Ifpack_CrsRiluk.cpp.o
CMakeFiles/ifpack.dir/Ifpack_ILU.cpp.o
CMakeFiles/ifpack.dir/Ifpack_IKLU.cpp.o
CMakeFiles/ifpack.dir/Ifpack_IKLU_Utils.cpp.o
CMakeFiles/ifpack.dir/Ifpack_ILUT.cpp.o
CMakeFiles/ifpack.dir/Ifpack_CrsIct.cpp.o
CMakeFiles/ifpack.dir/Ifpack_ICT.cpp.o
CMakeFiles/ifpack.dir/Ifpack_IC.cpp.o
CMakeFiles/ifpack.dir/Ifpack_IC_Utils.cpp.o
CMakeFiles/ifpack.dir/Ifpack_OverlapGraph.cpp.o
CMakeFiles/ifpack.dir/icrout_cholesky_mex.c.o
CMakeFiles/ifpack.dir/icrout_quicksort.c.o
CMakeFiles/ifpack.dir/ifp_parameters.cpp.o
CMakeFiles/ifpack.dir/Ifpack_DiagPreconditioner.cpp.o
CMakeFiles/ifpack.dir/Ifpack_PointRelaxation.cpp.o
CMakeFiles/ifpack.dir/Ifpack_Condest.cpp.o
CMakeFiles/ifpack.dir/Ifpack.cpp.o
CMakeFiles/ifpack.dir/Ifpack_DenseContainer.cpp.o
CMakeFiles/ifpack.dir/Ifpack_LocalFilter.cpp.o
CMakeFiles/ifpack.dir/Ifpack_NodeFilter.cpp.o
CMakeFiles/ifpack.dir/Ifpack_DropFilter.cpp.o
CMakeFiles/ifpack.dir/Ifpack_SparsityFilter.cpp.o
CMakeFiles/ifpack.dir/Ifpack_ReorderFilter.cpp.o
CMakeFiles/ifpack.dir/Ifpack_SingletonFilter.cpp.o
CMakeFiles/ifpack.dir/Ifpack_DiagonalFilter.cpp.o
CMakeFiles/ifpack.dir/Ifpack_Graph_Epetra_RowMatrix.cpp.o
CMakeFiles/ifpack.dir/Ifpack_Graph_Epetra_CrsGraph.cpp.o
CMakeFiles/ifpack.dir/Ifpack_Utils.cpp.o
CMakeFiles/ifpack.dir/Ifpack_Chebyshev.cpp.o
CMakeFiles/ifpack.dir/Ifpack_OverlappingPartitioner.cpp.o
CMakeFiles/ifpack.dir/Ifpack_LinearPartitioner.cpp.o
CMakeFiles/ifpack.dir/Ifpack_GreedyPartitioner.cpp.o
CMakeFiles/ifpack.dir/Ifpack_METISPartitioner.cpp.o
CMakeFiles/ifpack.dir/Ifpack_EquationPartitioner.cpp.o
CMakeFiles/ifpack.dir/Ifpack_UserPartitioner.cpp.o
CMakeFiles/ifpack.dir/Ifpack_RCMReordering.cpp.o
CMakeFiles/ifpack.dir/Ifpack_METISReordering.cpp.o
CMakeFiles/ifpack.dir/Ifpack_OverlappingRowMatrix.cpp.o
CMakeFiles/ifpack.dir/Ifpack_ValidParameters.cpp.o
CMakeFiles/ifpack.dir/Ifpack_HIPS.cpp.o
CMakeFiles/ifpack.dir/Ifpack_Hypre.cpp.o
CMakeFiles/ifpack.dir/Ifpack_Euclid.cpp.o
CMakeFiles/ifpack.dir/Ifpack_AMDReordering.cpp.o
CMakeFiles/ifpack.dir/Ifpack_Amesos.cpp.o
-L/scr_iter/cary/vpall/builds/trilinos/ser/packages/ifpack/src
-L/scr_iter/cary/vpall/builds/trilinos/ser/packages/amesos/src
-L/scr_iter/cary/vpall/builds/trilinos/ser/packages/epetraext/src
-L/scr_iter/cary/vpall/builds/trilinos/ser/packages/triutils/src
-L/scr_iter/cary/vpall/builds/trilinos/ser/packages/epetra/src
-L/scr_iter/cary/vpall/builds/trilinos/ser/packages/zoltan/src
-L/scr_iter/cary/vpall/builds/trilinos/ser/packages/zoltan/siMPI/pyMPI/siMPI
-L/scr_iter/cary/vpall/builds/trilinos/ser/packages/teuchos/src
-L/scr_iter/cary/vpall/builds/trilinos/ser/packages/aztecoo/src
../../amesos/src/libamesos.so ../../aztecoo/src/libaztecoo.so
../../epetraext/src/libepetraext.so ../../epetra/src/libepetra.so
../../teuchos/src/libteuchos.so ../../zoltan/src/libzoltan.so
../../zoltan/siMPI/pyMPI/siMPI/libsimpi.so
../../triutils/src/libtriutils.so ../../epetra/src/libepetra.so -llapack
-lblas
-Wl,-rpath,/scr_iter/cary/vpall/builds/trilinos/ser/packages/ifpack/src:/scr_iter/cary/vpall/builds/trilinos/ser/packages/amesos/src:/scr_iter/cary/vpall/builds/trilinos/ser/packages/epetraext/src:/scr_iter/cary/vpall/builds/trilinos/ser/packages/triutils/src:/scr_iter/cary/vpall/builds/trilinos/ser/packages/epetra/src:/scr_iter/cary/vpall/builds/trilinos/ser/packages/zoltan/src:/scr_iter/cary/vpall/builds/trilinos/ser/packages/zoltan/siMPI/pyMPI/siMPI:/scr_iter/cary/vpall/builds/trilinos/ser/packages/teuchos/src:/scr_iter/cary/vpall/builds/trilinos/ser/packages/aztecoo/src:
Now I don't see -L/usr/lib64, which would correspond to what was
in Trilinos_TPL_LIBRARIES. So with that absent, I guess this
goes to
iter.cary$ cat /etc/ld.so.conf
include ld.so.conf.d/*.conf
and with
iter.cary$ pwd
/etc/ld.so.conf.d
iter.cary$ ls
atlas-x86_64.conf mysql-x86_64.conf xulrunner-64.conf
java-x86.conf qt-x86_64.conf
kernel-2.6.29.4-167.fc11.x86_64.conf tix-x86_64.conf
iter.cary$
atlas being alphabetically first. So I think the build system
is selecting the lapack in /usr/lib64/atlas due to the system
settings and ignoring what I requested. It picks up the
unrequested libraries from
iter.cary$ ldd /usr/lib64/atlas/liblapack.so.3.0
linux-vdso.so.1 => (0x00007fff72144000)
libgfortran.so.3 => /usr/lib64/libgfortran.so.3 (0x00007f86695dd000)
libf77blas.so.3 => /usr/lib64/atlas/libf77blas.so.3 (0x00007f86693be000)
libcblas.so.3 => /usr/lib64/atlas/libcblas.so.3 (0x00007f866919e000)
libc.so.6 => /lib64/libc.so.6 (0x00007f8668e30000)
libm.so.6 => /lib64/libm.so.6 (0x00007f8668bac000)
libatlas.so.3 => /usr/lib64/atlas/libatlas.so.3 (0x00007f86682f1000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f86680d5000)
/lib64/ld-linux-x86-64.so.2 (0x0000003933200000)
If this is true, then I cannot count on Trilinos_TPL_LIBRARIES.
????
Thx....John
On 7/7/10 10:41 AM, Perschbacher, Brent M wrote:
> Just to give a quick and dirty description of what I just sent John. The
> values that are in TrilinosConfig.cmake are what Trilinos used to
> compile against, however, that doesn’t necessarily mean that will be
> used to link at runtime in the case of shared libraries. When linking at
> run time your environment, specifically your LD_LIBRARY_PATH and PATH
> can affect where shared libraries are found. For executables you can
> tell them where to find shared libraries overriding the LD_LIBRARY_PATH
> (for gcc this is done with rpath), however, I’m not sure if you can do
> that when building a library as rpath is a linker option. The tool ldd
> uses the same logic to determine what libraries will be needed which is
> good, but if rpath isn’t set for the library then it can only use the
> environment to figure out which shared libraries will be linked in. I’m
> guessing that Johns LD_LIBRARY_PATH has the atlas path before /usr/lib64
> which is why the atlas lapack was found. Blas however, was found in
> /usr/lib64 and that is likely because atlas doesn’t provide a
> “libblas.*” but instead relies on f77blas and cblas for its blas
> implementation which is why those appear on the list of libraries as well.
>
> Brent
>
>
> On 7/7/10 10:22 AM, "Buermann, Lee R" <LRBUERMANN at csbsju.edu> wrote:
>
> If what your building is using CMake, you should be able to write a
> Find Script for LAPACK. Trilinos has one in
> “yourTrilinosSource/cmake/TPLs/FindTPLLAPACK.cmake” I would think
> that if you use the same script as Trilinos, you would find the same
> libraries as Trilinos.
>
> Lee
>
>
> *From:* Kurtis Nusbaum [mailto:klnusbaum at gmail.com]
> *Sent:* Wednesday, July 07, 2010 11:07 AM
> *To:* John R Cary; Buermann, Lee R
> *Cc:* Trilinos Users
> *Subject:* Re: [Trilinos-Users] How to get lapack/blas library
> consistency?
>
> Lee Buermann has actually been doing some work with the way Trilinos
> detects LAPACK. He might be able to speak to this issue. Lee, any
> thoughts?
>
> -Kurtis
>
> On Wed, Jul 7, 2010 at 9:51 AM, John R Cary <cary at colorado.edu> wrote:
>
> When we build something with Trilinos, we want (I assume)
> to ensure that we use the same lapack libs that Trilinos
> uses. Is there a way to detect this?
>
> I see:
>
> iter.cary$ grep -i lapack TrilinosConfig.cmake
> SET(Trilinos_TPL_LIBRARIES
> "/usr/lib64/liblapack.so;/usr/lib64/libblas.so")
> SET(Trilinos_TPL_LIST "LAPACK;BLAS")
>
> which makes me this that those were used, but then I
> see:
>
> iter.cary$ ldd lib/libifpack.so | egrep '(lapack|blas|atlas)'
> liblapack.so.3 => /usr/lib64/atlas/liblapack.so.3 (0x00007f61d5999000)
> libblas.so.3 => /usr/lib64/libblas.so.3 (0x00007f61d5743000)
> libf77blas.so.3 => /usr/lib64/atlas/libf77blas.so.3 (0x00007f61d471f000)
> libcblas.so.3 => /usr/lib64/atlas/libcblas.so.3 (0x00007f61d44fe000)
> libatlas.so.3 => /usr/lib64/atlas/libatlas.so.3 (0x00007f61d3c44000)
>
> which seems to indicate that Trilinos chose not to use
> the specified libs, but chose to use atlas and the
> system blas instead.
>
> I need a strategy that works for static builds, so cannot
> use ldd to detect.
>
> Thanks....John
>
> _______________________________________________
> Trilinos-Users mailing list
> Trilinos-Users at software.sandia.gov
> http://software.sandia.gov/mailman/listinfo/trilinos-users
>
>
More information about the Trilinos-Users
mailing list