[Trilinos-Users] MKL / ACML

Bartlett, Roscoe A rabartl at sandia.gov
Sat Dec 11 18:41:28 MST 2010


Alireza,

Your experiments below, finishing with successful #5, are exactly how the TPL system which includes using the built-in CMake function find_library() is supposed to work.  Thanks for providing this detailed example of what to do and what not to do.

Cheers,

- Ross


From: trilinos-users-bounces at software.sandia.gov [mailto:trilinos-users-bounces at software.sandia.gov] On Behalf Of Alireza Nejadmalayeri
Sent: Saturday, December 11, 2010 12:00 AM
To: Hu, Jonathan; Meyer, Hal; Bartlett, Roscoe A; Tommy Binford; Kelly Fermoyle; Conjeepuram Subramanian, Natarajan; Willenbring, James M
Cc: trilinos-users at software.sandia.gov
Subject: Re: [Trilinos-Users] MKL / ACML

Thank you all so much for the hints and suggestions.

I have tried some different possibilities; Ross and Jonathan's suggestions work and Kelly's approach *partially* works (compiled but returns segmentation fault while running)

Let me summarize as follows:

1)
          -D BLAS_LIBRARY_DIRS:PATH=$MKL_HOME/lib/em64t \
          -D LAPACK_LIBRARY_DIRS:PATH=$MKL_HOME/lib/em64t \
Result: configuration will find   /usr/lib64/liblapack.so  and   /usr/lib64/libblas.so
But these are not part of MKL


2)
          -D BLAS_LIBRARY_DIRS:PATH=$MKL_HOME/lib/em64t \
          -D LAPACK_LIBRARY_DIRS:PATH=$MKL_HOME/lib/em64t \
          -D LAPACK_LIBRARY_NAMES:STRING="libmkl_intel_lp64;libmkl_intel_thread;libmkl_core;libiomp5" \
          -D BLAS_LIBRARY_NAMES:STRING="libmkl_intel_lp64;libmkl_intel_thread;libmkl_core;libiomp5" \
OR
          -D BLAS_LIBRARY_DIRS:PATH=$MKL_HOME/lib/em64t \
          -D LAPACK_LIBRARY_DIRS:PATH=$MKL_HOME/lib/em64t \
          -D LAPACK_LIBRARY_NAMES:STRING="lmkl_intel_lp64;lmkl_intel_thread;lmkl_core;liomp5" \
          -D BLAS_LIBRARY_NAMES:STRING="lmkl_intel_lp64;lmkl_intel_thread;lmkl_core;liomp5" \

i.e., LIBRARY_NAMES set the same as MKL environment variable

Result:  configuration fails. LAPACK and BLAS can not be found


3)
With Natarajan's script, I got  ld: cannot find -lmkl_intel_lp64    while compiling my code.


4)
Based on what Kelly recommended:
I used  the TPL_BLAS_LIBRARIES:STRING   and  TPL_LAPACK_LIBRARIES:STRING
With this script, I could configure successfully, built the Trilinos successfully, and compiled my code using both cmake and makefile.
BUT, both exe files return  "Segmentation fault"!?
Perhaps, when pthread  comes first, the link is broken?!


5)
Jonathan and Ross suggestion:
          -D BLAS_LIBRARY_DIRS:PATH=$MKL_HOME/lib/em64t \
          -D LAPACK_LIBRARY_DIRS:PATH=$MKL_HOME/lib/em64t \
          -D LAPACK_LIBRARY_NAMES:STRING="mkl_intel_lp64;mkl_intel_thread;mkl_core;iomp5" \
          -D BLAS_LIBRARY_NAMES:STRING="mkl_intel_lp64;mkl_intel_thread;mkl_core;iomp5" \
This one is working.


Thank you,
alireza




----- Original Message -----
From: "Jonathan Hu" <jhu at sandia.gov<mailto:jhu at sandia.gov>>
To: <trilinos-users at software.sandia.gov<mailto:trilinos-users at software.sandia.gov>>
Sent: Friday, December 10, 2010 9:07 AM
Subject: Re: [Trilinos-Users] MKL / ACML

> All,
>
>     Ross' recommendation should work.   I've seen the  "-l-L" when I
> use the Trilinos makefile.export system.  Once I started specifying the
> blas/lapack as is recommended in Trilinos10CMakeQuickstart.txt, this
> error went away.   Please try the following in your cmake script:
>
> \
>   -D
> BLAS_LIBRARY_DIRS:STRING="/usr/local/intel/Compiler/11.1/064/mkl/lib/em64t;/usr/local/intel/Compiler/11.1/064/lib/intel64;/usr/lib64"
> \
>   -D BLAS_LIBRARY_NAMES:STRING="mkl_intel_lp64; mkl_intel_thread;
> mkl_core; iomp5; pthread" \
>   -D
> LAPACK_LIBRARY_DIRS:STRING="/usr/local/intel/Compiler/11.1/064/mkl/lib/em64t;/usr/local/intel/Compiler/11.1/064/lib/intel64;/usr/lib64"
> \
>   -D LAPACK_LIBRARY_NAMES:STRING="mkl_intel_lp64; mkl_intel_thread;
> mkl_core; iomp5; pthread" \
>
>
> I've been using this for day-to-day compilation against Trilinos dev.
>
> Regards,
> Jonathan
>
> trilinos-users-request at software.sandia.gov<mailto:trilinos-users-request at software.sandia.gov> wrote on 12/10/2010 08:40 AM:
>>
>> Message: 2
>> Date: Fri, 10 Dec 2010 09:30:08 -0700
>> From: "Bartlett, Roscoe A"<rabartl at sandia.gov<mailto:rabartl at sandia.gov>>
>> Subject: Re: [Trilinos-Users] MKL / ACML
>> To: "Meyer, Hal"<hxm at sandia.gov<mailto:hxm at sandia.gov>>, Tommy Binford<tbinford at gmail.com<mailto:tbinford at gmail.com>>,
>>          Alireza Nejadmalayeri<Alireza.Nejadmalayeri at colorado.edu<mailto:Alireza.Nejadmalayeri at colorado.edu>>
>> Cc: "trilinos-users at software.sandia.gov<mailto:trilinos-users at software.sandia.gov>"
>>          <trilinos-users at software.sandia.gov<mailto:trilinos-users at software.sandia.gov>>
>> Message-ID:
>>          <9C5EDABC60AD90488D506008E9277E4A20BAA1C457 at ES02SNLNT.srn.sandia.gov<mailto:9C5EDABC60AD90488D506008E9277E4A20BAA1C457 at ES02SNLNT.srn.sandia.gov>>
>> Content-Type: text/plain; charset="iso-8859-1"
>>
>> Hello guys,
>>
>> Have you seen the documentation on setup in TPLs at:
>>
>>     http://trilinos.sandia.gov/Trilinos10CMakeQuickstart.txt
>>
>> ?
>>
>> I think that setting:
>>
>>     TPL_BLAS_LIBRARIES="-L$(DEPEND_INSTALL)/blas -lblas"
>>
>> is not supported in general given the way that CMake works.  Where did you find this approach documented?  Does CMake claim to support this approach in its find_library(...) function?  From looking at the documentation at:
>>
>>      http://www.cmake.org/cmake/help/cmake-2-8-docs.html#command:find_library
>>
>> it does not really mention this at all.
>>
>> Instead I would suggest:
>>
>>     -DTPL_BLAS_LIBRARIES="$(DEPEND_INSTALL)/blas/libblas.a"
>>
>> or whatever the full path to the BLAS library(s) is or:
>>
>>     -DBLAS_LIBRARY_DIRS:PATH=$(DEPEND_INSTALL)/blas
>>
>> Actually, the last approach is really the right way to change the library search path.
>>
>> Please give BLAS_LIBRARY_DIRS a try and see if that solves your problem.  If not, please let us know because it should.  That is really the right way to do this.
>>
>> Cheers,
>>
>> - Ross
>>
>>
>>
>>
>>> -----Original Message-----
>>> From: trilinos-users-bounces at software.sandia.gov<mailto:trilinos-users-bounces at software.sandia.gov> [mailto:trilinos-
>>> users-bounces at software.sandia.gov<mailto:users-bounces at software.sandia.gov>] On Behalf Of Meyer, Hal
>>> Sent: Friday, December 10, 2010 8:34 AM
>>> To: Tommy Binford; Alireza Nejadmalayeri
>>> Cc: trilinos-users at software.sandia.gov<mailto:trilinos-users at software.sandia.gov>
>>> Subject: Re: [Trilinos-Users] MKL / ACML
>>>
>>> I am seeing the same problem on a gnu/linux system where I have built
>>> blas and lapack from source. The error is in Makefile.export.Trilinos.
>>>
>>> The location of my blas/lapack are defined below:
>>>
>>>      BLAS_LIB                := -L$(DEPEND_INSTALL)/blas -lblas
>>>      LAPACK_LIB              := -L$(DEPEND_INSTALL)/lapack -llapack
>>>
>>> The cmake option is set here:
>>>
>>>          cd $(TRILINOS_BUILD_DIR)&&  ($(MODULE_COMMANDS) $(CMAKE) \
>>>                  ......
>>>                  -D TPL_BLAS_LIBRARIES:STRING="$(BLAS_LIB)" \
>>>                  -D TPL_LAPACK_LIBRARIES:STRING="$(LAPACK_LIB)" \
>>>                  $(TRILINOS_SRC_DIR))
>>>
>>> Trilinos builds without error, but the link fails with the following
>>> error.
>>>
>>>      /usr/bin/ld: cannot find -l-
>>> L/scratch1/hxm/aleph/build/tsali/aleph.2472/user/machines/tsali/lapack
>>>
>>> In my Makefile I am including:
>>>
>>>      -include $(TRILINOS_DIR)/include/Makefile.export.Trilinos
>>>
>>> When I look at the include file, the error is in that file.
>>>
>>>      grep -i lapack Makefile.export.*
>>>      Makefile.export.Trilinos:Trilinos_TPL_LIBRARIES=
>>> /scratch1/hxm/aleph/build/tsali/aleph.2472/user/machines/tsali/exodus/l
>>> ib/libnetcdf.a -l-
>>> L/scratch1/hxm/aleph/build/tsali/aleph.2472/user/machines/tsali/lapack
>>> -llapack -l-
>>> L/scratch1/hxm/aleph/build/tsali/aleph.2472/user/machines/tsali/blas -
>>> lblas
>>>
>>> Note the -l-L after libnetcdf.a. This file is created by trilinos.
>>>
>>> The work around I started using was to specify the full path, not "-
>>> L... -l ...", but I would rather use the "-L.. -l.." format.
>>>
>>> I hope this is helpful.
>>> Hal
>>>
>>> _______________________________________
>>> From: trilinos-users-bounces at software.sandia.gov<mailto:trilinos-users-bounces at software.sandia.gov> [trilinos-users-
>>> bounces at software.sandia.gov<mailto:bounces at software.sandia.gov>] On Behalf Of Tommy Binford
>>> [tbinford at gmail.com]
>>> Sent: Friday, December 10, 2010 07:21
>>> To: Alireza Nejadmalayeri
>>> Cc: trilinos-users at software.sandia.gov<mailto:trilinos-users at software.sandia.gov>
>>> Subject: Re: [Trilinos-Users] MKL / ACML
>>>
>>> I'm not sure because I've never seen the -l-L option, but I think you
>>> might have a problem in your makefile prepending a "-l" onto your
>>> library paths.  My guess is ld doesn't know what
>>>
>>> -l-L/opt/intel/mkl/10.2.5.035/lib/em64t
>>>
>>> means.  It should probably be
>>>
>>> -L/opt/intel/mkl/10.2.5.035/lib/em64t
>>>
>>> This also happens on
>>>
>>> -l-L/opt/intel/mkl/10.2.5.035/lib/em64t -lmkl_intel_lp64
>>>
>>> Hope this helps.
>>>
>>> TLB
>>>
>>> On Thu, Dec 9, 2010 at 11:52 PM, Alireza Nejadmalayeri
>>> <Alireza.Nejadmalayeri at colorado.edu<mailto:Alireza.Nejadmalayeri at colorado.edu>>  wrote:
>>>> Thanks, Chris. I have tried something similar. It works as long as I
>>> use
>>>> cmake. But if I compile my code using makefile, I'll get:
>>>>
>>>> /apps/rhel5/mpich2-1.3.1/64/nemesis-intel-11.1.072/bin/mpicxx -c -g -
>>> O0 -g
>>>> -I/usr/rmt_share/scratch95/a/alireza/The_Trilions_Project/trilinos-
>>> 10.6.1__Install_mpi_wrapper_debug_MKL2/include
>>>> -DMYAPP_EPETRA main.cpp
>>>> icpc: command line warning #10120: overriding '-O3' with '-O0'
>>>> /usr/bin/ar cr libmyappLib.a main.o
>>>> /apps/rhel5/mpich2-1.3.1/64/nemesis-intel-11.1.072/bin/mpicxx -g -O0
>>> -g
>>>> -I/usr/rmt_share/scratch95/a/alireza/The_Trilions_Project/trilinos-
>>> 10.6.1__Install_mpi_wrapper_debug_MKL2/include
>>>> -DMYAPP_EPETRA
>>>> -L/usr/rmt_share/scratch95/a/alireza/The_Trilions_Project/trilinos-
>>> 10.6.1__Install_mpi_wrapper_debug_MKL2/lib
>>>> libmyappLib.a -laztecoo -ltpetrainout -ltpetra -ltriutils -lepetra
>>>> -lkokkoslinalg -lkokkosnodeapi -lkokkos -ltpi -lteuchos
>>>> -l-L/opt/intel/mkl/10.2.5.035/lib/em64t -lmkl_intel_lp64 -lmkl_core
>>>> -lmkl_sequential -l-L/opt/intel/mkl/10.2.5.035/lib/em64t -
>>> lmkl_intel_lp64
>>>> -lmkl_core -lmkl_sequential -o MyApp.exe
>>>> icpc: command line warning #10120: overriding '-O3' with '-O0'
>>>> ld: cannot find -l-L/opt/intel/mkl/10.2.5.035/lib/em64t
>>>>
>>>> Any thoughts/suggestions?
>>>>
>>>> Thank you,
>>>> alireza
>>>>
>>>>
>>>> ----- Original Message -----
>>>> From: "Baker, Christopher G."<bakercg at ornl.gov<mailto:bakercg at ornl.gov>>
>>>> To:<Alireza.Nejadmalayeri at colorado.edu<mailto:Alireza.Nejadmalayeri at colorado.edu>>;
>>>> <trilinos-users at software.sandia.gov<mailto:trilinos-users at software.sandia.gov>>
>>>> Sent: Thursday, December 09, 2010 6:16 PM
>>>> Subject: RE: [Trilinos-Users] MKL / ACML
>>>>> The following will link in the sequential BLAS:
>>>>>
>>>>> set SEQMKL="-L${MKLROOT}/lib/em64t -lmkl_intel_lp64 -lmkl_core
>>>>> -lmkl_sequential"
>>>>>
>>>>>
>>>>> cmake ......
>>>>> ? -D
>>> TPL_BLAS_LIBRARIES:STRING="${SEQMKL}"???????????????????????????? \
>>>>> ? -D
>>> TPL_LAPACK_LIBRARIES:STRING="${SEQMKL}"?????????????????????????? \
>>>>>
>>>>>
>>>>> ?-----Original Message-----
>>>>> From: Alireza Nejadmalayeri
>>> [mailto:Alireza.Nejadmalayeri at colorado.edu]
>>>>> Sent: Thursday, December 09, 2010 06:39 PM Eastern Standard Time
>>>>> To: trilinos-users at software.sandia.gov<mailto:trilinos-users at software.sandia.gov>
>>>>> Subject: [Trilinos-Users] MKL / ACML
>>>>>
>>>>> Hello,
>>>>>
>>>>> I wonder if anyone has built Trilinos with  Intel MKL or ACML
>>> libraries
>>>>> (for BLAS and LAPACK). I would like to have a sample configure
>>> script since
>>>>> mine is causing various problems.
>>>>>
>>>>> Thank you,
>>>>> alireza
>>>>>
>>>>>
>>>>>
>>>>>
>>>> _______________________________________________
>
> _______________________________________________
> Trilinos-Users mailing list
> Trilinos-Users at software.sandia.gov<mailto:Trilinos-Users at software.sandia.gov>
> http://software.sandia.gov/mailman/listinfo/trilinos-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://software.sandia.gov/pipermail/trilinos-users/attachments/20101211/e4f1336f/attachment-0001.html 


More information about the Trilinos-Users mailing list