[Trilinos-Users] Segmentation fault in aztecoo with the Intel MKL ilp64 library

Heroux, Michael A maherou at sandia.gov
Mon Jun 22 13:52:20 MST 2020


If you do need to continue use of the Epetra stack, I think the changes to support ILP64 are modest.

Since BLAS are called by Epetra using local indices, we call the 32-bit BLAS/LAPACK interfaces.  However, all BLAS and LAPACK calls are wrapped and there is a single instance of any BLAS/LAPACK function call.  All of these calls are two small files in Epetra.  To use ILP64, the integers passed by Epetra could be promoted to 64 bit just prior to the function call.
In this code from Epetra_BLAS.cpp, here is the only call to DGEMM in all of Epetra.  A person could insert a bit of code in this function BLAS::GEMM() to promote the integer arguments to from LP64 to ILP64.  Then one could link to the ILP64 BLAS.  I think this is true, but I have not done any real coding work on Trilinos for a long time.
void Epetra_BLAS::GEMM(const char TRANSA, const char TRANSB, const int M, const int N, const int K,
            const double ALPHA, const double * A, const int LDA, const double * B,
            const int LDB, const double BETA, double * C, const int LDC) const {
         A, &LDA, B, &LDB, &BETA, C, &LDC);

On 6/22/20, 10:27 AM, "Trilinos-Users on behalf of John Jomo" <trilinos-users-bounces at trilinos.org on behalf of john.jomo at tum.de> wrote:

    Hallo Siva,

    that would explain the segfault. Thanks for the quick reply. I have 
    implemented most of my code using the old Trilinos stack and plan to 
    switch to the newer stack soon. I now have one more reason to do so.


    On 22.06.20 16:21, Rajamanickam, Sivasankaran wrote:
    > John
    >    AztecOO does not support 64-but ordinals. Can you switch to Belos / Tpetra software stack ?
    > -Siva
    > ________________________________________
    > From: Trilinos-Users <trilinos-users-bounces at trilinos.org> on behalf of John Jomo <john.jomo at tum.de>
    > Sent: Monday, June 22, 2020 7:41 AM
    > To: trilinos-users at trilinos.org
    > Subject: [EXTERNAL] [Trilinos-Users] Segmentation fault in aztecoo with the Intel MKL ilp64 library
    > Hallo guys,
    > I am trying to get the AztecOO solvers to run with the MKL ilp64 library
    > in intel 2019.  I however get a segfault in the function
    > mkl_blas_avx2_xddot () when running my code. The same segfault also
    > occurs when running the tests in the AztecOO package.  I used the Intel
    > Link line advisor to get the optimal compiler and linker options for my
    > machine e.g.
    > linker options : -L${MKLROOT}/lib/intel64 -Wl,--no-as-needed
    > -lmkl_intel_ilp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lm -ldl
    > compiler flag: -DMKL_ILP64 -m64 -I${MKLROOT}/include
    > I also compiled and run the code on two different architectures
    > (Haswell, Skylake) and using gcc/7.3 and icpc 2019 and the segfault
    > still persists.
    > I explicity set the blas library names in my configure file for Trilinos
    > e.g.\
    > -D
    > BLAS_LIBRARY_NAMES="mkl_intel_lp64;mkl_intel_thread;iomp5;pthread;mkl_core"
    > \
    > ...
    > Are there any open issues with the ilp64 library and AztecOO? Changing
    > to the lp64 MKL version gets rid of the segfault but slows down other
    > parts of my code that rely on MKL and pardiso.
    > Thanks for the help
    > John
    > _______________________________________________
    > Trilinos-Users mailing list
    > Trilinos-Users at trilinos.org
    > http://trilinos.org/mailman/listinfo/trilinos-users_trilinos.org

    John Jomo M.Sc.
    Technische Universität München
    Computation in Engineering
    Simulation in Applied Mechanics - SAM
    Arcisstraße 21
    80333 München
    Tel.:     0049 / 89 / 289 25064
    Fax:      0049 / 89 / 289 25051
    E-Mail:   john.jomo at tum.de<mailto:john.jomo at tum.de>
    Internet: www.cie.bgu.tum.de<http://www.cie.bgu.tum.de/>

    Trilinos-Users mailing list
    Trilinos-Users at trilinos.org

More information about the Trilinos-Users mailing list