[Trilinos-Users] Segmentation fault in aztecoo with the Intel MKL ilp64 library

Heroux, Mike MHeroux at CSBSJU.EDU
Mon Jun 22 18:55:05 MST 2020


John,

If you do need to continue use of the Epetra stack, I think the changes to support ILP64 are modest.

Since BLAS are called by Epetra using local indices, we call the 32-bit BLAS/LAPACK interfaces.  However, all BLAS and LAPACK calls are wrapped and there is a single instance of any BLAS/LAPACK function call.  All of these calls are two small files in Epetra.  To use ILP64, the integers passed by Epetra could be promoted to 64 bit just prior to the function call.

In this code from Epetra_BLAS.cpp, here is the only call to DGEMM in all of Epetra.  A person could insert a bit of code in this function BLAS::GEMM() to promote the integer arguments to from LP64 to ILP64.  Then one could link to the ILP64 BLAS.  I think this is true, but I have not done any real coding work on Trilinos for a long time.

//=============================================================================
void Epetra_BLAS::GEMM(const char TRANSA, const char TRANSB, const int M, const int N, const int K,
           const double ALPHA, const double * A, const int LDA, const double * B,
           const int LDB, const double BETA, double * C, const int LDC) const {

 DGEMM_F77(CHAR_MACRO(TRANSA), CHAR_MACRO(TRANSB), &M, &N, &K, &ALPHA,
        A, &LDA, B, &LDB, &BETA, C, &LDC);
}

Mike

On 6/22/20, 10:27 AM, "Trilinos-Users on behalf of John Jomo" <trilinos-users-bounces at trilinos.org on behalf of john.jomo at tum.de> wrote:

   Hallo Siva,

   that would explain the segfault. Thanks for the quick reply. I have 
   implemented most of my code using the old Trilinos stack and plan to 
   switch to the newer stack soon. I now have one more reason to do so.

   John

>    On 22.06.20 16:21, Rajamanickam, Sivasankaran wrote:
> John
>   AztecOO does not support 64-but ordinals. Can you switch to Belos / Tpetra software stack ?
> 
> -Siva
> 
> ________________________________________
> From: Trilinos-Users <trilinos-users-bounces at trilinos.org> on behalf of John Jomo <john.jomo at tum.de>
> Sent: Monday, June 22, 2020 7:41 AM
> To: trilinos-users at trilinos.org
> Subject: [EXTERNAL] [Trilinos-Users] Segmentation fault in aztecoo with the Intel MKL ilp64 library
> 
> Hallo guys,
> 
> I am trying to get the AztecOO solvers to run with the MKL ilp64 library
> in intel 2019.  I however get a segfault in the function
> mkl_blas_avx2_xddot () when running my code. The same segfault also
> occurs when running the tests in the AztecOO package.  I used the Intel
> Link line advisor to get the optimal compiler and linker options for my
> machine e.g.
> 
> linker options : -L${MKLROOT}/lib/intel64 -Wl,--no-as-needed
> -lmkl_intel_ilp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lm -ldl
> compiler flag: -DMKL_ILP64 -m64 -I${MKLROOT}/include
> 
> I also compiled and run the code on two different architectures
> (Haswell, Skylake) and using gcc/7.3 and icpc 2019 and the segfault
> still persists.
> 
> I explicity set the blas library names in my configure file for Trilinos
> e.g.\
> 
> -D TPL_ENABLE_BLAS:BOOL=ON \
> -D
> BLAS_LIBRARY_NAMES="mkl_intel_lp64;mkl_intel_thread;iomp5;pthread;mkl_core"
> \
> ...
> 
> 
> Are there any open issues with the ilp64 library and AztecOO? Changing
> to the lp64 MKL version gets rid of the segfault but slows down other
> parts of my code that rely on MKL and pardiso.
> 
> Thanks for the help
> 
> John
> 
> 
> _______________________________________________
> Trilinos-Users mailing list
> Trilinos-Users at trilinos.org
> http://trilinos.org/mailman/listinfo/trilinos-users_trilinos.org

   -- 
   John Jomo M.Sc.
   Technische Universität München
   Computation in Engineering
   Simulation in Applied Mechanics - SAM
   Arcisstraße 21
   80333 München
   Tel.:     0049 / 89 / 289 25064
   Fax:      0049 / 89 / 289 25051
   E-Mail:   john.jomo at tum.de<mailto:john.jomo at tum.de>
   Internet: www.cie.bgu.tum.de<http://www.cie.bgu.tum.de/>


   _______________________________________________
   Trilinos-Users mailing list
   Trilinos-Users at trilinos.org
   http://trilinos.org/mailman/listinfo/trilinos-users_trilinos.org

_______________________________________________
Trilinos-Users mailing list
Trilinos-Users at trilinos.org
http://trilinos.org/mailman/listinfo/trilinos-users_trilinos.org


More information about the Trilinos-Users mailing list