[Trilinos-Users] Tpetra - GPU

[Frey Matthias (PSI)]

Hi

I've developed a solver with Tpetra which runs smoothly on CPUs. However with GPUs, I experience sometimes error messages like

what():  cudaDeviceSynchronize() error( cudaErrorIllegalAddress): an illegal memory access was encountered /scratch/snx3000/freym/gpu/Trilinos-trilinos-release-12-14-1/packages/kokkos/core/src/Cuda/Kokkos_Cuda_Impl.cpp:119
Traceback functionality not available

Sometimes it even gets stuck. I use Trilinos release 12.14.1. Within my batch script I set the environment variables:

export CRAY_CUDA_MPS=1
export CUDA_LAUNCH_BLOCKING=1

Trilinos is compiled with following CMAKE - Tpetra and Kokkos - CUDA - configs:

-DTpetra_ASSUME_CUDA_AWARE_MPI=OFF \
-DKokkos_ENABLE_Cuda=ON \
-DKokkos_ENABLE_Cuda_UVM=ON \
-DKokkos_ENABLE_Cuda_Lambda=ON \
-DTPL_ENABLE_CUDA=ON \
-DTPL_ENABLE_CUSPARSE=ON

Is this a real bug or do I need to run my code differently? I.e. exporting/setting another environment variable etc?

The packages I use are: MueLu, Amesos2, Iffpack2, Tpetra, Zoltan2, Belos.


Thanks for your help.

Best,
Matthias

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://trilinos.org/pipermail/trilinos-users/attachments/20190902/6e5cc872/attachment.html>