[Trilinos-Users] 64bit indices in Hypre, Superlu-Dist and Metis/Parmetis
JR Cary
cary at colorado.edu
Fri Feb 24 13:05:11 EST 2017
Hi Mike,
So then another question -- to move to the TPetra stack, I have
heard that I need to build Kokkos, because TPetra depends on Kokkos,
but that Kokkos is not building on Windows. If all this is true,
then I would not be able to build on Windows with the same
stack, I guess. Is that true? Or is there a workaround?
(I understand that int is 64 bits on 64 bit Windows, but maintaining
two build stacks would be a hassle.)
This is important to us because we provide our software to DOE users
at, e.g., the US Particle Accelerator School, and they all have Windows
machines.
Thanks......John
On 2/23/17 12:29 PM, Heroux, Michael A wrote:
> John, others,
>
> There have been many discussions over the years about how to support 64-bit integers in Trilinos. Text substitution, while common in the scientific software ecosystem, is not a good way (IMO) and leads to downstream integration issues with large software bases. Suppose one component wants to use the 32-bit instance of a library and another wants to use 64-bits. There is no way to reconcile this if you can only build once instance of the library at a time.
>
> This approach is often used to support different numerical types as well. The same issue arises: You can’t have both a double precision real and double precision complex version available at the same time.
>
> There is a 64-bit Epetra API (Epetra64) along with AztecOO and Ifpack. It does not disrupt the 32-bit interface, but augments it. Also, while the global indices are 64-bit, the local indices are still 32-bit. This limits the local problem size (under each MPI process) to by 2B, but preserves sparse kernel performance, which is significantly impacted by the extra 4*nz bytes needed by 64-bit ints vs 32-bit. Epetra64 has distinct function prototypes, but is supported underneath by templated implementations to reduce code redundancy.
>
> We never got to doing ML. Muelu support for Epetra64 never got done either. This is because, as we were progressing with 64-bit support through the Epetra stack, the need for it diminished. Tpetra was becoming more production-capable and the supporting pieces (Muelu, Belos, Ifpack2, Anasazi, etc.) were in place.
>
> For full 64-bit integer support in Trilinos, the Tpetra stack is the best option. It is also the best option for 16-bit integers, or other ordinal types. Similarly, it readily supports 32-bit real and complex and extended types. Funky types may not work out of the gate, but there are some fun ways to play with the template scalar types. We are just discovering in Kokkos, which adopt the Tpetra approach for scalar type support, that templating is very nice for instantiating vector data types, removing ambiguity about address boundaries and whether loops are vectorizable or not.
>
> Unfortunately, templates bring build-time complexity, which has been a real headache, no doubt. At the same time, we are able to instantiate any and all types simultaneously without ambiguity. This is very valuable.
>
> There is a fair bit of functionality in the Epetra stack that supports 64-bit ints, and there is only a modest amount of effort to make Muelu support Epetra64. Even so, we decided some time ago to avoid further development of Epetra64 capabilities, since we prefer that people make the transition to the Tpetra stack.
>
> Thanks.
>
> Mike
>
> On 2/23/17, 10:54 AM, "Trilinos-Users on behalf of JR Cary" <trilinos-users-bounces at trilinos.org on behalf of cary at colorado.edu> wrote:
>
> Hi Andrey,
>
> So because of the 2e9 problem, we need to embark on the int -> ssize_t
> migration. You guys did it through templating over the integer type.
> Other approaching could include having a define or simply a global
> replacement.
>
> Can you share how you came to decide on templating?
>
> Thx....John
>
> On 2/23/2017 8:21 AM, Prokopenko, Andrey V. wrote:
> > There was an effort to allow Epetra with 64-bit indices. So,
> > technically most Epetra classes are now templated on the index type.
> >
> > I think the conversion mainly touched widely used classes. I don't
> > think it has 100% coverage. Looking at the code Denis mentioned, some
> > of the required classes there were not converted, and thus are not
> > compatible. I am not certain of an amount of effort required to do that.
> >
> > -Andrey
> >
> > On 02/23/2017 08:17 AM, Klinvex, Alicia Marie wrote:
> >> Hello Denis,
> >>
> >> Since Epetra isn't templated, I'm not sure I see a way to support
> >> 64bit ints in that hypre interface. (CCed Mark in case he has some
> >> ideas.) We should be able to support it in the tpetra-based
> >> interface though, if you're interested in that one.
> >>
> >> Best wishes,
> >> Alicia
> >>
> >> -----Original Message-----
> >> From: Trilinos-Users [mailto:trilinos-users-bounces at trilinos.org] On
> >> Behalf Of Denis Davydov
> >> Sent: Thursday, February 23, 2017 5:14 AM
> >> To: trilinos-users <trilinos-users at trilinos.org>
> >> Subject: [EXTERNAL] [Trilinos-Users] 64bit indices in Hypre,
> >> Superlu-Dist and Metis/Parmetis
> >>
> >> Dear all,
> >>
> >> I am building a stack (with 64bit indices) composed of PETSc,
> >> Trilinos, Superlu-Dist, Metis, etc and came across the following
> >> compiler error in EpetraExt:
> >>
> >> /home/user/spack/var/spack/stage/trilinos-12.10.1-kgdtcmcwce6xhetukputhnh2f7v4sljf/Trilinos-trilinos-release-12-10-1/packages/epetraext/src/hypre/EpetraExt_HypreIJMatrix.cpp:91:96:
> >> error: cannot convert ‘int*’ to ‘HYPRE_Int* {aka long long int*}’ for
> >> argument ‘3’ to ‘HYPRE_Int
> >> HYPRE_ParCSRMatrixGetRow(HYPRE_ParCSRMatrix, HYPRE_Int, HYPRE_Int*,
> >> HYPRE_Int**, HYPRE_Complex**)’
> >> ierr += HYPRE_ParCSRMatrixGetRow(ParMatrix_, i+MyRowStart_,
> >> &num_entries, &indices, &values);
> >> ^
> >>
> >> That’s of course because if PETSc is built with 64bit ints, then
> >> Hypre have to be also built accordingly.
> >>
> >> So what are the requirements from Trilinos side on those
> >> dependencies? Can it not be built against Hypre or Superlu-Dist when
> >> those packages are configured with —enable-biginit and -D_LONGINT,
> >> respectively?
> >> Or is it a bug in EpetraExt?
> >>
> >> Regards,
> >> Denis.
> >>
> >> _______________________________________________
> >> Trilinos-Users mailing list
> >> Trilinos-Users at trilinos.org
> >> https://trilinos.org/mailman/listinfo/trilinos-users
> >> _______________________________________________
> >> Trilinos-Users mailing list
> >> Trilinos-Users at trilinos.org
> >> https://trilinos.org/mailman/listinfo/trilinos-users
> >
> > _______________________________________________
> > Trilinos-Users mailing list
> > Trilinos-Users at trilinos.org
> > https://trilinos.org/mailman/listinfo/trilinos-users
> >
>
> _______________________________________________
> Trilinos-Users mailing list
> Trilinos-Users at trilinos.org
> https://trilinos.org/mailman/listinfo/trilinos-users
>
>
>
More information about the Trilinos-Users
mailing list