[Trilinos-Users] 64bit indices in Hypre, Superlu-Dist and Metis/Parmetis
Heroux, Michael A
maherou at sandia.gov
Sat Feb 25 10:46:42 EST 2017
John,
I see two options for addressing the situation:
1. Someone (not core Trilinos staff at this time, given other pressing needs) completes the adoption of Epetra64 through the Epetra stack. ML could be omitted from the effort, since Muelu is a suitable substitute and would require less effort to support Epetra64.
2. Anyone who wants to use the Tpetra stack on Windows uses the Intel C++ compiler with Visual Studio, which supports SFINAE features. While not a very attractive solution for some customer bases, it is probably the best option for DOE users.
Is option 2 a possibility for your users?
Mike
On 2/25/17, 8:38 AM, "Trilinos-Users on behalf of JR Cary" <trilinos-users-bounces at trilinos.org on behalf of cary at colorado.edu> wrote:
Unfortunate, Chris, but thanks for the information.
FYI for setting priorities... We do address DOE modeling needs both
by way of Windows for regular computations, and with the need for 64 bit,
large simulations for some design problems. Right now we cannot do the
latter without sacrificing the former.
Best....John
On 2/24/17 2:36 PM, Siefert, Christopher wrote:
> John,
>
> As of VS15, the subset of C++11 supported by Visual Studio was insufficient to build Kokkos on Windows. Ditto for the early 2016 version of the Intel Windows compiler at the same time. No Kokkos means no Tpetra.
>
> Things may have changed in the late 2016/2017 compiler updates, but AFAIK we have not checked recently.
>
> -Chris Siefert
>
>
> ________________________________________
> From: Trilinos-Users <trilinos-users-bounces at trilinos.org> on behalf of JR Cary <cary at colorado.edu>
> Sent: Friday, February 24, 2017 11:05 AM
> To: trilinos-users at trilinos.org
> Cc: Heroux, Michael A
> Subject: [EXTERNAL] Re: [Trilinos-Users] 64bit indices in Hypre, Superlu-Dist and Metis/Parmetis
>
> Hi Mike,
>
> So then another question -- to move to the TPetra stack, I have
> heard that I need to build Kokkos, because TPetra depends on Kokkos,
> but that Kokkos is not building on Windows. If all this is true,
> then I would not be able to build on Windows with the same
> stack, I guess. Is that true? Or is there a workaround?
>
> (I understand that int is 64 bits on 64 bit Windows, but maintaining
> two build stacks would be a hassle.)
>
> This is important to us because we provide our software to DOE users
> at, e.g., the US Particle Accelerator School, and they all have Windows
> machines.
>
> Thanks......John
>
>
>
> On 2/23/17 12:29 PM, Heroux, Michael A wrote:
>> John, others,
>>
>> There have been many discussions over the years about how to support 64-bit integers in Trilinos. Text substitution, while common in the scientific software ecosystem, is not a good way (IMO) and leads to downstream integration issues with large software bases. Suppose one component wants to use the 32-bit instance of a library and another wants to use 64-bits. There is no way to reconcile this if you can only build once instance of the library at a time.
>>
>> This approach is often used to support different numerical types as well. The same issue arises: You can’t have both a double precision real and double precision complex version available at the same time.
>>
>> There is a 64-bit Epetra API (Epetra64) along with AztecOO and Ifpack. It does not disrupt the 32-bit interface, but augments it. Also, while the global indices are 64-bit, the local indices are still 32-bit. This limits the local problem size (under each MPI process) to by 2B, but preserves sparse kernel performance, which is significantly impacted by the extra 4*nz bytes needed by 64-bit ints vs 32-bit. Epetra64 has distinct function prototypes, but is supported underneath by templated implementations to reduce code redundancy.
>>
>> We never got to doing ML. Muelu support for Epetra64 never got done either. This is because, as we were progressing with 64-bit support through the Epetra stack, the need for it diminished. Tpetra was becoming more production-capable and the supporting pieces (Muelu, Belos, Ifpack2, Anasazi, etc.) were in place.
>>
>> For full 64-bit integer support in Trilinos, the Tpetra stack is the best option. It is also the best option for 16-bit integers, or other ordinal types. Similarly, it readily supports 32-bit real and complex and extended types. Funky types may not work out of the gate, but there are some fun ways to play with the template scalar types. We are just discovering in Kokkos, which adopt the Tpetra approach for scalar type support, that templating is very nice for instantiating vector data types, removing ambiguity about address boundaries and whether loops are vectorizable or not.
>>
>> Unfortunately, templates bring build-time complexity, which has been a real headache, no doubt. At the same time, we are able to instantiate any and all types simultaneously without ambiguity. This is very valuable.
>>
>> There is a fair bit of functionality in the Epetra stack that supports 64-bit ints, and there is only a modest amount of effort to make Muelu support Epetra64. Even so, we decided some time ago to avoid further development of Epetra64 capabilities, since we prefer that people make the transition to the Tpetra stack.
>>
>> Thanks.
>>
>> Mike
>>
>> On 2/23/17, 10:54 AM, "Trilinos-Users on behalf of JR Cary" <trilinos-users-bounces at trilinos.org on behalf of cary at colorado.edu> wrote:
>>
>> Hi Andrey,
>>
>> So because of the 2e9 problem, we need to embark on the int -> ssize_t
>> migration. You guys did it through templating over the integer type.
>> Other approaching could include having a define or simply a global
>> replacement.
>>
>> Can you share how you came to decide on templating?
>>
>> Thx....John
>>
>> On 2/23/2017 8:21 AM, Prokopenko, Andrey V. wrote:
>> > There was an effort to allow Epetra with 64-bit indices. So,
>> > technically most Epetra classes are now templated on the index type.
>> >
>> > I think the conversion mainly touched widely used classes. I don't
>> > think it has 100% coverage. Looking at the code Denis mentioned, some
>> > of the required classes there were not converted, and thus are not
>> > compatible. I am not certain of an amount of effort required to do that.
>> >
>> > -Andrey
>> >
>> > On 02/23/2017 08:17 AM, Klinvex, Alicia Marie wrote:
>> >> Hello Denis,
>> >>
>> >> Since Epetra isn't templated, I'm not sure I see a way to support
>> >> 64bit ints in that hypre interface. (CCed Mark in case he has some
>> >> ideas.) We should be able to support it in the tpetra-based
>> >> interface though, if you're interested in that one.
>> >>
>> >> Best wishes,
>> >> Alicia
>> >>
>> >> -----Original Message-----
>> >> From: Trilinos-Users [mailto:trilinos-users-bounces at trilinos.org] On
>> >> Behalf Of Denis Davydov
>> >> Sent: Thursday, February 23, 2017 5:14 AM
>> >> To: trilinos-users <trilinos-users at trilinos.org>
>> >> Subject: [EXTERNAL] [Trilinos-Users] 64bit indices in Hypre,
>> >> Superlu-Dist and Metis/Parmetis
>> >>
>> >> Dear all,
>> >>
>> >> I am building a stack (with 64bit indices) composed of PETSc,
>> >> Trilinos, Superlu-Dist, Metis, etc and came across the following
>> >> compiler error in EpetraExt:
>> >>
>> >> /home/user/spack/var/spack/stage/trilinos-12.10.1-kgdtcmcwce6xhetukputhnh2f7v4sljf/Trilinos-trilinos-release-12-10-1/packages/epetraext/src/hypre/EpetraExt_HypreIJMatrix.cpp:91:96:
>> >> error: cannot convert ‘int*’ to ‘HYPRE_Int* {aka long long int*}’ for
>> >> argument ‘3’ to ‘HYPRE_Int
>> >> HYPRE_ParCSRMatrixGetRow(HYPRE_ParCSRMatrix, HYPRE_Int, HYPRE_Int*,
>> >> HYPRE_Int**, HYPRE_Complex**)’
>> >> ierr += HYPRE_ParCSRMatrixGetRow(ParMatrix_, i+MyRowStart_,
>> >> &num_entries, &indices, &values);
>> >> ^
>> >>
>> >> That’s of course because if PETSc is built with 64bit ints, then
>> >> Hypre have to be also built accordingly.
>> >>
>> >> So what are the requirements from Trilinos side on those
>> >> dependencies? Can it not be built against Hypre or Superlu-Dist when
>> >> those packages are configured with —enable-biginit and -D_LONGINT,
>> >> respectively?
>> >> Or is it a bug in EpetraExt?
>> >>
>> >> Regards,
>> >> Denis.
>> >>
>> >> _______________________________________________
>> >> Trilinos-Users mailing list
>> >> Trilinos-Users at trilinos.org
>> >> https://trilinos.org/mailman/listinfo/trilinos-users
>> >> _______________________________________________
>> >> Trilinos-Users mailing list
>> >> Trilinos-Users at trilinos.org
>> >> https://trilinos.org/mailman/listinfo/trilinos-users
>> >
>> > _______________________________________________
>> > Trilinos-Users mailing list
>> > Trilinos-Users at trilinos.org
>> > https://trilinos.org/mailman/listinfo/trilinos-users
>> >
>>
>> _______________________________________________
>> Trilinos-Users mailing list
>> Trilinos-Users at trilinos.org
>> https://trilinos.org/mailman/listinfo/trilinos-users
>>
>>
>>
> _______________________________________________
> Trilinos-Users mailing list
> Trilinos-Users at trilinos.org
> https://trilinos.org/mailman/listinfo/trilinos-users
>
_______________________________________________
Trilinos-Users mailing list
Trilinos-Users at trilinos.org
https://trilinos.org/mailman/listinfo/trilinos-users
More information about the Trilinos-Users
mailing list