[Trilinos-Users] 64bit indices in Hypre, Superlu-Dist and Metis/Parmetis
Heroux, Michael A
maherou at sandia.gov
Sat Feb 25 10:58:22 EST 2017
Rich,
This should be filed as a GitHub issue with some details. Who can we contact for details?
Thanks for mentioning this.
Mike
On 2/25/17, 9:55 AM, "Drake, Richard R" <rrdrake at sandia.gov> wrote:
FYI, currently Muelu won't build on Windows with *any* compiler due to
binary format limits (number of symbols in a translation unit, or
something). At least using the configure options needed by Alegra. I
think the offending file is a template factory type thing.
-rich
-----Original Message-----
From: Trilinos-Users <trilinos-users-bounces at trilinos.org> on behalf of
"Heroux, Michael A" <maherou at sandia.gov>
Date: Saturday, February 25, 2017 at 8:46 AM
To: JR Cary <cary at colorado.edu>, "trilinos-users at trilinos.org"
<trilinos-users at trilinos.org>
Subject: [EXTERNAL] Re: [Trilinos-Users] 64bit indices in Hypre,
Superlu-Dist and Metis/Parmetis
John,
I see two options for addressing the situation:
1. Someone (not core Trilinos staff at this time, given other pressing
needs) completes the adoption of Epetra64 through the Epetra stack. ML
could be omitted from the effort, since Muelu is a suitable substitute and
would require less effort to support Epetra64.
2. Anyone who wants to use the Tpetra stack on Windows uses the Intel C++
compiler with Visual Studio, which supports SFINAE features. While not a
very attractive solution for some customer bases, it is probably the best
option for DOE users.
Is option 2 a possibility for your users?
Mike
On 2/25/17, 8:38 AM, "Trilinos-Users on behalf of JR Cary"
<trilinos-users-bounces at trilinos.org on behalf of cary at colorado.edu> wrote:
Unfortunate, Chris, but thanks for the information.
FYI for setting priorities... We do address DOE modeling needs both
by way of Windows for regular computations, and with the need for 64
bit,
large simulations for some design problems. Right now we cannot do the
latter without sacrificing the former.
Best....John
On 2/24/17 2:36 PM, Siefert, Christopher wrote:
> John,
>
> As of VS15, the subset of C++11 supported by Visual Studio was
insufficient to build Kokkos on Windows. Ditto for the early 2016 version
of the Intel Windows compiler at the same time. No Kokkos means no Tpetra.
>
> Things may have changed in the late 2016/2017 compiler updates, but
AFAIK we have not checked recently.
>
> -Chris Siefert
>
>
> ________________________________________
> From: Trilinos-Users <trilinos-users-bounces at trilinos.org> on behalf
of JR Cary <cary at colorado.edu>
> Sent: Friday, February 24, 2017 11:05 AM
> To: trilinos-users at trilinos.org
> Cc: Heroux, Michael A
> Subject: [EXTERNAL] Re: [Trilinos-Users] 64bit indices in Hypre,
Superlu-Dist and Metis/Parmetis
>
> Hi Mike,
>
> So then another question -- to move to the TPetra stack, I have
> heard that I need to build Kokkos, because TPetra depends on Kokkos,
> but that Kokkos is not building on Windows. If all this is true,
> then I would not be able to build on Windows with the same
> stack, I guess. Is that true? Or is there a workaround?
>
> (I understand that int is 64 bits on 64 bit Windows, but maintaining
> two build stacks would be a hassle.)
>
> This is important to us because we provide our software to DOE users
> at, e.g., the US Particle Accelerator School, and they all have
Windows
> machines.
>
> Thanks......John
>
>
>
> On 2/23/17 12:29 PM, Heroux, Michael A wrote:
>> John, others,
>>
>> There have been many discussions over the years about how to
support 64-bit integers in Trilinos. Text substitution, while common in
the scientific software ecosystem, is not a good way (IMO) and leads to
downstream integration issues with large software bases. Suppose one
component wants to use the 32-bit instance of a library and another wants
to use 64-bits. There is no way to reconcile this if you can only build
once instance of the library at a time.
>>
>> This approach is often used to support different numerical types as
well. The same issue arises: You can¹t have both a double precision real
and double precision complex version available at the same time.
>>
>> There is a 64-bit Epetra API (Epetra64) along with AztecOO and
Ifpack. It does not disrupt the 32-bit interface, but augments it. Also,
while the global indices are 64-bit, the local indices are still 32-bit.
This limits the local problem size (under each MPI process) to by 2B, but
preserves sparse kernel performance, which is significantly impacted by
the extra 4*nz bytes needed by 64-bit ints vs 32-bit. Epetra64 has
distinct function prototypes, but is supported underneath by templated
implementations to reduce code redundancy.
>>
>> We never got to doing ML. Muelu support for Epetra64 never got
done either. This is because, as we were progressing with 64-bit support
through the Epetra stack, the need for it diminished. Tpetra was becoming
more production-capable and the supporting pieces (Muelu, Belos, Ifpack2,
Anasazi, etc.) were in place.
>>
>> For full 64-bit integer support in Trilinos, the Tpetra stack is
the best option. It is also the best option for 16-bit integers, or other
ordinal types. Similarly, it readily supports 32-bit real and complex and
extended types. Funky types may not work out of the gate, but there are
some fun ways to play with the template scalar types. We are just
discovering in Kokkos, which adopt the Tpetra approach for scalar type
support, that templating is very nice for instantiating vector data types,
removing ambiguity about address boundaries and whether loops are
vectorizable or not.
>>
>> Unfortunately, templates bring build-time complexity, which has
been a real headache, no doubt. At the same time, we are able to
instantiate any and all types simultaneously without ambiguity. This is
very valuable.
>>
>> There is a fair bit of functionality in the Epetra stack that
supports 64-bit ints, and there is only a modest amount of effort to make
Muelu support Epetra64. Even so, we decided some time ago to avoid
further development of Epetra64 capabilities, since we prefer that people
make the transition to the Tpetra stack.
>>
>> Thanks.
>>
>> Mike
>>
>> On 2/23/17, 10:54 AM, "Trilinos-Users on behalf of JR Cary"
<trilinos-users-bounces at trilinos.org on behalf of cary at colorado.edu> wrote:
>>
>> Hi Andrey,
>>
>> So because of the 2e9 problem, we need to embark on the int ->
ssize_t
>> migration. You guys did it through templating over the integer
type.
>> Other approaching could include having a define or simply a
global
>> replacement.
>>
>> Can you share how you came to decide on templating?
>>
>> Thx....John
>>
>> On 2/23/2017 8:21 AM, Prokopenko, Andrey V. wrote:
>> > There was an effort to allow Epetra with 64-bit indices. So,
>> > technically most Epetra classes are now templated on the
index type.
>> >
>> > I think the conversion mainly touched widely used classes. I
don't
>> > think it has 100% coverage. Looking at the code Denis
mentioned, some
>> > of the required classes there were not converted, and thus
are not
>> > compatible. I am not certain of an amount of effort required
to do that.
>> >
>> > -Andrey
>> >
>> > On 02/23/2017 08:17 AM, Klinvex, Alicia Marie wrote:
>> >> Hello Denis,
>> >>
>> >> Since Epetra isn't templated, I'm not sure I see a way to
support
>> >> 64bit ints in that hypre interface. (CCed Mark in case he
has some
>> >> ideas.) We should be able to support it in the tpetra-based
>> >> interface though, if you're interested in that one.
>> >>
>> >> Best wishes,
>> >> Alicia
>> >>
>> >> -----Original Message-----
>> >> From: Trilinos-Users
[mailto:trilinos-users-bounces at trilinos.org] On
>> >> Behalf Of Denis Davydov
>> >> Sent: Thursday, February 23, 2017 5:14 AM
>> >> To: trilinos-users <trilinos-users at trilinos.org>
>> >> Subject: [EXTERNAL] [Trilinos-Users] 64bit indices in Hypre,
>> >> Superlu-Dist and Metis/Parmetis
>> >>
>> >> Dear all,
>> >>
>> >> I am building a stack (with 64bit indices) composed of PETSc,
>> >> Trilinos, Superlu-Dist, Metis, etc and came across the
following
>> >> compiler error in EpetraExt:
>> >>
>> >>
/home/user/spack/var/spack/stage/trilinos-12.10.1-kgdtcmcwce6xhetukputhnh2f
7v4sljf/Trilinos-trilinos-release-12-10-1/packages/epetraext/src/hypre/Epet
raExt_HypreIJMatrix.cpp:91:96:
>> >> error: cannot convert Œint*¹ to ŒHYPRE_Int* {aka long long
int*}¹ for
>> >> argument Œ3¹ to ŒHYPRE_Int
>> >> HYPRE_ParCSRMatrixGetRow(HYPRE_ParCSRMatrix, HYPRE_Int,
HYPRE_Int*,
>> >> HYPRE_Int**, HYPRE_Complex**)¹
>> >> ierr += HYPRE_ParCSRMatrixGetRow(ParMatrix_,
i+MyRowStart_,
>> >> &num_entries, &indices, &values);
>> >> ^
>> >>
>> >> That¹s of course because if PETSc is built with 64bit ints,
then
>> >> Hypre have to be also built accordingly.
>> >>
>> >> So what are the requirements from Trilinos side on those
>> >> dependencies? Can it not be built against Hypre or
Superlu-Dist when
>> >> those packages are configured with ‹enable-biginit and
-D_LONGINT,
>> >> respectively?
>> >> Or is it a bug in EpetraExt?
>> >>
>> >> Regards,
>> >> Denis.
>> >>
>> >> _______________________________________________
>> >> Trilinos-Users mailing list
>> >> Trilinos-Users at trilinos.org
>> >> https://trilinos.org/mailman/listinfo/trilinos-users
>> >> _______________________________________________
>> >> Trilinos-Users mailing list
>> >> Trilinos-Users at trilinos.org
>> >> https://trilinos.org/mailman/listinfo/trilinos-users
>> >
>> > _______________________________________________
>> > Trilinos-Users mailing list
>> > Trilinos-Users at trilinos.org
>> > https://trilinos.org/mailman/listinfo/trilinos-users
>> >
>>
>> _______________________________________________
>> Trilinos-Users mailing list
>> Trilinos-Users at trilinos.org
>> https://trilinos.org/mailman/listinfo/trilinos-users
>>
>>
>>
> _______________________________________________
> Trilinos-Users mailing list
> Trilinos-Users at trilinos.org
> https://trilinos.org/mailman/listinfo/trilinos-users
>
_______________________________________________
Trilinos-Users mailing list
Trilinos-Users at trilinos.org
https://trilinos.org/mailman/listinfo/trilinos-users
_______________________________________________
Trilinos-Users mailing list
Trilinos-Users at trilinos.org
https://trilinos.org/mailman/listinfo/trilinos-users
More information about the Trilinos-Users
mailing list