[Trilinos-Users] [EXTERNAL] Re: 64bit indices in Hypre, Superlu-Dist and Metis/Parmetis
JR Cary
cary at colorado.edu
Sat Feb 25 12:06:57 EST 2017
Hi Andrey,
I probably don't know enough about this, but our users want us just
to deliver an app, not to have them install a container with our app
in it. But again, perhaps this is easier than it sounds?
Thx....John
On 2/25/17 9:56 AM, Prokopenko, Andrey V. wrote:
> John,
>
> Have you considered a container-based solutions, like Docker or
> Singularity?
>
> -Andrey
>
> On 2/25/17 9:38 AM, JR Cary wrote:
>> Unfortunate, Chris, but thanks for the information.
>>
>> FYI for setting priorities... We do address DOE modeling needs both
>> by way of Windows for regular computations, and with the need for 64
>> bit,
>> large simulations for some design problems. Right now we cannot do the
>> latter without sacrificing the former.
>>
>> Best....John
>>
>>
>> On 2/24/17 2:36 PM, Siefert, Christopher wrote:
>>> John,
>>>
>>> As of VS15, the subset of C++11 supported by Visual Studio was
>>> insufficient to build Kokkos on Windows. Ditto for the early 2016
>>> version of the Intel Windows compiler at the same time. No Kokkos
>>> means no Tpetra.
>>>
>>> Things may have changed in the late 2016/2017 compiler updates, but
>>> AFAIK we have not checked recently.
>>>
>>> -Chris Siefert
>>>
>>>
>>> ________________________________________
>>> From: Trilinos-Users <trilinos-users-bounces at trilinos.org> on behalf
>>> of JR Cary <cary at colorado.edu>
>>> Sent: Friday, February 24, 2017 11:05 AM
>>> To: trilinos-users at trilinos.org
>>> Cc: Heroux, Michael A
>>> Subject: [EXTERNAL] Re: [Trilinos-Users] 64bit indices in Hypre,
>>> Superlu-Dist and Metis/Parmetis
>>>
>>> Hi Mike,
>>>
>>> So then another question -- to move to the TPetra stack, I have
>>> heard that I need to build Kokkos, because TPetra depends on Kokkos,
>>> but that Kokkos is not building on Windows. If all this is true,
>>> then I would not be able to build on Windows with the same
>>> stack, I guess. Is that true? Or is there a workaround?
>>>
>>> (I understand that int is 64 bits on 64 bit Windows, but maintaining
>>> two build stacks would be a hassle.)
>>>
>>> This is important to us because we provide our software to DOE users
>>> at, e.g., the US Particle Accelerator School, and they all have Windows
>>> machines.
>>>
>>> Thanks......John
>>>
>>>
>>>
>>> On 2/23/17 12:29 PM, Heroux, Michael A wrote:
>>>> John, others,
>>>>
>>>> There have been many discussions over the years about how to
>>>> support 64-bit integers in Trilinos. Text substitution, while
>>>> common in the scientific software ecosystem, is not a good way
>>>> (IMO) and leads to downstream integration issues with large
>>>> software bases. Suppose one component wants to use the 32-bit
>>>> instance of a library and another wants to use 64-bits. There is
>>>> no way to reconcile this if you can only build once instance of the
>>>> library at a time.
>>>>
>>>> This approach is often used to support different numerical types as
>>>> well. The same issue arises: You can’t have both a double
>>>> precision real and double precision complex version available at
>>>> the same time.
>>>>
>>>> There is a 64-bit Epetra API (Epetra64) along with AztecOO and
>>>> Ifpack. It does not disrupt the 32-bit interface, but augments
>>>> it. Also, while the global indices are 64-bit, the local indices
>>>> are still 32-bit. This limits the local problem size (under each
>>>> MPI process) to by 2B, but preserves sparse kernel performance,
>>>> which is significantly impacted by the extra 4*nz bytes needed by
>>>> 64-bit ints vs 32-bit. Epetra64 has distinct function prototypes,
>>>> but is supported underneath by templated implementations to reduce
>>>> code redundancy.
>>>>
>>>> We never got to doing ML. Muelu support for Epetra64 never got
>>>> done either. This is because, as we were progressing with 64-bit
>>>> support through the Epetra stack, the need for it diminished.
>>>> Tpetra was becoming more production-capable and the supporting
>>>> pieces (Muelu, Belos, Ifpack2, Anasazi, etc.) were in place.
>>>>
>>>> For full 64-bit integer support in Trilinos, the Tpetra stack is
>>>> the best option. It is also the best option for 16-bit integers,
>>>> or other ordinal types. Similarly, it readily supports 32-bit real
>>>> and complex and extended types. Funky types may not work out of
>>>> the gate, but there are some fun ways to play with the template
>>>> scalar types. We are just discovering in Kokkos, which adopt the
>>>> Tpetra approach for scalar type support, that templating is very
>>>> nice for instantiating vector data types, removing ambiguity about
>>>> address boundaries and whether loops are vectorizable or not.
>>>>
>>>> Unfortunately, templates bring build-time complexity, which has
>>>> been a real headache, no doubt. At the same time, we are able to
>>>> instantiate any and all types simultaneously without ambiguity.
>>>> This is very valuable.
>>>>
>>>> There is a fair bit of functionality in the Epetra stack that
>>>> supports 64-bit ints, and there is only a modest amount of effort
>>>> to make Muelu support Epetra64. Even so, we decided some time ago
>>>> to avoid further development of Epetra64 capabilities, since we
>>>> prefer that people make the transition to the Tpetra stack.
>>>>
>>>> Thanks.
>>>>
>>>> Mike
>>>>
>>>> On 2/23/17, 10:54 AM, "Trilinos-Users on behalf of JR Cary"
>>>> <trilinos-users-bounces at trilinos.org on behalf of
>>>> cary at colorado.edu> wrote:
>>>>
>>>> Hi Andrey,
>>>>
>>>> So because of the 2e9 problem, we need to embark on the int ->
>>>> ssize_t
>>>> migration. You guys did it through templating over the
>>>> integer type.
>>>> Other approaching could include having a define or simply a
>>>> global
>>>> replacement.
>>>>
>>>> Can you share how you came to decide on templating?
>>>>
>>>> Thx....John
>>>>
>>>> On 2/23/2017 8:21 AM, Prokopenko, Andrey V. wrote:
>>>> > There was an effort to allow Epetra with 64-bit indices. So,
>>>> > technically most Epetra classes are now templated on the
>>>> index type.
>>>> >
>>>> > I think the conversion mainly touched widely used classes. I
>>>> don't
>>>> > think it has 100% coverage. Looking at the code Denis
>>>> mentioned, some
>>>> > of the required classes there were not converted, and thus
>>>> are not
>>>> > compatible. I am not certain of an amount of effort required
>>>> to do that.
>>>> >
>>>> > -Andrey
>>>> >
>>>> > On 02/23/2017 08:17 AM, Klinvex, Alicia Marie wrote:
>>>> >> Hello Denis,
>>>> >>
>>>> >> Since Epetra isn't templated, I'm not sure I see a way to
>>>> support
>>>> >> 64bit ints in that hypre interface. (CCed Mark in case he
>>>> has some
>>>> >> ideas.) We should be able to support it in the tpetra-based
>>>> >> interface though, if you're interested in that one.
>>>> >>
>>>> >> Best wishes,
>>>> >> Alicia
>>>> >>
>>>> >> -----Original Message-----
>>>> >> From: Trilinos-Users
>>>> [mailto:trilinos-users-bounces at trilinos.org] On
>>>> >> Behalf Of Denis Davydov
>>>> >> Sent: Thursday, February 23, 2017 5:14 AM
>>>> >> To: trilinos-users <trilinos-users at trilinos.org>
>>>> >> Subject: [EXTERNAL] [Trilinos-Users] 64bit indices in Hypre,
>>>> >> Superlu-Dist and Metis/Parmetis
>>>> >>
>>>> >> Dear all,
>>>> >>
>>>> >> I am building a stack (with 64bit indices) composed of PETSc,
>>>> >> Trilinos, Superlu-Dist, Metis, etc and came across the
>>>> following
>>>> >> compiler error in EpetraExt:
>>>> >>
>>>> >>
>>>> /home/user/spack/var/spack/stage/trilinos-12.10.1-kgdtcmcwce6xhetukputhnh2f7v4sljf/Trilinos-trilinos-release-12-10-1/packages/epetraext/src/hypre/EpetraExt_HypreIJMatrix.cpp:91:96:
>>>> >> error: cannot convert ‘int*’ to ‘HYPRE_Int* {aka long long
>>>> int*}’ for
>>>> >> argument ‘3’ to ‘HYPRE_Int
>>>> >> HYPRE_ParCSRMatrixGetRow(HYPRE_ParCSRMatrix, HYPRE_Int,
>>>> HYPRE_Int*,
>>>> >> HYPRE_Int**, HYPRE_Complex**)’
>>>> >> ierr += HYPRE_ParCSRMatrixGetRow(ParMatrix_,
>>>> i+MyRowStart_,
>>>> >> &num_entries, &indices, &values);
>>>> >> ^
>>>> >>
>>>> >> That’s of course because if PETSc is built with 64bit ints,
>>>> then
>>>> >> Hypre have to be also built accordingly.
>>>> >>
>>>> >> So what are the requirements from Trilinos side on those
>>>> >> dependencies? Can it not be built against Hypre or
>>>> Superlu-Dist when
>>>> >> those packages are configured with —enable-biginit and
>>>> -D_LONGINT,
>>>> >> respectively?
>>>> >> Or is it a bug in EpetraExt?
>>>> >>
>>>> >> Regards,
>>>> >> Denis.
>>>> >>
>>>> >> _______________________________________________
>>>> >> Trilinos-Users mailing list
>>>> >> Trilinos-Users at trilinos.org
>>>> >> https://trilinos.org/mailman/listinfo/trilinos-users
>>>> >> _______________________________________________
>>>> >> Trilinos-Users mailing list
>>>> >> Trilinos-Users at trilinos.org
>>>> >> https://trilinos.org/mailman/listinfo/trilinos-users
>>>> >
>>>> > _______________________________________________
>>>> > Trilinos-Users mailing list
>>>> > Trilinos-Users at trilinos.org
>>>> > https://trilinos.org/mailman/listinfo/trilinos-users
>>>> >
>>>>
>>>> _______________________________________________
>>>> Trilinos-Users mailing list
>>>> Trilinos-Users at trilinos.org
>>>> https://trilinos.org/mailman/listinfo/trilinos-users
>>>>
>>>>
>>>>
>>> _______________________________________________
>>> Trilinos-Users mailing list
>>> Trilinos-Users at trilinos.org
>>> https://trilinos.org/mailman/listinfo/trilinos-users
>>>
>> _______________________________________________
>> Trilinos-Users mailing list
>> Trilinos-Users at trilinos.org
>> https://trilinos.org/mailman/listinfo/trilinos-users
>>
>
More information about the Trilinos-Users
mailing list