[Trilinos-Users] 64bit indices in Hypre, Superlu-Dist and Metis/Parmetis
James
jjellio3 at ncsu.edu
Sat Feb 25 13:33:23 EST 2017
Hi John and Mike,
I got curious so I downloaded MSYS and installed Mingw-64 on my home
windows 7 machine.
OpenBLAS built fine (required TPL).
I managed to configure fine. MSYS/MinGW-w64 provides GCC-6.3, so
compiler language support should not be an issue.
It did not build (darn, but I didn't expect it to be that easy).
The Kokkos Windows specific files failed, but not due to compiler
errors, they simply are out of date.
It also seems you would need to use Microsoft's MPI. Or build MPICH from
source. I tried OpenMPI and had configure errors due to autotools not
finding some required system headers.
That said, much of this could be resolved I believe, but it would take
effort.
Andrey's recommendation for a VirtualMachine/Container type approach is
much easier, but it means the app would not run natively on Windows.
Rather, it would run inside a lightweight container.
Supporting MSYS/MinGW-w64 would be more viable than supporting Visual
Studio's compiler. Microsoft has very little incentive to promote C++
nowadays. Development with MinGW isn't that bad either. Eclipse (running
in Windows) use to handle it fairly well for C++.
James
On 2/25/2017 10:08 AM, JR Cary wrote:
> Thanks, Mike.
>
> For the most part, our users never see the compilation. They just
> execute binaries,
> so it is mostly a matter of us putting in the effort to support multiple
> compilers.
> (E.g., VisIt uses only Visual Studio on Windows, and we need that too.)
> So modulo
> the comment by Richard Drake (that MueLu cannot build on Windows with
> any compiler),
> we could make option 2 work.
>
> Have you guys looked at making mingw64 one of your supported compilers?
>
> Thx....John
>
>
>
>
>
> On 2/25/17 8:46 AM, Heroux, Michael A wrote:
>> John,
>>
>> I see two options for addressing the situation:
>>
>> 1. Someone (not core Trilinos staff at this time, given other pressing needs) completes the adoption of Epetra64 through the Epetra stack. ML could be omitted from the effort, since Muelu is a suitable substitute and would require less effort to support Epetra64.
>> 2. Anyone who wants to use the Tpetra stack on Windows uses the Intel C++ compiler with Visual Studio, which supports SFINAE features. While not a very attractive solution for some customer bases, it is probably the best option for DOE users.
>>
>> Is option 2 a possibility for your users?
>>
>> Mike
>>
>> On 2/25/17, 8:38 AM, "Trilinos-Users on behalf of JR Cary" <trilinos-users-bounces at trilinos.org on behalf of cary at colorado.edu> wrote:
>>
>> Unfortunate, Chris, but thanks for the information.
>>
>> FYI for setting priorities... We do address DOE modeling needs both
>> by way of Windows for regular computations, and with the need for 64 bit,
>> large simulations for some design problems. Right now we cannot do the
>> latter without sacrificing the former.
>>
>> Best....John
>>
>>
>> On 2/24/17 2:36 PM, Siefert, Christopher wrote:
>> > John,
>> >
>> > As of VS15, the subset of C++11 supported by Visual Studio was insufficient to build Kokkos on Windows. Ditto for the early 2016 version of the Intel Windows compiler at the same time. No Kokkos means no Tpetra.
>> >
>> > Things may have changed in the late 2016/2017 compiler updates, but AFAIK we have not checked recently.
>> >
>> > -Chris Siefert
>> >
>> >
>> > ________________________________________
>> > From: Trilinos-Users <trilinos-users-bounces at trilinos.org> on behalf of JR Cary <cary at colorado.edu>
>> > Sent: Friday, February 24, 2017 11:05 AM
>> > To: trilinos-users at trilinos.org
>> > Cc: Heroux, Michael A
>> > Subject: [EXTERNAL] Re: [Trilinos-Users] 64bit indices in Hypre, Superlu-Dist and Metis/Parmetis
>> >
>> > Hi Mike,
>> >
>> > So then another question -- to move to the TPetra stack, I have
>> > heard that I need to build Kokkos, because TPetra depends on Kokkos,
>> > but that Kokkos is not building on Windows. If all this is true,
>> > then I would not be able to build on Windows with the same
>> > stack, I guess. Is that true? Or is there a workaround?
>> >
>> > (I understand that int is 64 bits on 64 bit Windows, but maintaining
>> > two build stacks would be a hassle.)
>> >
>> > This is important to us because we provide our software to DOE users
>> > at, e.g., the US Particle Accelerator School, and they all have Windows
>> > machines.
>> >
>> > Thanks......John
>> >
>> >
>> >
>> > On 2/23/17 12:29 PM, Heroux, Michael A wrote:
>> >> John, others,
>> >>
>> >> There have been many discussions over the years about how to support 64-bit integers in Trilinos. Text substitution, while common in the scientific software ecosystem, is not a good way (IMO) and leads to downstream integration issues with large software bases. Suppose one component wants to use the 32-bit instance of a library and another wants to use 64-bits. There is no way to reconcile this if you can only build once instance of the library at a time.
>> >>
>> >> This approach is often used to support different numerical types as well. The same issue arises: You can’t have both a double precision real and double precision complex version available at the same time.
>> >>
>> >> There is a 64-bit Epetra API (Epetra64) along with AztecOO and Ifpack. It does not disrupt the 32-bit interface, but augments it. Also, while the global indices are 64-bit, the local indices are still 32-bit. This limits the local problem size (under each MPI process) to by 2B, but preserves sparse kernel performance, which is significantly impacted by the extra 4*nz bytes needed by 64-bit ints vs 32-bit. Epetra64 has distinct function prototypes, but is supported underneath by templated implementations to reduce code redundancy.
>> >>
>> >> We never got to doing ML. Muelu support for Epetra64 never got done either. This is because, as we were progressing with 64-bit support through the Epetra stack, the need for it diminished. Tpetra was becoming more production-capable and the supporting pieces (Muelu, Belos, Ifpack2, Anasazi, etc.) were in place.
>> >>
>> >> For full 64-bit integer support in Trilinos, the Tpetra stack is the best option. It is also the best option for 16-bit integers, or other ordinal types. Similarly, it readily supports 32-bit real and complex and extended types. Funky types may not work out of the gate, but there are some fun ways to play with the template scalar types. We are just discovering in Kokkos, which adopt the Tpetra approach for scalar type support, that templating is very nice for instantiating vector data types, removing ambiguity about address boundaries and whether loops are vectorizable or not.
>> >>
>> >> Unfortunately, templates bring build-time complexity, which has been a real headache, no doubt. At the same time, we are able to instantiate any and all types simultaneously without ambiguity. This is very valuable.
>> >>
>> >> There is a fair bit of functionality in the Epetra stack that supports 64-bit ints, and there is only a modest amount of effort to make Muelu support Epetra64. Even so, we decided some time ago to avoid further development of Epetra64 capabilities, since we prefer that people make the transition to the Tpetra stack.
>> >>
>> >> Thanks.
>> >>
>> >> Mike
>> >>
>> >> On 2/23/17, 10:54 AM, "Trilinos-Users on behalf of JR Cary" <trilinos-users-bounces at trilinos.org on behalf of cary at colorado.edu> wrote:
>> >>
>> >> Hi Andrey,
>> >>
>> >> So because of the 2e9 problem, we need to embark on the int -> ssize_t
>> >> migration. You guys did it through templating over the integer type.
>> >> Other approaching could include having a define or simply a global
>> >> replacement.
>> >>
>> >> Can you share how you came to decide on templating?
>> >>
>> >> Thx....John
>> >>
>> >> On 2/23/2017 8:21 AM, Prokopenko, Andrey V. wrote:
>> >> > There was an effort to allow Epetra with 64-bit indices. So,
>> >> > technically most Epetra classes are now templated on the index type.
>> >> >
>> >> > I think the conversion mainly touched widely used classes. I don't
>> >> > think it has 100% coverage. Looking at the code Denis mentioned, some
>> >> > of the required classes there were not converted, and thus are not
>> >> > compatible. I am not certain of an amount of effort required to do that.
>> >> >
>> >> > -Andrey
>> >> >
>> >> > On 02/23/2017 08:17 AM, Klinvex, Alicia Marie wrote:
>> >> >> Hello Denis,
>> >> >>
>> >> >> Since Epetra isn't templated, I'm not sure I see a way to support
>> >> >> 64bit ints in that hypre interface. (CCed Mark in case he has some
>> >> >> ideas.) We should be able to support it in the tpetra-based
>> >> >> interface though, if you're interested in that one.
>> >> >>
>> >> >> Best wishes,
>> >> >> Alicia
>> >> >>
>> >> >> -----Original Message-----
>> >> >> From: Trilinos-Users [mailto:trilinos-users-bounces at trilinos.org] On
>> >> >> Behalf Of Denis Davydov
>> >> >> Sent: Thursday, February 23, 2017 5:14 AM
>> >> >> To: trilinos-users <trilinos-users at trilinos.org>
>> >> >> Subject: [EXTERNAL] [Trilinos-Users] 64bit indices in Hypre,
>> >> >> Superlu-Dist and Metis/Parmetis
>> >> >>
>> >> >> Dear all,
>> >> >>
>> >> >> I am building a stack (with 64bit indices) composed of PETSc,
>> >> >> Trilinos, Superlu-Dist, Metis, etc and came across the following
>> >> >> compiler error in EpetraExt:
>> >> >>
>> >> >> /home/user/spack/var/spack/stage/trilinos-12.10.1-kgdtcmcwce6xhetukputhnh2f7v4sljf/Trilinos-trilinos-release-12-10-1/packages/epetraext/src/hypre/EpetraExt_HypreIJMatrix.cpp:91:96:
>> >> >> error: cannot convert ‘int*’ to ‘HYPRE_Int* {aka long long int*}’ for
>> >> >> argument ‘3’ to ‘HYPRE_Int
>> >> >> HYPRE_ParCSRMatrixGetRow(HYPRE_ParCSRMatrix, HYPRE_Int, HYPRE_Int*,
>> >> >> HYPRE_Int**, HYPRE_Complex**)’
>> >> >> ierr += HYPRE_ParCSRMatrixGetRow(ParMatrix_, i+MyRowStart_,
>> >> >> &num_entries, &indices, &values);
>> >> >> ^
>> >> >>
>> >> >> That’s of course because if PETSc is built with 64bit ints, then
>> >> >> Hypre have to be also built accordingly.
>> >> >>
>> >> >> So what are the requirements from Trilinos side on those
>> >> >> dependencies? Can it not be built against Hypre or Superlu-Dist when
>> >> >> those packages are configured with —enable-biginit and -D_LONGINT,
>> >> >> respectively?
>> >> >> Or is it a bug in EpetraExt?
>> >> >>
>> >> >> Regards,
>> >> >> Denis.
>> >> >>
>> >> >> _______________________________________________
>> >> >> Trilinos-Users mailing list
>> >> >> Trilinos-Users at trilinos.org
>> >> >> https://trilinos.org/mailman/listinfo/trilinos-users
>> >> >> _______________________________________________
>> >> >> Trilinos-Users mailing list
>> >> >> Trilinos-Users at trilinos.org
>> >> >> https://trilinos.org/mailman/listinfo/trilinos-users
>> >> >
>> >> > _______________________________________________
>> >> > Trilinos-Users mailing list
>> >> > Trilinos-Users at trilinos.org
>> >> > https://trilinos.org/mailman/listinfo/trilinos-users
>> >> >
>> >>
>> >> _______________________________________________
>> >> Trilinos-Users mailing list
>> >> Trilinos-Users at trilinos.org
>> >> https://trilinos.org/mailman/listinfo/trilinos-users
>> >>
>> >>
>> >>
>> > _______________________________________________
>> > Trilinos-Users mailing list
>> > Trilinos-Users at trilinos.org
>> > https://trilinos.org/mailman/listinfo/trilinos-users
>> >
>>
>> _______________________________________________
>> Trilinos-Users mailing list
>> Trilinos-Users at trilinos.org
>> https://trilinos.org/mailman/listinfo/trilinos-users
>>
>>
>>
> _______________________________________________
> Trilinos-Users mailing list
> Trilinos-Users at trilinos.org
> https://trilinos.org/mailman/listinfo/trilinos-users
More information about the Trilinos-Users
mailing list