[Trilinos-Users] [EXTERNAL] IO and general questions

[Sjaardema, Gregory D]

From: Carlos Breviglieri <carbrevi at gmail.com<mailto:carbrevi at gmail.com>>
Date: Friday, August 16, 2013 9:20 AM
To: "trilinos-users at software.sandia.gov<mailto:trilinos-users at software.sandia.gov>" <trilinos-users at software.sandia.gov<mailto:trilinos-users at software.sandia.gov>>
Subject: [EXTERNAL] [Trilinos-Users] IO and general questions

Hello, I am Carlos, a PhD candidate in Brazil. I've been recently checking out Trilinos as a basis for a CFD application. My research focus is on high-order unstructured methods for aeronautical/space applications. I have checked the TUG videos and must say that its a great way to get started. I've done most of the tutorials and read some doxygen stuff.

I've come across a couple of points with some packages that I hope someone can clarify. These are somewhat entrypoint questions for me and, hopefully, might be usefull to someone else get started. I am using git sources from august 13th. I apologize if this is not the proper mail-list to do so and the lengthy topic...

SEACAS::Ioss
I need to do parallel IO. I've got experience with the CGNS/HDF5 library and I am aware of the par_exo factory (Iopx) for the exodus format and got it compiled with specific flags from cmake. I can switch to exodus but I was not able to confirm, from the source code, that:

a) can it do parallel read/write + parallel decomposition? From "packages/seacas/libraries/ioss/src/main/io_shell.C" it is possible to partition (zoltan or metis) the domain but I believe that at one moment one proc will hold the entire mesh, before calling the partitioner interface. Afterwards, it can compose the output, which is a feature I am interested in. Moreover the source states that, at this point, no parallel RESTART is supported.

-- At no time is the entire mesh on a single processor.  The code initially performs a linear decomposition (#element/#nprocessors) elements to each processor.  This initial decomposition is then fed to zoltan or metis to get a better decomposition.

So, the questions here are:
a.1) Is Iopx able to do parallel IO (I assume so since it uses the netcdf4 format)?
It is, but note that it is a fairly new capability and hasn't been fully productionized and optimized.  The automatic decomposition is farther along than the composition.  Note that the historical method of doing parallel with exodus has been to provide a "file-per-processor" at the beginning of the analysis which then writes a file-per-processor.  The decomp script is used to decompose a mesh into a file-per-processor and the epu executable is used to join the file-per-processor output into a single file.

a.2) Is Zoltan (did not check Zoltan2) able to do parallel decomposition internally (not rely on ParMETIS)?

It would be possible to modify the code to only support zoltan, but currently it is set up to require both zoltan and parmetis.


a.3) Are composition methods (join the output solution files) experimental? (had issues with netcdf library versioning)

The composition method is definitely not production ready at this time.  There are efficiency issues with large numbers of processors and some tuning of hdf5 and mpiio settings is needed (as is probably some internal code work in Iopx::DatabaseIO).  There have been multiple bugs fixed in the parallel portions of netcdf; I would definitely recommend using netcdf-4.3.0 or later if you are planning to use the parallel-netcdf (either hdf5-based or pnetcdf-based).

Note that we hope to have a CGNS read/write capability for structured meshes in Ioss sometime in the next fiscal year.

STK

b) I was able to read and decompose an exodus/netcdf4 mesh (through meshData.properties) in Panzer using exodus stk_interface class, but I am not sure if the decomposition and IO happens in parallel...?

Not sure about that method, but if you use the property method or the environment method to specify the DECOMPOSITION_METHOD (see 'e' below), it is done in parallel.

c) I have not found any indication from stk sources that it supports output composition... Can it be done? It uses internal write routines that act upon an Ioss database, which does support it. I have also checked the tExodusReaderFactory.cpp from Panzer package that interfaces with STK but such functionality was not present.

The composition is selectable via the setting of "properties" that are passed down to the Ioss class.  I'm not sure how to do that in panzer.  Another method of doing this is by setting an environment variable prior to running the application.  For mesh composition, you would do something like:  export IOSS_PROPERTIES=COMPOSE_RESULTS=YES:COMPOSE_RESTART=YES




d) Is STK able to do AMRC (Adaptive Mesh Refinement/Coarsening)? I would use such feature for shock-capturing algorithms.

e) Assuming one is able to read a mesh in parallel (Iopx) each proc ends up with an unordered slice of the mesh. Is there functionallity in STK to decompose it in parallel and send/receive the mesh data to its appropriate proc? Would this be related to the rebalance algorithms in STK?

The mesh decomposition capability can be enabled in the same way via "export IOSS_PROPERTIES=DECOMPOSITION_METHOD={method}" where {method} is one of linear, rcb, rib, hsfc, block, cyclic, random, kway, geom_kway, metis_sfc.

--Greg Sjaardema

Regards

Carlos Breviglieri

-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://software.sandia.gov/pipermail/trilinos-users/attachments/20130816/979f0b7e/attachment.html