[Trilinos-Users] [EXTERNAL] 2 zoltan questions

Devine, Karen D kddevin at sandia.gov
Wed Dec 19 10:50:30 MST 2012

On 12/19/12 5:57 AM, "Biddiscombe, John A." <biddisco at cscs.ch> wrote:

>Question 1:
>When I¹m receiving points from a remote process, I know their global Id
>because It is one of the params passed to the unpack function. The points
>have Š
>their original local Id on the remote process which I don¹t know on this
>process (unless I send it myself as part of the data or as part of a
>multi-component global Id)
>their global Id, which is what I see when they arrive
>their new local Id ­ this is generated as they are unpacked and added to
>the local list.
>Typically I¹m seeing that global Id 12345 is being received and become
>local ID 10000 and then global Id 12346 becomes 10001, and so on for a
>sometimes quite long sequence of Ids until some gap or break in the
>points from the remote process
> occurs and then I¹ll get one or more points like 12500 globally becomes
>12400 locally. Etc etc.
>When cells arrive (later), I need to map their own remote local (but
>they¹ll be global as I¹ll need to substitute them before sending) ids to
>points to the new truly local Ids and so I need to create a map to
>convert global to local ids
> for the points I¹ve received. This may take a lot of memory and I would
>like to save some space by using a kind of interval lookup. Since I know
>that 12345-10000 = 2345 and 12346-10001=2345, etc etc, I can build a list
>of offsets based on global->local Ids
> (which will be more compact in the usual case than a map). If every
>Œother¹ point is being sent then it won¹t be so efficient, but that seems
>So the question becomes, are the global Ids I receive in the unpack
>callback always monotonically increasing (this simplifies my interval
>build routine) and is there a better way of doing this (if I use a global
>Id that is multi-component,
> I¹ll still need to create a map to work out the transformed Ids when the
>cells arrive, so there seems to be no advantage, unless I¹ve missed a
>simple idea). Someone must have done this already, is there a code
>snippet for handling remote local/global to new
> local local Ids?

In the unpack callback, there are no guarantees about the Ids being
monotonically increasing.  I suspect you are getting this lucky behavior
because your Ids are nicely ordered within processors to begin with.  For
purely on-processor data, you can efficiently map global Ids to local
indices with a hash table or std::map.  For mapping global Ids to remote
local ids, you can use the Zoltan distributed data directory.  If you
decide to use the data directory, I am happy to provide instructions.

>Question 2:
>If I¹m using the standard zoltan callbacks like
>    Zoltan_Set_Fn(zz,ZOLTAN_PACK_OBJ_FN_TYPE,
>    Zoltan_Set_Fn(zz,ZOLTAN_UNPACK_OBJ_FN_TYPE,
>    Zoltan_Set_Fn(zz,ZOLTAN_PRE_MIGRATE_PP_FN_TYPE,(void(*)())f4,&mesh);
>And then calling 
>    rc=Zoltan_LB_Partition(zz,// input (all remaining fields are output)
>Will this change much when I move to zoltan2? (yes or no will do, I
>didn¹t really look at the zoltan2 pages yet, I¹m just curious if it¹ll
>change completely or I¹ll be using lambda functions or functors instead
>of callbacks or something)

Yes, the interface to Zoltan2 is much different from Zoltan.  In Zoltan2,
users will instantiate an input adapter that describes their problem;
input adapters currently supported describe meshes, matrices, coordinates,
identifiers, and graphs.  The benefit is that users who have, say, a mesh,
no longer will have to convert their mesh to graphs or hypergraphs;
Zoltan2 will do the appropriate conversion based on mesh adjacencies.
While we plan to create a Zoltan1-input adapter to allow easier conversion
to Zoltan2, this adapter is not yet available.  Which partitioning methods
are you using?


>John Biddiscombe,                        email:biddisco @.at.@ cscs.ch
>CSCS, Swiss National Supercomputing Centre  | Tel:  +41 (91) 610.82.07
>Via Trevano 131, 6900 Lugano, Switzerland   | Fax:
> +41 (91) 610.82.82

More information about the Trilinos-Users mailing list