[Beowulf] MPI programming question: Interleaved MPI_Gatherv?

[Robert G. Brown]

On Thu, 3 Mar 2005, Joachim Worringen wrote:

> Michael Gauckler wrote:
> > Dear List, 
> > 
> > I would like to gather the data from several processes. 
> > Instead of the comonly used stride, I want to interleave 
> > the data:
> > 
> > Rank 0: AAAAA -> ABCDABCDABCDABCDABCD
> > Rank 1: BBBBB ----^---^---^---^---^
> > Rank 2: CCCCC -----^---^---^---^---^
> > Rank 3: DDDDD ------^---^---^---^---^
> > 
> > Since the stride of the receive type is indicated 
> > in multpiles of its mpi_type, no interleaving is 
> > possible (the smallest striping factor leads to 
> > AAAAABBBBBBCCCCCDDDDD).
> > 
> > Is there a way to achieve this behaviour in an 
> > elegant way, as MPI_Gather promises it? Or do
> > I need to do Send/Recv with self-aligned offsets?

What about RMA-like commands?  MPI_Get in a loop?  Since that is
controlled by the gatherer, one would presume that it preserves call
order (although it is non-blocking).

Or of course there are always raw sockets... where you have complete
control.  Depending on how critical it is that you preserve this strict
interleaving order.

   rgb

> 
> Actually, I don't see an 'elegant' way to do this, either. The decision 
> between multiple MPI_Gatherv() calls and a Irecv/Send/Waitall construct 
> depends on the quality of the MPI implementation you use (MPI_Gatherv 
> can be optimized well for small amounts of data), the characteristics of 
> you interconnect (high latency gives more room for optimization) and the 
> number of processes you use. For small process numbers, you wont see 
> much of a difference anyway.
> 
> You could also try to gather all data on the root in separate buffers, 
> and then let this process send/recv to itself using the proper datatypes.
> 
> Finally, if this communication is not a significant part of your 
> runtime, you shouldn't spend much time optimizing it anyway.
> 
>   Joachim
> 
> 

-- 
Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu