Fwd: [Beowulf] Re: Why is communication so expensive for very small messages

[Jonathan Boyle]

Thanks for the reply.

I tried reducing the number of loops from 5000, I went as low at 7, and still 
measured times of 1ms per communication. Below 7 loops I measured a time of 
zero, but MPI_Wtime() only has a resolution of 4ms.

I wasn't sure what you meant by handling the return status.

I changed the code to check the returned error value, but find no errors.

And I find putting MPI_Barrier(MPI_COMM_WORLD) after every communication 
reduces the time per communication from 1ms to 70microseconds.




----------  Forwarded Message  ----------

Subject: [Beowulf] Re: Why is communication so expensive for very small 
messages
Date: Tue, 24 Apr 2007 12:30:03 -0700
From: "David Mathog" <mathog at caltech.edu>
To: beowulf at beowulf.org

Jonathan Boyle <j.boyle at manchester.ac.uk> wrote:
> I apologise if this is a naive question, but I'm new to this world of
> beowulfs.
>
> I'm using C++/mpi, to get a feel for communication costs I ran tests

using

> mpptest and my own programs.
>
> For 2 processor blocking calls, mpptest indicates a latency of about 30
> microseconds.
>
> However when I measure communication times in my own program using a

loop as

> follows....
>
> MPI_Barrier(MPI_COMM_WORLD);
> start = MPI_Wtime();
> for (unsigned t=1; t<=5000; t++)
> {
>  if (my_rank==0)
>  {
>   MPI_Send(data, size, MPI_INT, 1, tag, MPI_COMM_WORLD);
>  }
>  else
>  {
>   MPI_Recv(data, size, MPI_INT, 0, tag, MPI_COMM_WORLD, &status);
>  }
> }
> end = MPI_Wtime();
>
> for size>=4, I get a latency of about 30 microseconds as expected,

however for

> size<4, communication costs increase massively, and latency now

appears to be

> 1ms!

Odd.  Could be a couple of things going on.

1.  By pure bad luck dropping size 4 -> 3 (for instance) may be the
point at which your network saturates.  That is, the slightly smaller
packets may be generated slightly faster so that you hit a bottleneck
(max packets per second inbound would be my guess, I've hit that limit
before) and so you see a nonlinear response.  You can test this by
dropping the 5000 to 100 or so, so no matter how fast they go,
they don't max anything out.

2.  Could be a bug in the mpi implementation you are using.

I've never written anything for MPI in C++, is the return status
being handled somewhere?  I don't see it in this code fragment but
C++ is notorious for putting side effects in another section of code,
far far away. If at all possible the status values should be
checked. Sooner or later one of these functions is going to
come back with a bad status and your program should handle that.

Regards,

David Mathog
mathog at caltech.edu
Manager, Sequence Analysis Facility, Biology Division, Caltech