[Beowulf] Broadcast - not for HPC - or is it?

[Lux, Jim (337C)]

> -----Original Message-----
> From: beowulf-bounces at beowulf.org [mailto:beowulf-bounces at beowulf.org] On Behalf Of Daniel Kidger
> Sent: Thursday, October 07, 2010 8:10 AM
> To: Matt Hurd
> Cc: beowulf at beowulf.org
> Subject: Re: [Beowulf] Broadcast - not for HPC - or is it?
> 
>   Matt,
> 
> Are you really claiming <9ns port to port ?
> (Quadrics used to think they were leading edge with 40ns latency port to
> port latency on their switches)

That's a fairly impressive spec in itself.  

> 
> At <9ns for the 'switch' then surely the speed of light in copper (a
> massive 1ns per foot) will dominate over the switch itself?

1 ns/foot in copper would be a mighty fine accomplishment.  More realistic is something along the lines of .65-.70c for unshielded twisted pair (e.g. the ubiquitous Cat 5/6 wiring has a delay spec of 570-536ns/100m depending on frequency)

Even in optical fiber, you're looking at somewhat lower than 3E8 m/s... the refractive index is around 1.5... call it .65 to .7c   (there's a reason it's the same general magnitude as copper)

On the PCB, you're probably looking at propagation speeds of 0.5c.

And, in a *real* system, there are delays in every transition from one mode of propagation or widget to another.  Charging up the lead capacitance takes non-zero time. Filling the junction in a VCSEL transmitter takes non-zero time.  The devices and designs may have huge bandwidths (GHz), but they're like a pipeline, and it takes some time for stuff to get from one place to another.

(a colleague has a copy of a book called "High-Speed Signal Propagation, Advanced Black Magic", by Johnson & Graham that is a great compendium of all sorts of handy rules of thumb and design principles.  His version is about 10 years old. I wonder if there's a newer one out.)

> 
> 
> Plus as others say it is not the broadcast that is the hard bit - it is
> getting the consolidated acks back.
> 
> Daniel
> 
> 
> > distribution of market data to multiple users as the port to port
> > latency is in the range of 5-7 nanoseconds as it is pretty passive
> > device with optical foo at the core.  No rocket science here, just
> > convenient opto-electrical foo.
> >
> >
> > Hmmm, we could build such a thing at about 8-9 ns latency but I don't
> > quite get the point just being used to embarrassingly parallel stuff


I can believe a "signal at pin on chip to signal on another pin on same chip" latencies of a few ns: that's pretty standard high speed MSI logic. On a well designed ASIC, you can probably get few ns kind of delay for pins that are close to each other, if the package isn't too big.  But "connector on box to connector on box" < 10ns... that's an accomplishment.