[Beowulf] Register article on Cray Cascade

[atchley tds.net]

Vincent, it is easy to measure.

1. Connect to NICs back-to-back.
2. Measure latency
3. Connect machines to switch
4. Measure latency
5. Subtract (2) from (4)

That is how we did it at Myricom and that is how we do it at ORNL.

Try it sometime.

Scott


On Fri, Nov 9, 2012 at 2:36 PM, Vincent Diepeveen <diep at xs4all.nl> wrote:

>
> On Nov 9, 2012, at 7:31 PM, atchley tds.net wrote:
>
>  Modern switches need 100-150 ns per hop.
>>
>
> yeah that's BS when you have software that goes measure that with all
> cores busy.
>
> I wrote a benchmark to measure that with all cores busy.
>
> The SGI box back then that was having 50 ns switches which would have 'in
> theory' a latency of 480 ns @ 500 cpu's,
> so 960 for a blocked read, i couldn't get it down to less than 5.8 us on
> average.
>
>
>
>
>  There are some things that do not scale per hp such as traversing the
>> PCIE link from socket to NIC and back. So, I see it as 1.2 to go to the
>> router and back and 100 ns per hop.
>>
>> Scott
>>
>>
>> On Fri, Nov 9, 2012 at 11:17 AM, Vincent Diepeveen <diep at xs4all.nl>
>> wrote:
>> The latency estimate taking 5 hops seems a tad optimistic to me
>> except when i read the English wrong and they mean 1.7 microseconds a
>> hop making it for a 5 hop 5 * 1.7 = 8.5 microseconds in total.
>>
>> "Not every node is only one hop away, of course. On a fully
>> configured system, you are five hopes away maximum from any socket,
>> so there is some latency. But the delta is pretty small with
>> Dragonfly, with a minimum of about 1.2 microseconds for a short hop,
>> an average of 1.5 microseconds on average, and a maximum of 1.7
>> microseconds for the five-hop jump, according to Bolding."
>>
>> On Nov 8, 2012, at 7:13 PM, Hearns, John wrote:
>>
>> > Well worth a read:
>> >
>> >
>> >
>> > http://www.theregister.co.uk/**2012/11/08/<http://www.theregister.co.uk/2012/11/08/>
>> > cray_cascade_xc30_**supercomputer/
>> >
>> >
>> >
>> > John Hearns | CFD Hardware Specialist | McLaren Racing Limited
>> > McLaren Technology Centre, Chertsey Road, Woking, Surrey GU21 4YH, UK
>> >
>> >
>> > T:  +44 (0) 1483 262000
>> >
>> > D:  +44 (0) 1483 262352
>> >
>> > F:  +44 (0) 1483 261928
>> > E:  john.hearns at mclaren.com
>> >
>> > W: www.mclaren.com
>> >
>> >
>> >
>> > The contents of this email are confidential and for the exclusive
>> > use of the intended recipient. If you receive this email in error
>> > you should not copy it, retransmit it, use it or disclose its
>> > contents but should return it to the sender immediately and delete
>> > your copy.
>> >
>> > ______________________________**_________________
>> > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin
>> > Computing
>> > To change your subscription (digest mode or unsubscribe) visit
>> > http://www.beowulf.org/**mailman/listinfo/beowulf<http://www.beowulf.org/mailman/listinfo/beowulf>
>>
>> ______________________________**_________________
>> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
>> To change your subscription (digest mode or unsubscribe) visit
>> http://www.beowulf.org/**mailman/listinfo/beowulf<http://www.beowulf.org/mailman/listinfo/beowulf>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20121109/2db2ba19/attachment.html>