[Beowulf] Register article on Opteron - disagree

[Joe Landman]

This is more to Patrick than Jeff

Jeff Layton wrote:

> Patrick Geoffray wrote:
>

[...]

> It's interesting how PR works and how companies position themselves.
> It might make an interesting study in human pyschology.
>
>>> However, I guess it's understandable since, as rgb points out, the 
>>> Top500
>>> is an advertising thing anyway. Doug Eadline has a great quote about
>>
>>
>>
>> RGB is wrong. I will repeat it one more time: the top500 is a 
>> statistical study. It is an advertising thing only in the eyes of the 
>> believer. If you don't like what the Top500 is trying to do, just 
>> ignore it. 
>
>

Um...  must be some new definition of "random sampling" that I am 
unaware of.   Top500 is a self-selecting list, which by definition 
introduces a bias with regards to those who choose to report.   It is 
not a random unbiased sampling of a population, in fact it is an attempt 
to get extremal values of a "measurement" from the population, hence it 
is not likely to be random.

It is used in advertising, though as many people point out (some of us 
for quite a while), it is inherently bereft of significant meaning to 
most users:  to wit, I do not know if there is more than one or two 
people on the entire planet whose job it is to run HPL day in and day 
out.  This is not to say it is completely devoid of meaning, but 
correllations between top500 position, market share, application 
performance seem to be random ... (unless everyone but me has had an 
earth simulator in their garage for a while).

I would look at those numbers with a few kilograms of NaCl for their 
overall predictive capacity. 

>
> I watch the Top500 for the pure joy of watching a benchmark number
> over time. Myself and others have written that one very good thing
> the Top500 has going for it is that it has a very long history. This 
> allows
> you to watch HPC development over time. As for an absolute benchmark
> I agree with rgb on this. It's a beauty contest pure and simple. Besides
> there are faster parallel codes for solving LU problems anyway.


The other very important aspect of HPC is that it has grown up quite a 
bit.  HPC != large linear algebra problems anymore.  LU decomposition 
benchmarks have very little .... well none actually ... meaning to folks 
building HMMs, or doing homology searches.  It has some limited meaning 
to those folding proteins.


>
>>> the SuperComputer show and the Top500. I can't repeat the quote here,
>>> but it has something to do with "size." After experiencing the show, 
>>> the
>>> observation is ABSOLUTELY right on target.
>>
>>
>>
>> Do you know any effort involving benchmarks that do not ultimately 
>> look and smell like a penis contest ? It's human nature :-) 
>
>
>
> Sure I know of several. Larry McVoy's lmbench that rgb has spoken about
> is one of the better ones. It's a bit dated, but still useful. I also 
> think the new
> ClusterWorld Benchmarking project, while still in the forming stages, 
> will
> be a good benchmark. However, it's not really a benchmarking project 
> in the
> sense of "my machine is faster than yours." It has a much better idea 
> of what
> benchmarking should be used for.
>
> The way to help out the benchmarking "thing" is to explain to people
> how benchmarks should be used and how to always look at benchmark
> results with a skeptical eye.


This is the idea behind BBS (http://bioinformatics.org/bbs), where you 
start with your code and your data, and work to obtain meaningful (to 
you) numbers ... 

-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: landman at scalableinformatics.com
web  : http://www.scalableinformatics.com
phone: +1 734 612 4615