[Beowulf] Hadoop

[Jeff Layton]

Good points (as always). I've seen good examples of people using an interpreted language. It is easy to develop new code and algorithms and test them (I do that quite often). As you point out, this is great for some people who don't need to run the code many times. I've also seen people use the prototypes to develop compiled applications resulting in huge performance gains when they finalize an algorithm or as you point out, need the speed for their particular application.

What I worry about are the "in-between" cases where people stick with interpreted code for whatever reason, even though they need the speed of a compiled application. I've seen example of this all too often. In many of the cases I've seen, these interpreted applications don't evolve and then the users start screaming that their applications don't scale so they need faster hardware, implying $$$.

Ultimately, I think it's a trade-off between $$ spent on rewriting the application for for compiled languages or spending $$ on faster hardware. So, it's a case by case decision :)

Thanks for the comments. Good insight that I missed. Plus, as usual, I've gotten way off track and I hate this silly web based email tool that doesn't have a good way to do quoted or indented replies. :) So I'll stop here.

Jeff




________________________________
From: "Lux, James P" <james.p.lux at jpl.nasa.gov>
To: Jeff Layton <laytonjb at att.net>; Tim Cutts <tjrc at sanger.ac.uk>
Cc: Beowulf Mailing List <beowulf at beowulf.org>
Sent: Sunday, December 28, 2008 1:28:59 PM
Subject: Re: [Beowulf] Hadoop




On 12/28/08 7:17 AM, "Jeff Layton" <laytonjb at att.net> wrote:

> I think I understand why people want the toolboxes - it makes coding easy.
> From what I've seen people then stay with the "prototype" code and never move
> to a compiled language such as C or Fortran. It's been a long, long time, but
> I did all of my code prototyping for my PhD in Matlab and rewrote it in
> Fortran. I was easily able to get a 10x speedup. I guess people don't like 10x
> improvements in performance any more :)

Bearing in mind Hamming's admonition "the purpose of computing is insight,
not numbers", it could well be that for a "research" application (contrasted
with production) you don't need the 10x speedup.  If the slow, easy to code,
version gives you the answers you need in reasonable time, why change.

OTOH, if your dissertation problem requires computation that takes months in
Matlab, you've got two paths (at least):
1) spend some time (less than months) to learn how to code in a faster
style/language/syste
2) reframe your problem so it doesn't require the computation (and then
convince your committee of this, which could take longer than just doing the
computation, eh?)

If you're interested in faster than real time modeling, though, (say, you're
doing real time control in robotics), then fast speed is essential.

>
> An option for people is to use the Matlab compiler to build faster code. I've
> never used it but the reports I've seen is that it works quite well. I'm not
> sure about how the toolboxes work with it - whether you compile them as well
> or they run as interpreted along with the compiled code.

Works quite well (a lot of the toolboxes are compiled as well).. Just don't
expect to look at and debug the output of the compiler, so good test cases
are important.  Ad hoc development styles tend to be dicey.

>
> Octave has a number of toolboxes. I'm not sure if it covers what various
> people are doing, but they are out there.

Octave also does compilation.  That is, just like Matlab, it does a just in
time style of compilation to an intermediate form.  A loop doesn't get
reinterpreted on each pass through the loop.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20081228/01277535/attachment.html>