\section{Introduction}
\label{s:introduction}

During the last twenty years, the application of supercomputers to the
numerical study of fluids, in particular the atmosphere and ocean, has been
an unquestionable success, paving the way for vast improvements in the
fidelity of computational fluid simulations.  Throughout this period,
low-volume, high-end vector supercomputers have clearly led in absolute
performance and vector computers have been an indispensable tool in meeting
the seemingly insatiable demands of computational fluid dynamics (CFD).
Recently, however, commodity microprocessor-based multiprocessors have
reached a level of performance where they can be seriously considered as an
affordable alternative to specialized, purpose-built, and hence generally
expensive, vector supercomputers.

Today there is a widely articulated viewpoint that mainstream symmetric
multi-processor (SMP) architectures, based on the same commodity
microprocessor technology used in desktop workstations, will become the
building blocks for a new generation of high-end, scalable supercomputer
platforms.  Computer architects now enthuse over the possibility of
multiprocessor supercomputers based on commodity microprocessors delivering
significant performance to numerical applications at a fraction of the cost
of traditional vector supercomputers.  Skeptics question the soundness of
this conviction, arguing that vector processing is a natural paradigm for
core supercomputer fields such as computational fluid dynamics, and that
many legacy applications do not perform well on current generation
commodity platforms\cite{McKee,McCalpin}. The skeptic's argument is
compelling.  Current generation commodity microprocessor platforms all rely
on hierarchical cache-based memories to give them high performance --
however, this memory organization performs poorly with the vector-centric
numerical procedures that lie at the heart of many supercomputer
applications, including most meteorological and oceanographic applications.

\begin{fig}{fig:mflop-vs-bw}{Peak floating point performance versus main
memory bandwidth.  Peak floating point performance is the maximum floating
point operations which can be issued in a cycle multiplied by the cycles
per second.  Main memory bandwidth is measured using the STREAM benchmark.}
\begin{center}
\illustrationx{4.8in}{figures/mflop-vs-bw.ps}
\end{center}
\end{fig}

Figure \ref{fig:mflop-vs-bw} illustrates one important architectural
difference between vector, massively parallel processor (MPP), and
mainstream workstation/SMP platforms.  The Cray vector computers (Y-MP,
C90, and T94) are a yardstick for the performance of vector supercomputers.
They show a close to one-to-one balance between main memory bandwidth (as
measured by the STREAM benchmark\cite{McCalpin}) and peak floating point
performance, with both increasing proportionately over time.  Such a
balance makes it possible for Cray and other similar vector supercomputers
to fetch an array element and to perform a floating point operation on an
array element each cycle.

The two platforms we consider in this paper, a 32-processor CM-5 and a
4-processor DEC 8400/300, are also shown on Figure \ref{fig:mflop-vs-bw}.
The CM-5 is a vector MPP supercomputer, and accordingly it shows a
one-to-three balance between main memory bandwidth and floating point
performance -- not quite as evenly balanced as the Cray vector computers,
but significantly more balanced than cache-based workstations or SMP's.  As
the CM-5 scales to larger configurations, the balance remains roughly
fixed.  The DEC 8400, on the other hand, shows comparable peak floating
point performance to even the current-generation of Cray vector machine,
but has significantly lower main memory bandwidth.  The DEC 8400 has about
a one-to-thirty balance between floating point and main memory performance.

Generally, oceanographic and atmospheric models, either grid-point or
spectral, are relatively straightforward to map efficiently to
architectures like the Crays, with their even balance between main memory
bandwidth and floating point performance.  Typically, the models use
low-order finite difference schemes, which can be expressed naturally using
vector-processor-friendly constructs with a roughly equal ratio of floating
point operations to memory references.  As Figure \ref{fig:mflop-vs-bw}
shows, such constructs are not well-suited to the DEC 8400 or other
cache-based microprocessor platforms, because the main memory bandwidth
soon becomes the limiting factor.

Nevertheless, there are numerical codes for which the performance of a
4-processor DEC 8400 is comparable with a top-of-the-line vector platform.
The LINPACK\cite{LINPACK1,LINPACK2} benchmark rates a 4-processor DEC
8400/300 at 1351 MFlop/s and a 1-processor Cray T94 at 1603 MFlop/s, for
the hand-optimized LINPACK 1000 test.  A workstation of the same era as the
Cray Y-MP, on the other hand, had both low floating point and low main
memory performance, and would have been unsuitable for any form of
large-scale computational fluid dynamics.  The rapidly increasing
performance of mainstream microprocessor computers, along with their
relatively low costs, makes it tempting to investigate the role they might
play in large-scale meteorological and oceanographic modeling, and to
anticipate how that role might evolve in the future.

In this paper, we report on the design, implementation and evaluation of
the hydro-dynamical kernel of the MIT ocean-atmosphere model on a
mass-market SMP platform, the DEC 8400 5/300.  We compare this
implementation with an existing implementation on a CM-5 MPP currently used
for daily production runs of the MIT model.  A vision of ``personal
supercomputers'' as effective research and teaching tools underlies this
study, and so we limit our experiments to high-level programming
environments that would be readily accessible to researchers and students
who are not necessarily expert scientific programmers or numerical
modelers.  Our study employs a hierarchy of problems, culminating in the
MIT model hydro-dynamical kernel, to probe the strengths and weaknesses of
the platforms, and to identify promising implementation strategies.
