\section{The computer platforms}
\label{s:compare}

We compare the implementation of the MIT model on two very different
platforms: the CM-5 vector MPP, and the DEC 8400 cache-based SMP.  The CM-5
is well-suited for scientific simulation applications, and it, along with
the Cray Y-MP, is the mainstay of our day-to-day production runs.  We will
be testing a 32-processor configuration of the CM-5.

The DEC 8400 is a very high-end mainstream SMP system, designed for use in
a broad variety of both technical and non-technical situations.  Digital
has already sold several thousand 8400 systems, while in contrast, Thinking
Machines shipped less than 100 CM-5 systems.  We will be testing a 300 MHz,
4-processor configuration of the DEC 8400.

\subsection{Machine architectures}

Both systems have theoretical peak floating point performance well in
excess of 100 MFlop/s, as illustrated in Figure \ref{fig:mflop-vs-bw}.
However, the two systems have radically different architectures, which we
discuss here in more detail.

\begin{figure}
\illustrationx{3.9in}{figures/cm5.eps}
\caption{High-level architecture of the CM-5}
\label{figure:CM-5}
\end{figure}

The CM-5 MPP\cite{TMC91} was one of the first parallel computers which
could routinely out-perform traditional vector supercomputers for real
scientific problems.  The machine is built from relatively modest vector
processors connected through two communications networks, as shown in
Figure \ref{figure:CM-5}.  Each vector unit runs at 16 MHz and can issue a
multiply-add instruction each cycle, giving it a peak performance of 32
MFlop/s.  Each CM-5 node contains 4 vector units, yielding a peak
performance of 128 MFlop/s per node.
In real applications, however, sustained performance of
10-30 Mflop/s per node is more typical.

The key to exploiting the CM-5's vector units is to express computation in
terms of data-parallel vector operations, which is a natural paradigm for
regular finite difference/volume models. The vector units act as both
memory controllers and vector floating point units -- each vector unit can
read or write one 64-bit word of memory every cycle, as well as performing
one floating point operation.

Main memory on the CM-5 is distributed across the vector units and so any
sort of inter-processor communication on the CM-5 is relatively expensive,
motivating the programmer to structure codes to minimize inter-processor
communication.  Often, this entails a tradeoff of extra memory use for less
communication.

The architecture of the DEC 8400 \cite{Fenwick95} is typical of the current
generation of microprocessor-based SMP's.  As shown in Figure
\ref{figure:smp}, the DEC 8400 consists of nodes connected by a shared
system bus.  Sharing of data between the processors can only occur through
reads and writes to memory.  Each node of the 8400 consists of an Alpha
21164 microprocessor \cite{Edmondson95} with three levels of cache -- the
first two levels of cache are on-chip, and the third level of cache is
off-chip, but on the same board.  The caches are small, fast-access memories
which mirror sections of the computer's larger, slower access main memory.
Computation using data from cache is much faster than using data from main
memory.  As the caches are situated farther from the processor core, they
are successively larger and slower.

\begin{figure}
\illustrationx{3.7in}{figures/dec8400.eps}
\caption{
High-level architectures of the DEC 8400 SMP}
\label{figure:smp}
\end{figure}

Other SMP's and even successive generations of DEC SMP's may have different
sized caches, or even a different number of levels of caches.  However, all
microprocessor-based SMP's have one or more levels of cache and a shared
view of main memory.

The 8400 and other mainstream SMP's rely on caches to get good performance.  
If a memory reference from the processor is returned by the 1st-level cache, the
processor can continue working on the next cycle.  References which are
returned by the 2nd or 3rd level cache cause the processor to wait longer,
and references to the main memory can cause the processor to wait for tens
of cycles.  Each Alpha microprocessor runs at 300 MHz, and if all of the
operands are available, the Alpha can perform one floating point operation
(which may be a multiply-add) per cycle, thus having a peak performance of
600 MFlop/s.

The shared memory architecture of the 8400 allows for lower communication
costs than on the CM-5.  However, if the problem does not fit into the
8400's caches, performance may be very bad, so it is important to minimize
memory use.  It should be noted, however, that main memory itself can be
very large.

\subsection{Programming environments}

Our goal is to make a model that is accessible to the general scientist, as well
as to specialist scientific programmers and numerical modelers. A
high-level, intuitive scientific programming environment is therefore
of interest to us.

Motivated by this goal, we implemented the model on the CM-5 in
CM Fortran (CMF) \cite{CMF}, which is a high-level, data-parallel dialect
of Fortran.  For the DEC 8400, we implemented the model in 
High Performance Fortran (HPF) using Digital's native
High Performance Fortran (HPF) compiler.\cite{DEC-HPF} HPF is very similar
to CMF because the design of the HPF language standard was strongly
influenced by CMF.

HPF compilers are available for most SMP platforms, provided either by the
hardware vendor or third-party compiler vendors.  However, the quality of
HPF compilers can vary widely -- although HPF compiling technology is
fairly mature in general, some hardware vendors have not made a full
commitment in manpower and other resources to develop high-quality HPF
compilers for their platforms.  Furthermore, in our experience, third-party
compiler vendors are at a disadvantage in developing high-quality compilers
for any particular platform because (1) they must support several
platforms, and (2) details of the hardware necessary to obtain maximum
performance are usually only well-understood by the hardware vendor.

The state of affairs is improving, but any decision to move to a new
platform should be made only after evaluation of both hardware capabilities
as well as software support.
