\Section{Comparison to CM-5 implementation}
\label{Section:comparison}

\begin{fig}{figure:cm5-comparison}{PCG efficiency on the CM-5 versus Monsoon.
(Pacific ocean, $ib=5$, $jb=3$, 2 processor Monsoon, 32 processor CM-5)}
\illustrationx{3in}{figures/cm5-comparison.ps}
\end{fig}

To put the performance of the PCG on Monsoon into perspective, we compare
it to a version of PCG being used in daily production runs of the ocean
code.  The GCM code has been implemented in CM Fortran \cite{CMF} for the
CM-5 \cite{CM-5}.  CM Fortran is a data parallel dialect of Fortran, which
is similar to HPF.  We use the same measure of efficiency to compare the
data parallel implementation to the dataflow implementation: cycles per
required floating point operation.  Note that the data parallel model can
also be implemented efficiently on dataflow machines \cite{shaw92b}, but
our Id implementation uses a multithreaded computational model.  For both
the Id and CM Fortran implementations, the number of required floating
point operations is identical.  The cycles we are counting in the CM
Fortran case are CM-5 vector unit cycles, because the CM-5 vector units
perform almost all of the work in this application and most data parallel
scientific applications.

Figure \ref{figure:cm5-comparison} shows the performance of PCG on a
variety of ocean geometries.  For a shallow, 4-layer Pacific ocean, the Id
version executes fewer overhead instructions per required floating point
operation.  As we increase the depth (and therefore problem size) of the
ocean geometry, the efficiency of the CM Fortran version also increases,
due to longer vector lengths.  We were unable to run a larger than 12-layer
ocean on Monsoon because of heap memory limitations.

Most of the efficiency on the CM Fortran version gained by adding ocean
layers is reached at 12 layers -- the 48-layer ocean has about the same
efficiency as the 8-layer ocean.

Further increases in the floating-point efficiency can be extracted when
the geometry is fitted to the machine such that there is no additional
array padding to even out array distribution across the processors.  If the
ocean geometry is $192 \times 96 \times 48$, the data parallel code
executes in exactly the same amount of time as the $168 \times 90
\times 48$ version of the code, but executes 22\% more floating point
operations. \footnote{Note that the seemingly unusual $168 \times 90$
problem size is a real size used in production runs, and was not chosen to
make the Monsoon performance seem relatively better than the CM-5.}

The ``raw'' vector performance of the CM-5 on PCG (i.e.\ without the 60\%
water overhead or array padding) is one floating point operation every four
cycles.  Thus, each 16MHz vector unit can produce 4 MFlop/s, which is $1/4$
of the peak performance of the CM-5.  There are few realistic applications
which achieve this level of performance.  Additional overheads from
padding, computing on land, and short vector lengths add up quickly to
degrade performance by a factor of 2 to 5.  For most real geometries, we
expect that performance of GCM will be closer to 10-20 cycles per required
floating point operation than the idealized peak performance of 4 cycles
per floating point operation, which is comparable to the performance we
achieved on Monsoon.
