\section{Conclusion}
\label{s:conclusion}

We have implemented a complete hydrodynamical code in HPF on a DEC 8400
SMP.  We have shown that its performance on a 4-processor DEC 8400 is
competitive with a 32-processor CM-5 for problems we run routinely
on the CM-5. We found that the size of a problem which could
be feasibly integrated on a mainstream SMP machine like the DEC 8400 is
limited by the aggregate cache memory size.  For a 4-processor DEC 8400,
performance is sufficient for coarse resolution ($\approx 4^\circ$ horizontal
resolution)
climatological time scale integrations, or for higher resolution (down to
$\approx 1^\circ$) decadal simulation.

To obtain good performance on the SMP, we analyzed and optimized a series
of isolated kernels that are the building-blocks of the full model.  These
test codes highlight factors which affect performance of the full
application.  For example, we see how performance scales with problem size,
machine size, and cache size.  Each test also reveals strengths and
weaknesses of the compiler, and allows us to explore alternative
implementations which may better match the system.  These alternatives
range from simple axis order re-ordering to completely different numerical
procedures such as the substitution of a direct linear solver for an
iterative one.

Achieving good performance on microprocessor-based SMP's depends upon many
factors, including the number of processors in the system, the size of the
caches for each processor, the speed of the individual processors, and the
maturity of the compilation system.  The DEC 8400 represents a successful
combination of these attributes for our application, although other
applications (e.g.~spectral models or elaborate physics packages) may have
different requirements.  We note that processor counts, cache sizes, and
processor speeds for SMP systems can vary significantly from vendor to
vendor, although these hardware characteristics are easy to measure.  Less
easy to measure is compiler technology, which plays an equally important
role in attaining good performance -- it has been our experience that some
systems with adequate hardware do not have compilers which can exploit the
hardware to their full potential.

We are optimistic that microprocessor-based SMP platforms will, for real
oceanographic and meteorological applications, bridge the gap between
frequently over-burdened, centralized supercomputer center facilities and
under-powered desktop workstations.  However, the widespread adoption of
SMP's is contingent on their affordability -- currently, the 4-processor
DEC 8400 system is significantly more expensive than four individual
workstations.  Nevertheless, it seems likely that prices will fall as
high-performance SMP technology gains popularity in the marketplace, driven
primarily by demands of the business community, as well as the scientific
and engineering communities.

Finally, we note that our adherence to a high-level, global programming
model has made it significantly easier to automate the complex data flow
analysis required to differentiate our forward model code. As a result we
are able to apply the TAMC adjoint model generation tool\cite{Giering} to
our HPF code, and generate adjoint and tangent linear models automatically.
Furthermore, through appropriate replication of the data distribution
directives from the forward model, the automatically derived codes
transparently inherit the parallelism of our forward model. Using this
technique we are incorporating our model into a systematic optimization
framework with relative ease.
