\section{Conclusion}
\label{sec:conclusion}

We have presented a design for an open, scalable cluster system
supporting both shared memory and message passing.  For economic
reasons, scalable parallel machines in the near future are likely to
be constructed this way.  The main contributions of this work are:
%
\begin{itemize}
\item{ 
A communication architecture which efficiently layers shared memory
implementation on a high performance, multi-user message passing
substrate. }
\item{
An integrated, configurable hardware and firmware CCDSM implementation 
platform which permits multiple user-defined coherence protocol to be
operational simultaneously. 
}
\item{
A memory-bus based message-passing interface supporting several
data-transfer mechanisms, each suitable for different
message granularity, data location, and programming model.
}
\item{
Use of inherently atomic single uncached reads and/or writes at the
``commit points'' of message passing operations to achieve a
thread-safe interface.
}
\item{
Use of a hardware-enforced, private, virtual message destination
name space as a general, flexible means for enforcing protection and
facilitating job migration in a multi-tasking environment.  }
\item{
Direct support for a large number of simultaneously active message
queues, implemented in a high performance and cost-effective way
with Resident and Non-resident queue resources. }
\item{
A low-overhead one-poll mechanism that returns the highest priority
message from a set of message queues.}
\end{itemize}
%

The biggest difference between our design and other related work is
our insistent that protection, high performance, and flexibility for
both shared memory and message passing must all be addressed in one
single design.  Most earlier designs have less ambitious requirements,
usually assuming a single communication mechanism is adequate for most
parallel applications.  Those that are support both shared memory and
message passing, \eg/ the Alewife machine, do  not consider system
level issues like protection and multitasking.

Our use of a commodity microprocessor to execute coherence protocol
code (also used in Typhoon\cite{Reinhardt94}) is controversial.  Many
feel that global cache-miss latency would be too high~\cite{Michael97}
to be practical, motivating designers of commercial DSM's such as the
Origin 2000~\cite{Laudon97} to implement the entire cache protocol in
hardware.  

This belief, however, may be based on inefficient designs for hardware
supporting the protocol processor.  For example, requiring the \sp/ to
poll for completion of one event before another can be issued will
increase \sp/ occupancy, dramatically increase latency and complicate
firmware because of the need to continue servicing new requests.  Our
complete design solves this problem with hardware command queues which
accommodate multiple outstanding commands and enforce ordering between
related commands.  The \sp/ can service a request by issuing a small
stream of commands into these queues without waiting for completion of
individual command.

Another insight is the realization that the special NIU hardware units
are rather general, and are often used in many different communication
operations.  Potentially, these can be further developed so the
collection of special NIU hardware eventually becomes a more general
communication processor.  This is the subject of future work.

We are currently in the final phases of hardware implementation and
expect hardware in January 1998.  Once that is up and running, much
experimentation and evaluation will be conducted.  The final version
of this paper will incorporate numbers from those experiments.

