\section{Conclusions}
\label{sec:conclusion}

A 32-node \start/ is currently being built and is expected to be
running both message passing and coherent distributed shared memory by
mid 1997.  Despite the usual impact on performance that comes when
implementation must meet real-world demands, our detailed simulations
indicate that \start/ is expected to provide superior performance
while supporting the \3m/ necessary to make the machine generally
usable, a claim that few other machines (if any) can make.  The
simulations have shown other facts as well.  Express messages are
significantly faster in all metrics than any other mechanism for
messages of about a word.  We believe that a substantial fraction of
messages in many codes could exploit this mechanism, increasing the
importance of Express message performance.  \tagon/ with Reclaim
significantly outperformed Basic messages in all metrics in a
multi-threaded environment and equaled Basic messages in a
single-threaded environment.

Express/\tagon/ messaging is achieved by combining two well known
techniques, namely (i) using a single atomic memory operation to
indicate an event and (ii) a FIFO push/pop abstraction rather than a
random access memory abstraction, along with a lesser known technique,
address bit stealing.  This combination of techniques should be
considered for any processor-to-bus-device communication requiring
thread-safety.

Real-world considerations restricted the amount of SRAM embedded in
the NES. Fortunately, in addition to providing hardware protection,
the virtual queue mechanism enables support for a large number of
queues of which only a small number can fit in SRAM.  Such tradeoffs
are probably not limited to \start/.

As a note to processor and system architects, attaching specialized
hardware to the memory bus promises great performance benefits.  This
performance is often limited due to inadequate facilities; for
example, inexpensive mechanisms to flush cache-lines, achieve control
over memory operations ordering and synchronize would be of great
value.  A better processor architecture would enable \start/ to
achieve even better performance.
