\renewcommand{\editdate}{Nov 24 03:47 1997}
\renewcommand{\useredit}{rudolph}
\renewcommand{\mytitle}{introduction.tex}
%\markboth{Draft File: \mytitle \ \ \ \  Last Edit: \editdate(\useredit)}{Draft File: \mytitle \ \ \ \ Last Edit: \editdate (\useredit) }

\section{Introduction}
\label{sec:introduction}
%~~~~~~~~~~~~~~~~~~~~

It is generally believed that scalable parallel computers will consist
of commodity SMP servers augmented with an intelligent interface to a
high-performance interconnection network.  Most designs of the network
interface unit (NIU) directly support only one communication mechanism
or one memory model\cite{Lenoski92,Exemplar93,Kessler93,Numa-q}, and
thereby limit the functionality and performance.  This paper presents
an architecture that overcomes these limitations and provides
mechanisms to support a variety of message passing and shared memory
models in an open system.

Shared memory and message passing (S\&M) provide different programming
abstractions and the debate as to which is better has been raging for
more than a decade with no clear winner in sight.  Each appears to
have its advantages, and each is likely to be around for a long
time\cite{Kranz93}.  Unfortunately, while it is generally accepted
that large distributed shared memory systems (DSM) should be built on
top of a message passing substrate, DSM systems do not expose this
substrate.  Similarly, in systems that only support message-passing,
it is difficult to implement cache-coherent DSM since there is no
access to internal hardware cache state.  Employing two different
hardware solutions works but is expensive and wasteful both in terms
of development time and hardware costs.  An appropriate architecture
should support applications with complex communications interactions
without compromising on protection.


As SMP's become commonplace, they not only become the natural basic
building blocks of a larger multiprocessor, but also enable an {\em
open} scalable parallel processing system.  An open system allows
commodity items to be easily assembled and replaced without hardware
modifications.  They avoid the difficult task of tightly coupling the
development and delivery of both the processing nodes, the network,
and the processor-network interface unit (NIU), Machines such as Flash
\cite{Kuskin94} and Alewife
\cite{Agarwal95} share  goals similar to \start/'s but are not open
systems.  Connecting the NIU to a standard I/O bus is the typical way
of providing an open system
\cite{Agerwala95,Stunkel95,Boden95,Gillett96,Blumrich94a,Hoe94}, but
current I/O bridges severely limit the performance and do not 
expose enough processor cache state to build cache coherent DSM's.
The multiple processor slots of SMP's make ideal attachment points and
do not require modifications to the SMP architecture.  However,
cooperation with the SMP producer is required because the memory bus
interface, unlike the I/O bus, tends to be proprietary.

A minimal hardware system meeting our design goals is an NIU
consisting of an  embedded microprocessor and interface units to the SMP and the
network.  As the number of processors within the SMP and the number of
concurrent, interactive, parallel jobs increase, the embedded
microprocessor is likely to become a performance bottleneck.  The solution is
to migrate common case functionality to specialized functional units.

The \start/ machine embodies these features in an actual
implementation consisting of 32 IBM Doral PowerPC 604e-based dual SMPs
and the Arctic \cite{Boughton94} high performance interconnection
network.  The \start/ NIU contains an embedded processor with hardware
assistance to enhance performance. {\em \start/ can implement
virtually any type of message-passing, shared memory, or
synchronization primitives found on any multiprocessor with
performance competitive to that of a direct implementation.}  It also
allows easy extensions to the semantics of memory access operations.
\start/ is expected to be operational in the spring of 1998.

We begin with a discussion of the requirements for the NIU for open S\&M
systems (Section \ref{sec:requirements}) and then present a minimalist design for
such an NIU in Section \ref{sec:minimal}.
The paper suggests various hardware enhancements for improving
performance in Section \ref{sec:enhancements}. The actual \start/
design is given in Section \ref{sec:arch}, and its projected
performance, based on synthesizable Verilog code, is given in Section
\ref{sec:performance}. Finally we present some conclusions in Section
\ref{sec:conclusion}. 



%%%%%%%%%%%%%%%%%%%
%  End of Section %
%%%%%%%%%%%%%%%%%%%










