%=====================================================================
\section{\start/ Architecture Overview}
\label{sec: arch}
%=====================================================================

% \start/, a collection of SMPs
% Each SMP site, NES placement

\start/ is a parallel system designed to support both message passing
and coherent distributed shared memory.  \start/ is constructed from a
collection of commercial SMPs connected by a fast network which
interfaces to each SMP's coherent memory bus via a Network Endpoint
Subsystem (NES) card.  Each SMP, referred to as a {\em site}, is a
desktop-class, commercial, dual-PowerPC 604 SMP.  It uses a stock
PC/workstation class motherboard which can accommodate two processor
cards.  Each processor card normally contains one 604 processor and an
in-line L2 cache.  For \start/, we replace one of the two processor
cards in each SMP site with an NES card, making each site into a
uniprocessor system\footnote{We desired multiple processors per site,
but currently no such PowerPC 604 motherboard exists at a reasonable
price.}  This 604 processor, referred to as the application processor,
or {\em aP}, runs a copy of AIX, augmented with a parallel layer to
coordinate parallel job execution.

% \arctic/ network

Each NES card is attached to the \arctic/ network, a Fat-tree
\cite{Leiserson85} network constructed with \arctic/ router
chips\cite{Boughton94}.  Both the \arctic/ router chip and the network
boards are designed and implemented at \mit/ as part of the \start/
project.  Each \arctic/ router chip is a 4x4 packet-switched router with
support for variable length packets, virtual-cut-through, FIFO
ordering, and two logical networks.  The \arctic/ network employs
hardware link-level flow-control to provide a lossless packet routing
service.  Each link is 16 bit-wide and runs at 80MHz to achieve 160
MBytes/sec bandwidth.  With two links, one in each direction, between
each NES card and the network, each site has a peak network
communication bandwidth of 320 MBytes/sec.  A fast router together
with the Fat-tree topology allows the \arctic/ network to provide
redundancy\footnote{Multiple paths through a Fat-tree network between
most pairs of source-destination sites provide redundancy and hence
fault tolerance.}, the ability to exploit network locality and very
high bisection bandwidth -- over 5 GBytes/sec in a 32 node machine.

% NES parts: sP subsection and NES core
% * message queue buffers in SRAM banks

\begin{figure}
\illustrationpercent{sT-site-ov}{35}
\caption{A \start/ site.} 
\label{fig:start-site-ov}
\end{figure}

Figure~\ref{fig:start-site-ov} shows the organization of a \start/
site, with details of the NES.  The NES manages several regions of
memory, each providing a different functionality.  The NES is designed
for both speed and flexibility.  The NES Core provides dedicated
control hardware and data paths for the most common operations.
Actions that are expected to occur very frequently are handled {\em
completely} in hardware and are thus very fast.  An embedded
processor, referred to as the service processor ({\em sP}) provides
flexibility and extensibility.  The sP is organized with its own
sub-system comprising of off-the-shelf parts: a PowerPC 604 processor,
a memory controller, and DRAM.  Design complexity is further reduced
by the symmetry of the organization: from the NES Core's perspective,
the sP-subsystem is almost identical to the aP complex.  The sP
subsystem, however, has additional functionality.  The sP observes all
aP memory operations to one of the NES address regions called the {\em
sP Serviced Space}, and is capable of initiating transactions on the
aP-bus, controlling any NES state, and handling all exceptional
situations.  These features are extremely flexible and can be used to
extend the functionality of \start/.  They will be used to implement
coherent shared memory and {\em non-resident message queues} (see
Section~\ref{sec: multitasking}).  Both the aP and sP have access to
the same message passing mechanisms, but each has its own set of
resources.  Two banks of dual-ported SRAM, the aSRAM and sSRAM,
provide storage space for resident message queues.

% * Control functionalities and state, including message queue state,
%   resides in NESCtrl chip
% * I-bus
% * CRC, destination address translation, addition of source node, and
%   expansion/contraction of Express messages done in TxU and RxU.  
% * a note on \arctic/ interface
% * a note on implementation technology
% * say going to disregard the clsSRAM. 

The NESCtrl, a small ASIC, implements all the hardware control
functionality.  It also holds all the control state, including the
queue pointers of message queues.  The I-bus, the central data path of
the NES core over which all messages flow, has a 64 bit-wide data path
running at 50MHz.  This roughly matches both the bandwidth of the 60X
bus which runs at the same speed and has the same data path width, and
the 320 MBytes/s aggregate bandwidth between an NES and the \arctic/
network.  Outgoing messages undergo destination address translation
(See Section~\ref{sec:trans}) and are tagged with a source identifier
in the Transmit Unit (TxU), which also computes and appends a CRC to
the message packet.  This CRC is checked at each \arctic/ stage, and
again in the Receive Unit (RxU) at the destination NES.  The TxU is
also responsible for the expansion of an Express message
(Section~\ref{sec: short-msg}) into an \arctic/ packet, while the RxU
performs the compression.  Finally, the \arctic/ Interface handles
low-level signaling and analog electrical conversions (TTL to ECL)
between the NES and the \arctic/ Network itself. The clsSRAM, which
stores cache-line state bits in coherent distributed shared memory
implementations, is not involved in message passing.

%=====================================================================
%                                END
%=====================================================================