\renewcommand{\editdate}{Nov 21 15:39 1997}
\renewcommand{\mytitle}{arch-ov.tex}
%\markboth{Draft: \mytitle \ \ \ \  Last Edit: \editdate }{Draft: \mytitle \ \ \ \ Last Edit: \editdate }
%==================================
%=====================================================================
\section{\start/ Implementation}
\label{sec:arch}
%=====================================================================

The high-level design outlined in the previous two sections is
implemented in the \start/ system.  Each site is a desktop-class,
commercial, dual-PowerPC 604e SMP.  The processor card contains one
604e processor and an in-line L2 cache.  The \start/ system replaces
one of the two processor cards in each SMP site with an NIU card,
making each site into a uniprocessor system\footnote{We desired
multiple processors per site, but could not find an adequate PowerPC
604 motherboard for our purposes.}.  Each site runs AIX extended to
handle Voyager address mapping requirements and a scheduler for
parallel job coordination.

Each NIU card is attached to the \arctic/ network, a fat-tree network
constructed with \arctic/ router chips\cite{Boughton94} which were
designed as part of this project.  The \arctic/ chip is a 4x4
packet-switched router which supports variable length packets, with
two priority levels.  It does virtual-cut-through routing and has
support for maintaining FIFO ordering under user directive.  Each link
is 16 bit-wide and runs at 80MHz to achieve 160 MBytes/sec bandwidth.
With two links, one in each direction, between each NIU card and the
network, each site has a peak network communication bandwidth of 320
MBytes/sec.

\begin{figure}
\illustrationpercent{site}{35}
\caption{A \start/ site.} 
\label{fig:start-site-ov}
\end{figure}

Figure~\ref{fig:start-site-ov} shows the organization of a \start/
site, with details of the NIU.  The \sp/ is organized with its own
sub-system comprising of off-the-shelf parts: a PowerPC 604e
processor, a memory controller, and DRAM.  Design complexity is
further reduced by the symmetry of the organization: from the NIU
Core's perspective, the \sp/ subsystem is almost identical to the \ap/
complex.  Both have access to the same message passing mechanisms, but
each has its own set of resources.  Two banks of dual-ported SRAM
provide scratch memory. The clsSRAM provides three HAL bits per cache
line (Section~\ref{sec:enhancedHAL}) and is accessed by the \ap/ snoop
device to implement CCDSM protocols.

The NIUCtrl, a small ASIC, implements all the hardware control
functionality and the enhanced hardware functionality (Section
\ref{sec:enhancements}) and contains all the control state, including the
producer/consumer message buffer pointers.  A total of 512 queues per
site are supported for all communication. This number is dictated
largely by our desire to encode the destination RxQ's in small enough
space to fit the \arctic/ message headers. \express/ messages consist
of 32 bits of data, 5 bits of tag, 9 bits for RxQ specifier, and 1 bit
for priority.  \tagon/ messages specify either one or two cache lines
to be appended as well. Scratch memory is organized to hold 16 pairs
of resident transmit and receive queues (8 \express/ / \tagon/ and 8
Basic).

All messages in the NIU core flow over the the I-bus, which is 64
bit-wide and runs at 50MHz. This roughly matches both the bandwidth of
the PowerPC 60X bus which runs at the same speed and has the same data
path width, and the 320 MBytes/s aggregate bandwidth between an NIU
and the \arctic/ network.  Outgoing messages undergo destination
address translation (Section~\ref{sec:enhancements}) in the Transmit
Unit (TxU), which also computes and appends a CRC to the message
packet.  This CRC is checked at each \arctic/ stage, and again in the
Receive Unit (RxU).  The TxU also formats messages into \arctic/
packets. The \arctic/ Interface handles low-level signaling and analog
electrical conversions (TTL to ECL) between the NIU and the \arctic/
Network itself.
 
The Arctic network has been operational since September 1997.  The NIU
card is expected to be fabricated by January 1998.  Since the OS
extensions and cache coherence protocols have been under development
for some time, we expect the whole system to be operational in the
spring of 1998.  

%=====================================================================
%                                END
%=====================================================================


