% Section on implementation considerations in the ISCA *T paper

\section{Implementation Considerations}

\label{sec-implementation}

So far, {\archname} should be viewed as a {\em logical\/} model for a
node in a multithreaded MPA; specialized, separate coprocessors are
otherwise familiar RISC processors extended with instructions for the
rapid creation and consumption of network messages, and for the
efficient synchronization and scheduling of lightweight threads (in
fact, many of the ``extended'' instructions can be realized as
memory-mapped operations).  In this section we explore possible
physical realizations of the {\archname} node architecture.

\subsection{Coprocessor Data Sharing and Loading Issues}

All three coprocessors share the same local memory, and their view
must be consistent.  Generally speaking, there is a significant
sharing of data between the {\sc sP} and {\sc dP}, but relatively
little sharing between either of these and the {\sc Rmem}. The data
values on {\em every\/} message handled by the {\sc sP} are written
into frames and are subsequently read by the {\sc dP}. It is also
likely that the {\sc dP} will initialize join counters (or query them)
which are subsequently modified by the {\sc sP}.  In contrast, data
sharing between the {\sc dP} and {\sc Rmem} is limited to those
structures which are both globally accessed by other nodes using split
phase transactions, and locally accessed using traditional load/store
operations. Thus, if the three coprocessors are implemented as
distinct units (presumably with each having a cache for local memory)
then we can expect a large volume of coherence traffic between the
{\sc sP} and {\sc dP} coprocessors.

The relative load among the coprocessors will also influence an
implementation. The {\sc sP} typically executes several more threads
than the {\sc dP} but the threads are apt to be proportionately
shorter.  Assuming an average {\cf join} arity of about four messages
corresponding to an average data thread length of approximately 15
instructions, then the two coprocessors should execute roughly the
same number of instructions (assuming three or four instructions per
{\sc sP} message handler).

In terms of network messages, the accounting is straightforward.
Every {\cf rload} or {\cf rstore} request generated by a {\sc dP} is
processed by some {\sc Rmem} which subsequently generates a {\cf
start} message for the requester's {\sc sP}. Added to the remote
memory traffic are {\sc dP}-generated {\cf start} messages
corresponding to function arguments and results. For many applications
the remote memory traffic dominates this extra traffic. Thus, assuming
a uniform distribution of remote memory requests, an {\sc Rmem} will
process about as many messages as an {\sc sP}.

\subsection{Multiplexing the Coprocessors}

While the workload appears nicely balanced among the coprocessors,
engineering considerations may suggest multiplexing them on common
resources. For example, we are actively pursuing a design where the
{\sc sP} and {\sc dP} are combined onto a single chip.  The pipeline
and caches are time multiplexed on a thread-by-thread basis. By
sharing the same data cache the {\sc sP--dP} coherence traffic is
eliminated. Obviously, the throughput of the node will suffer because
we will not be able to exploit concurrency between the {\sc sP} and
{\sc dP}. A more subtle consequence is pressure on the incoming
message queue. Data processor threads are typically several times
longer than message handlers. If data threads execute to completion
before executing message handlers, then the variance on message
service times will cause increased queuing of network messages.

In the ETL EM-4~\cite{Sakai89a}, the Fetch and Matching Unit (FMU)
corresponds to our {\sc sP} and the Execution Unit (EU) corresponds to
our {\sc dP}.  There is no separate {\sc RMem}--- its function is
performed by the EU.  Further, the FMU is not programmable like our
{\sc sP}; rather, it is specialized for various {\em wait-match\/}
operations similar to Monsoon~\cite{Papadopoulos88a} (two-way joins on
frame presence bits, accompanied by a single frame read/write).  Also,
the FMU and the EU share a single port to local memory.

An implementation might go further and multiplex all three
coprocessors onto the same shared resource (as in the MDP). This might
be considered when remote memory operations are network bandwidth
limited anyway.  However, we would still consider this an improvement
over, say, a CM-5 node (represented by Figure~\ref{fig-msg-queue}),
because it still provides exceptionally efficient support for {\cf
join}, {\cf start}, {\cf rload}, {\cf rstore}, and {\cf next}.  That
is, we distinguish {\archname} architectures by their direct
implementation of synchronization and scheduling primitives for a
potentially large set of very lightweight threads {\em in addition
to\/} a tightly integrated network interface wherein internode
communication is exposed to the compiler.

An orthogonal dimension of multiplexing is that multiple threads can
be interleaved in the processor pipeline, as in the
HEP~\cite{Smith78}, the Tera~\cite{Alverson90a} and
Monsoon~\cite{Papadopoulos88a}.  This can simplify the pipeline, since
the pressure due to inter-instruction dependencies is less, but it
also degrades single thread performance by the degree of multiplexing.
