%(=====================================================================
\section{Projected Performance}
\label{sec:performance}
%=====================================================================

This section presents the estimated performance of \start/ using the
three LogP\cite{Culler93} metrics:
%
\begin{itemize}
\item{{\bf Processor overhead:}
Processor overhead captures the overhead incurred by application code
when sending and receiving messages.  These were generated by running
representative code on a PowerPC 604 machine.  To model accesses to
NES SRAM, the test code is crafted to cause L1 misses which hit in L2
to simulate NES SRAM accesses.  Several areas where the machine on
which measurements were taken differed from
\start/ are: a 2:1 processor/bus ratio instead of 3:1; and a look-aside
L2 cache instead of in-line L2 cache. }
%
\item{{\bf Latency:}
Latency measures the one-way end-to-end latency, including the
software overhead for sending and receiving messages.  The processor
overhead component is obtained as described above; the latency through
the NES and network is obtained with a cycle-by-cycle RTL C simulator.
This simulator is extremely accurate as it also serves as the hardware
specification for our hardware implementors.  The latency reported
includes the time for one network switch (2 hops).
}
%
\item{{\bf Gap/bandwidth:}
Gap measures the sustainable interval between messages of a specific
type.  The inverse of gap multiplied by message size gives bandwidth.
We obtained two sets of gap numbers, one from our RTL C simulator
showing the bottleneck in the NES and network, and another from the
Processor Overhead as measured by our test program.  The larger of
these two number is the gap number reported. 
}
\end{itemize}
%
We focus on the performance of small to medium size messages as these
are the more challenging cases.  Large block transfers are efficiently
handled by our hardware supported DMA, where performance is limited by
the bandwidth of the bus and network and not the NES.

%----------------------------------------------------------------------
\subsection{ Performance in a Multi-threading Environment }
\label{sec:mT-perf}
%----------------------------------------------------------------------

\begin{figure}
\begin{center}
\illustrationpercent{mt0}{50} 
\end{center}
\caption{ Processor overhead and latency in a multi-threading environment.} 
\label{fig:mt0}
\end{figure}

\begin{figure}
\begin{center}
\illustrationpercent{mt1}{50} 
\end{center}
\caption{ Gap and bandwidth in a multi-threading environment.} 
\label{fig:mt1}
\end{figure}

Figures~\ref{fig:mt0} and~\ref{fig:mt1} show the performance of the
various message passing mechanisms in a multi-threaded environment
where a message queue is shared between several threads.  The
horizontal axis is message size in units of 4B words; the vertical
axes is in bus clocks (20 ns) for all but Bandwidth, which is in
Bytes/sec.  Each graph includes six different cases.  Most of the them
are self explanatory.  ``No Reclaim'' vs ``Reclaim'' refers to how
cached message data is kept coherent.  The no Reclaim case involves
software using explicit Flush instructions to push message data out of
the caches.  For the Reclaim case, the NES extracts transmit data from
the aP cache and flushes old receive data from the aP cache.  In
Reclaim, the NES does not implement full consistency, but performs
coherence operations when notified by pointer update that messages has
been sent or received.  Such lazy coherence requires no cache-tags and
is easy to implement.  ``\tagon/, SRAM Resident, no Reclaim'' refers
to the case where the data that is ``tagged on'' is already in the NES
SRAM.  This is the case for the second and subsequent destinations of
a multi-casted message.

For Basic message queues, correct operation requires locking the queue
when sending or receiving a message.  Owing to their inherent
thread-safe design, Express and \tagon/ messages queues do not require
locking.  As shown in the figures, \tagon/ messages have the lowest
processor overhead, latency, and highest bandwidth across almost the
entire range of message sizes.  Basic messages is always worse than
\tagon/ messages.  Comparing \tagon/ with Reclaim and Basic with
Reclaim, one sees that processor overhead for Basic is more than twice
that for \tagon/.  In fact, the performance numbers for Basic message
does not take into account actual lock contention between multiple
threads; the current numbers only account for the cost of successfully
obtaining a lock on first try.

The performance numbers also shows that Reclaim support for \tagon/
and Basic messages is always superior to having the processor
performance explicit flushes.  We had expected better processor
overhead but worse latency with Reclaim support, however the high cost
of explicit software Flush makes Reclaim competitive even for latency.

For very small messages, Express messages are superior; the cross-over
point from Express to \tagon/ messages is about two words.  This
cross-over point is lower than we had expected and is due to the fact
that uncached accesses are relatively inefficient compared to
cache-line burst transfers both in its bus and bus interface buffer
utilization.  Nevertheless, in certain applications, such as
cache-coherent distributed shared memory protocols, single-word
messages are so common, they warrant special consideration.  A
mechanism with such low overhead and latency will likely be used by
applications that previously did not use small messages because
of their high cost.

%----------------------------------------------------------------------
\subsection{ Performance in a Single-threading Environment }
\label{sec:sT-perf}
%----------------------------------------------------------------------

\begin{figure}
\begin{center}
\illustrationpercent{st0}{50} 
\end{center}
\caption{ Processor overhead and latency in a single-threading environment.}
\label{fig:st0}
\end{figure}

\begin{figure}
\begin{center}
\illustrationpercent{st1}{50} 
\end{center}
\caption{ Gap and bandwidth in a single-threading environment.} 
\label{fig:st1}
\end{figure}

Though \tagon/ messages are better than Basic messages in a
multi-threaded environment, are they still competitive in a
single-threaded environment?  Figures~\ref{fig:st0} and~\ref{fig:st1}
show the performance of the different message types when used in
single-threaded fashion.  Because locking is no longer necessary,
Basic messages now deliver comparable, and sometimes slightly better
performance than \tagon/ messages.  The difference is not significant
in most cases.  Reclaim remains a better choice than explicit software
Flush, while Express message is still better for messages of one word.

With little penalty in the single-threaded scenario and significantly
better performance in the multi-threaded case, \tagon/ is a better
mechanism.  Basic message does have some benefits which are not
reflected in the performance numbers.  The use of pointers allows
aggregation, so that when several Basic messages are sent out or
received together, some savings in handshake between application code
and NES is possible.  

The absolute performance of the message Passing mechanism in \start/,
with message latency of between 1 to 5 $\micro$s, bandwidth of over
100 MBytes/sec with messages of only 88 Bytes is far superior than all
but the very best supercomputers today.


%=====================================================================
% END
%=====================================================================
