%=====================================================================
\section{Projected Performance}
\label{sec:performance}
%=====================================================================

Estimated message passing performance of \start/ is measured using the
three LogP\cite{Culler93} metrics.  The numbers are obtained with
micro-benchmarks running on a simulator which models the \start/ NIU
with sythesizable Verilog code.
%
\begin{itemize}
\item{{\bf Processor overhead:}  This is the application software
overhead for sending and receiving the message.  }
%
\item{{\bf Latency:}
We measure the one-way end-to-end latency, including the
software overhead for sending and receiving messages, and 
2 hops on the network (one network switch).
}
%
\item{{\bf Gap/bandwidth:}
Gap measures the sustainable interval between messages.  The inverse
of gap multiplied by message size gives bandwidth. 
}
\end{itemize}
%

Two sets of numbers are presented, when each message queue is only
used by a single thread, and when each message queue is shared by
multiple, dynamically forked short threads. 

Graphs in Figure~\ref{fig:mt0} plot the various performance metric as
the message payload, in units of 4Byte words, is increased.  We focus
on the performance of small to medium size messages as these are the
more challenging cases.  Large block transfers are efficiently handled
by our hardware supported DMA, where performance is limited by the
bandwidth of the memory bus and network and not the NIU.

The performance numbers shows that Reclaim support (\ie/ NIU
explicitly flushes cache lines) for \tagon/ and Basic messages is
always superior to having the processor performance explicit flushes.
We had expected better processor overhead but worse latency with
Reclaim support, however the high cost of explicit software Flush
makes Reclaim competitive even for latency.

For very small messages, Express messages are superior.  The
cross-over point from Express to \tagon/ messages is about two words
and is lower than we had expected. This is due to the inefficiency of
uncached accesses compared to cache-line burst transfers.
Nevertheless, in certain applications, such as CCDSM protocols,
single-word messages are so common, they warrant special
consideration.

\begin{figure}
\begin{center}
\illustrationpercent{performance3eps}{80} 
\end{center}
\caption{Processor overhead and latency in a multi-threading environment.} 
\label{fig:mt0}
\end{figure}

When multiple threads share a message queue, correct operation of a
Basic message queue requires locking the queue when sending or
receiving a message, which shows up as increased processor overhead
and latency.  Owing to their inherent thread-safe design, Express and
\tagon/ messages queues do not require locking and incur no additional
penalty.  With little penalty in the single-threaded scenario and
significantly better performance in the multi-threaded case, \tagon/
is a better mechanism.  Basic message does have some benefits which
are not reflected in the performance numbers.  The use of pointers
allows aggregation, so that when several Basic messages are sent out
or received together, some savings in handshake between application
code and the NIU is possible.

The absolute performance of the message Passing mechanism in \start/,
with message latency of between 1 to 5 $\micro$s, bandwidth of over
100 MBytes/sec with messages of only 88 Bytes is superior to all but
the very best supercomputers
today\cite{Liu94,Martin94,Liu95,Culler96,Chang96b,Hoe96,Karamcheti,Pakin95}.


\subsection*{Shared Memory Performance}

Shared memory performance is highly dependent on cache miss ratios.
By implementing S-COMA in hardware with HAL bits, \start/ has the
capability of having a 64MB L3 cache, which will drastically reduce
capacity and conflict misses in many programs.  

To give an idea of the cost of across network cache miss processing,
Figure~\ref{fig: sm performance} gives the latency breakdown for
servicing a read cache-miss which finds data in the clean state at the
home site.  The overall latency is about 15 times the cache-miss
latency to local DRAM.  Since the \sp/ comes into play only in cases
where the data is not in the huge L3 cache, the extra latency may not
have deep impact on the overall performance.

\begin{figure}
\footnotesize
\begin{center}
\begin{tabular}{|p{2in}|r|}
\hline
Action	 			& Latency\\
				& (bus cycles)\\\hline
\hline
aP bus op capture		&  7\\\hline
Local sP receives request	&  15\\\hline % read latency
sP sends request to Home 	&  12\\       \hline
Network Latency (2 hops)	&  26\\\hline
Home sP receives request	&  15\\\hline
Home sP gathers data and sends response		& 46 \\\hline
Network Latency			&  26\\\hline
Local sP receives reply		& 15\\\hline
Local sP satisfy bus request	& 32 \\\hline
\hline
Total				& 194\\\hline
\end{tabular}
\end{center}
\caption{A breakdown of a shared memory read which is clean at a
remote home site.}
\label{fig: sm performance}
\end{figure}



%=====================================================================
% END
%=====================================================================
