%%------------------------------------------------------------------------
\subsection{Penalty of a Cache Miss}
%%------------------------------------------------------------------------
\label{sec:performance}
%%------------------------------------------------------------------------

Since \start/'s support for \ccdsm/ is mostly in software, a cache
miss on global data incurs a significant penalty.  The penalties of
cache-misses are shown in Figure~\ref{fig: miss penalties}.  The times
are given in processor cycles and are our current best estimates.  The
corresponding numbers for \flash/, as reported in~\cite{Heinrich94}
are given for comparison.


\begin{figure}
\begin{center}
\begin{tabular}{||l|l|l|l|l|c||c||}
\hline
\multicolumn{5}{||c|}{Type of Miss}             & \start/       & \flash/ \\
\cline{1-5}
Add Space & Home & L3 hit? & Dir State & Owner & (proc cycles)  & (proc cycles)\\ 
\hline
\hline
Local  & --     & --   & --    & --            & 35             & 54 \\\hline
Global & Local  & --   & Clean & --            & 199            & 54 \\\hline
Global & Local  & --   & Dirty & Remote        & 692            & 286 \\\hline
Global & Remote & Hit  & --    & --            & 157            & 54 \\\hline
Global & Remote & Miss & Clean & --            & 699            & 222 \\\hline
Global & Remote & Miss & Dirty & Home          & 786            & 290 \\\hline
Global & Remote & Miss & Dirty & Remote        & 1219           & 382 \\\hline
\end{tabular}
\end{center}
\caption{Miss penalties in processor cycles.  The \flash/
numbers\protect\cite{Heinrich94} have been normalized to processor
cycles to allow easier comparison with \start/.  The \start/ numbers
assume an 8-site machine where 6 network hops are needed to get to a
remote home and 6 network hops to get to a dirty site from the home.
(see text) }
\label{fig: miss penalties}
\end{figure}

To the first order, the global memory miss penalties for \start/ are
roughly three times those of \flash/, with the penalty for one case
approaching four times.  Our miss penalty for local address space
appears to be faster than \flash/'s partly because our machine will
run at a slower clock rate, making the memory latency a fewer number
of processor cycles.  Though our miss penalties are significantly
higher than those of \flash/, it is important to remember that we only
pay these penalties when we miss in the L1 and L2 caches.  We can
express the relative efficiency of a particular implementation by its
average CPI (clocks-per-instruction) which is computed using the
following equation.

\[CPI_{global\_mem\_ops} = CPI_{mem\_op\_cache\_hit} + miss\_rate
\times miss\_penalty.\]  
\[CPI_{average} = (1 - \% global\_mem\_ops) \times CPI_{non\_global\_ops} + \% 
global\_mem\_ops \times CPI_{global\_memory\_ops}\] 

A low miss-rate and/or a low percentage of global memory operations
will reduce the impact of \start/'s disadvantage.  A recent \flash/
paper\cite{Heinrich94}, which measured the performance of \flash/
using a simulator, gives some ideas about the kind of miss rates that
one may expect.  For the seven programs (mostly SPLASH programs) that
they ran on the simulator, three have miss rates below 0.1\%, another
three have miss rates between 0.1-1.0\%, and only one had the rather
high miss rate of 6.0\%.

To get some idea of what these miss rates mean for performance, we did
some quick back-of-the-envelope calculation for the Average CPI, using
some reasonable estimated numbers for the missing components of the
Average CPI formula.  The numbers we used are:
$CPI_{non\_global\_ops}$ is 0.8; $CPI_{mem\_op\_cache\_hit}$ = 0.8; \%
$global\_mem\_ops$ = 10\%; $miss\_penalty$ = 800 for \start/ and 266
for \flash/.  This exercise shows that for miss rate of under 0.1\%,
the difference in miss penalty between \start/ and \flash/ does not
make any significant difference.  At 1.0\% miss rate, the difference
becomes significant, with \flash/ about 50\% faster.  However, it is
not clear that both machines will have the same miss rate, and any
difference in miss rate around the 1\% neighborhood have great impact
on overall performance.  \start/ should encounter lower miss rates
than \flash/ since each site is an SMP, making the home site of more
global data the local site. The use of a large L3 cache should also
increase the sharing of remote data on a site.  Miss rates that are in
the range of 6\% is not of very great interest because both machines
will be suffering from such high ``parallel slowdown'' that neither
can handle the program with any efficiency.

Thus, for programs that run well on an aggressive implementation like
\flash/, \start/ has the potential of running reasonably.  Clearly
``embarrassingly local'' programs will run equally well on either
machine.  Programs that do not run as well on \start/ as on \flash/ do
exist, but even \flash/ has its limits.  It will be interesting to see
just what kind of performance we can get out of \start/'s minimal cost
\ccdsm/.

%%------------------------------------------------------------------------
%%                             Section End
%%------------------------------------------------------------------------

