%%------------------------------------------------------------------------
\section{Introduction}
%%------------------------------------------------------------------------
\label{sec:intro}
%%------------------------------------------------------------------------


This paper describes the challenges we faced when designing \start/
and their solutions.  \start/ is a parallel machine build from {\em
sites} each consisting of a modified commercial SMP based on the
PowerPC 620.  It is a message passing machine with a minimal amount of
custom hardware to support cache-coherent distributed shared memory
(\ccdsm/).  The amount of custom hardware for coherent shared memory
is kept minimal by dedicating a 620 on each SMP, which we call the
service processor or \sP/, to coherency maintanence.

There are two basic motivations for supporting \ccdsm/ on \start/:
running unmodified SMP codes on multiple sites and experimenting with
coherency protocols to understand their interaction with the memory
models they support.  \start/ will run a stock SMP operating system on
each site.  In order to provide a more useable environment, we wish to
extend virtual memory to encompass global address space.

The problems we will discuss in this paper originated from three
sources: the processor and its snoopy bus protocol, buffering within
the system and the network.  Many of the problems we encountered are
not applicable to all systems.  Some of these are particular to the
way we are implementing coherent shared memory.  Our experience,
however, will point out characteristics of processors that a designer
should look out for.  Certain problems, on the other hand, are general
to all machines.  Without careful design of the buffering of the
system, for example, elusive deadlocks can creep in.  The bulk of the
paper, therefore, discusses these problems in some detail, along with
their solutions.

In the next section, we will give the background information necessary
to understand the challenges we faced.  It will describe particular
aspects of processors, memory buses, buffering and the network without
getting into the problems they may cause.  Section~\ref{sec:start}
describes the \start/ implementation and how we plan to implement
\ccdsm/ with a small piece of custom hardware and a dedicated
processor.  Estimates for the performance of \start/'s \ccdsm/ are
included in that section.  In Section~\ref{sec:ccdsm-issues}, we
discuss the challenges that we encountered while implementing \ccdsm/.
The final section gives some conclusions.

The contributions of this paper are the exploration of very simple
hardware for supporting cache-line granularity cache-coherence, the
identification of the problems of this and other approaches, and the
development of solutions for those problems.


%%------------------------------------------------------------------------
%% Section End
%%------------------------------------------------------------------------


