\section{Programming data parallel GCM}
\label{section:programming-fortran}

Given the algorithmic specification in Section \ref{section:ocean-model},
the high-level view of both the data parallel and multithreaded
implementations are very similar.  Both implementations define the same
high-level procedures, and both have the same structure in the
time-stepping loop.

We begin with the data parallel version because we feel that the reader is
likely to be more familiar with it, and many studies have been done on the
performance of the CM-5 and other data parallel platforms.

Although programming models are separate from the hardware they are
implemented on, there is a very close relationship between what the
programmer writes and what is executed on the hardware.  Without a clear
understanding of how the hardware works and how a high-level programming
language construct is mapped to the hardware, the programmer cannot develop
an efficient implementation of his application.  Therefore, we first
describe the hardware at a level of detail that is relevant to the user or
the reader of this paper, then the programming and compiling
of components of GCM, and finally evaluate the overall performance of the
data parallel implementation of GCM.

\subsection{The CM-5 hardware}

\noindent Figure \ref{figure:cm5} shows a high-level view of the CM-5
architecture.  Scalar computation and control are performed on the front-end
workstation, which is typically a Sparc 10, while parallel computation is
performed on the nodes.  Each node consists of a 32 Mhz Sparc 2 processor
with four 16 MHz vector units (VU's), with each VU capable of 32 Mflop/s
64-bit floating point peak performance.  Memory is local to each node, and
each VU has direct access to its own bank of memory.  Every node is
connected to two separate networks: a data network and a control network,
which perform different kinds of data parallel communications.  The data
network is capable of a maximum of 20 MB per second per node, and is
optimized to handle short messages.  The control network is used for global
synchronization, broadcast and reduction operations.

\begin{fig}{figure:cm5}{The CM-5 consists of nodes connected through a
point-to-point data network and a global control network.  A front end
workstation is used for scalar computation, while nodes are used for
parallel computation.  Each node consists of a Sparc 2 microprocessor and
four vector units, which provide most of the computing power.}
\illustrationx{2.4in}{figures/cm5.eps}
\end{fig}

The CM-5 nodes are controlled by the Sparc 2's, which have poor floating
point performance.  The Sparc 2's should be considered sequencers or
controllers for the VU's, which are fairly powerful -- a VU can
simultaneously generate memory addresses, load data and execute pipelined
floating point operations if properly scheduled by the CM Fortran compiler.
Figure \ref{figure:vector-assembly} shows a simplified version of VU
assembly code emitted by the CM Fortran compiler for the inner loop of the
$aX+Y$ daxpy computation.  An estimated cycle count and exact floating
point operation count are annotated by the compiler.  (Note that Step S1
seems to belong outside of the loop; in this case, the CM Fortran
compiler may not have been able to determine that it was a loop constant.)

\begin{fig}{figure:vector-assembly}{The sequence of steps executed by each
vector unit for the daxpy computation $Z = aX+Y$ as generated by the CM
Fortran compiler.  {\it S} is a scalar register and {\it V1} and {\it V2}
are vector registers that hold eight doublewords.  The cycles counted are 16
MHz VU cycles and are only approximate counts used by the compiler.}
\begin{center}
\begin{tabular}{llcc}
   & {\bf Operation}                                  & {\bf Cycles} & {\bf FP ops} \\
\hline
S1: & load scalar {\it a} into {\it S}                & 4 & 0 \\
S2: & load 8 dwords of {\it X} into {\it V1}          & 8 & 0 \\
S3: & load 8 dwords of {\it Y} into {\it V2}          & 8 & 16 \\
    & \hspace{.2cm} {\it and} $V2 = S \times V1 + V2$ &   & \\
S4: & store {\it V2} into {\it Z}                     & 8 & 0 \\
    & \hspace{.2cm} {\it and} bump offset             &   & \\
    & \hspace{.2cm} {\it and} compare offset to end   &   & \\
    & \hspace{.2cm} {\it and} goto S1 if not done     &   & \\
\hline
\multicolumn{2}{l}{\bf Total} & {\bf 28} & {\bf 16} \\
\end{tabular}
\end{center}
\end{fig}

The VU's can be very powerful and efficient when executing highly regular
vector code, as seen in step S3, where the VU performs a vector load
simultaneously with a multiply-add operation.  The Sparc 2 can concurrently
execute with the VU, as is shown in step S4, where the loop
compare-and-branch is handled by the Sparc 2, while the vector register
store is handled by the VU.

If we only consider this daxpy inner loop, a floating point operation is
executed every $28/16=1.75$ VU cycles.  This ratio is considerably smaller
than on most conventional RISC processors for this computation, especially
for large problem sizes where the RISC processors would go out of cache.
Because the VU is also a memory controller, the VU does not stall for
memory operations that are vectorizable.

We should note, however, this inner loop of the daxpy computation is about
as efficient as the CM-5 gets.  Furthermore, there are other overheads even
for this computation such as garbage padding, and the cycle count estimate
generated by the compiler is overly optimistic.  As we will see in Section
\ref{subsection:pcg-fortran}, the actual speed of the daxpy code is not as
fast as this inner loop would indicate.

The CM Fortran compiler can also generate code to run without VU's (i.e.\
directly on the Sparc processors), but the performance is exceptionally
poor because the code generated for execution without VU's is naive, and
the Sparc processors themselves have very poor floating point performance.
Therefore, in this study we have chosen to show performance numbers only
for the CM-5 with VU's.

All performance measurements on the CM-5 were taken with the CM-5 hardware
timers, which can measure at microsecond resolution time spent by the
front-end, the vector units, control communications and data
communications.  Front end scalar computation can execute at the same time
as parallel computation on the nodes, so in the best case, time spent in
scalar computation is completely masked by parallel computation, where we
expect most of the work to be done.

\subsection{Data parallel programming in CM Fortran}

CM Fortran (CMF) \cite{CMF} is a data parallel extension to Fortran 77 for
the CM-5.  Many of the features of CM Fortran are present in
Fortran 90 and High Performance Fortran (HPF) \cite{HPF}.
CM Fortran
includes syntax for array operations such as elementwise operations,
permutations, and extraction of array subsections.  The CM Fortran compiler
is back compatible to Fortran 77, but will not automatically parallelize
Fortran 77 code: the programmer must explicitly specify parallel array
operations that the compiler distributes among the nodes.

Execution of a CM Fortran program on the CM-5 comprises alternating
computation and communication phases with each phase separated by a global
synchronization barrier.  During the computation phases, the front end is
reponsible for scalar computation such as the addition of two scalar
values, as well as the overall control of the program, such as looping,
procedure calls, and conditionals.  The nodes are responsible for parallel
computation, and all nodes perform identical operations on different data.
Although the nodes may be closely synchronized, each node computes results
and modifies data independently, and consequently modified data values are
exclusive to individual nodes.  At the end of a computation phase,
communication between nodes and between the front-end and the nodes is
orchestrated by the front-end.

Data parallel communication occurs primarily in two highly stylized
manners:
\begin{itemize}
\item {\it Control} communications, which include barriers, reductions and
broadcasts.  These operations are mapped to the control network on the CM-5.
\item {\it Data} communications, which cause the rearrangement of elements
of an array, are mapped to the data network.  They include nearest neighbor
communications which are used in the GCM code, and more general
permutations of data that are significantly more expensive.
\end{itemize}

%Efficient programming in CM Fortran requires a clear idea of what kind of
%code the compiler will generate for the CM-5, and what costs are associated
%with that code.  

In the next sections, we describe how the data parallel
version of GCM is written, and some of the decisions we had to make to
optimize its performance.

\subsubsection{Array operations and layout directives}

CM Fortran allows the expression of element-wise array operations to be
specified very concisely.  For instance, a three-dimensional daxpy
computation $aX+Y$ can be expressed as follows:
\begin{code}
\begin{verbatim}
      Z = a*X+Y
\end{verbatim}
\end{code}
where {\tt X}, {\tt Y} and {\tt Z} are declared statically as array
variables, and {\tt a} is a scalar variable.

Parallelism for the above expression is exploited by different nodes
performing operations on different parts of the {\tt X}, {\tt Y}, and {\tt
Z} arrays.  How the work is divided up is determined by the layout of the
arrays across the nodes -- each node only works on portions of the arrays
that are in its local memory.

Array extents and layouts must be explicitly declared at compile-time.  For
example, a state array for GCM of size $171 \times 93 \times 5$ may be
declared as follows:
\begin{code}
\begin{verbatim}
      real X(5,171,93)
cmf$layout X(:serial,:NEWS,:NEWS)
\end{verbatim}
\end{code}
Given this declaration, we have decided to map the up-down ($k$) dimension
to the leftmost index, the east-west ($i$) dimension to the second index,
and the north-south ($j$) dimension to the rightmost index.

\begin{fig}{figure:array-layout}{Array distribution on distributed-memory
parallel computer.  The water zones are in white, and land zones are in
gray; distribution of water on nodes differs widely.}
\illustrationx{3.2in}{figures/layout-new.eps}
\end{fig}

The {\tt :serial} layout declaration for the $k$ axis places elements
within the same column (i.e.\ with the same $i$ and $j$ coordinates) on the
same processor.  The {\tt :NEWS} declaration spreads the elements along
that axis across the machine.  The resulting array is laid out along the
nodes as shown schematically in Figure~\ref{figure:array-layout}.
All of the ocean state arrays in the GCM code have
the same extents and layouts as described above.

For given array extents, array layout and machine size, the CM Fortran
compiler cannot always distribute the array across the machine evenly, for
the simple reason that array extents do not always divide evenly across the
machine.  Furthermore, the unit of division is not nodes, but VU's, so for
a 32 node CM-5, the two {\tt :NEWS} dimensions of the array must be divided
across 128 VU's.  The array {\tt X} as declared above is therefore padded
out to size {\tt (5,172,96)} to fit evenly on the CM-5 -- the additional
padding elements are ``garbage'' elements that are added to the sides of
the arrays.\footnote{The array is actually padded out 
to $192 \times 96 \times 5$ 
unless the compiler flag {\tt -nopadding} is used -- the larger amount of
padding is a historical artifact due to the desire to make iterations over
array dimensions even multiples of the VU vector length of 8.  The flag
{\tt -nopadding} refers to vector padding, not garbage padding as described
above.} These garbage elements are ignored by the VU hardware but add
overhead in the sense that a problem of size $171 \times 93 \times 5$
requires exactly the same amount of time to execute as a problem of size
$172 \times 96 \times 5$.

Because the array layout declarations affect padding, we must choose the
layouts carefully not just to minimize inter-processor communications,
which is the primary concern in selecting a layout, but also to make sure
that there no more padding than is necessary.  For example, if we
decide to make only one of the axes {\tt :NEWS}, then more garbage
padding is necessary to make the array distribution even over the machine.
Because there are 128 VU's, the {\tt :NEWS} axis must be padded out to a
multiple of 128.

If we have the option of choosing our problem sizes, it is usually easy to
choose a problem size which does not require too much padding, if any.  In
the case of GCM, the problem size is often determined by external factors
(such as the data collection intervals of satellites), so a seemingly odd
problem size like $171 \times 93$ is actually used despite the
additional, though generally small, overhead it may incur.

\subsubsection{Array reduction}

The vector inner product of two arrays $X \cdot Y$ can be expressed as
follows:
\begin{code}
\begin{verbatim}
      b = sum(X*Y)
\end{verbatim}
\end{code}
where {\tt sum} is a CM Fortran function that sums the elements of the
array passed by its argument.  The control network is used in operations
that require {\it reduction} of values across the whole machine.  The {\tt
sum} function is implemented by sequentially summing elements local to each
processor and then globally summing the results of each processor using
the control network.

The CM Fortran compiler tends to do less analysis beyond a single array
operation than other Fortran compilers, which try to group
array operations and execute them together to reduce memory accesses.
This sort of optimization is especially crucial for cache-based RISC
machines. Instead, the CM Fortran handles
each array operation individually, using temporaries rather than
attempting to gain performance through loop fusion and chaining vector 
registers. On the one hand, this simplifies the job of programming because 
we can
estimate the speed of each individual line of code and determine the
overall performance of a procedure by simply summing the times required by
each line of code.  On the other hand, even for an operation as simple as
this inner product, the CM-5 performs the operation in two steps: by first
multiplying {\tt X} and {\tt Y} and storing the result into a temporary
array and then calling the {\tt sum} library function with the temporary
array as an argument.  The time breakout for the reduction communication
shown later therefore includes time spent summing the local parts of the
array.

The scalar value {\tt b} that is calculated by the reduction network is
sent to the front end processor, which uses it for scalar computations.  If
a scalar value is needed for a parallel computation by the nodes, the front
end must broadcast it to them using the control network.  In general, the
movement of scalar values between the front end and nodes are not
expensive because the control network has fairly low latency.

\subsubsection{Shifting array elements}

Some computations in GCM require each element of an array to be calculated
using values from neighboring elements of other arrays. In CM Fortran,
this operation is implemented by {\it shifting} entire arrays by one
element and then performing element-wise operations on the shifted arrays.

Shifting an array $X$ to the north one direction can be expressed as
follows:
\begin{code}
\begin{verbatim}
      Y = CSHIFT(X, +1, 3)
\end{verbatim}
\end{code}
where {\tt +1} indicates shifting in the positive direction, and {\tt 3}
indicates shifting of the third dimension of the array {\tt X}.  (Array
dimensions are named $1,2,3$ respectively). Each value at location
$(k,i,j+1)$ in array {\tt X} is now residing at location $(k,i,j)$ in array
{\tt Y}.  The values residing at the border wrap around to the other
side in the obvious manner.  The wrap-around behavior is correct for
representing circular geometries such as the globe, but in cases where the
geometry is not circular, a ``ghost zone'' of land is added around the
entire array on each side.  These ghost zones are part of the application
code and may cause additional garbage padding to be introduced by the
compiler, as discussed earlier.

The core of the 7-point stencil used for the matrix-vector multiplication
$\A b$ can therefore be expressed as follows:
\begin{code}
\begin{verbatim}
       Ab =
      &     b              * AC
      &   + CSHIFT(b,+1,1) * AL
      &   + CSHIFT(b,-1,1) * AU
      &   + CSHIFT(b,+1,2) * AE
      &   + CSHIFT(b,-1,2) * AW
      &   + CSHIFT(b,+1,3) * AN
      &   + CSHIFT(b,-1,3) * AS
\end{verbatim}
\end{code}
The ``vector'' {\tt b} is actually a three-dimensional array, and $\A$ is
represented only by its diagonals in the three-dimensional arrays {\tt AC},
{\tt AL}, {\tt AU}, etc.  Although this is a very non-intuitive way of
performing matrix-vector multiplication, this stencil actually implements
the multiplication using a data representation that fits the model, and
using operations that map well to the CM-5.  (The ``{\tt \&}'' character
is used to continue the statement to the next textual line in
the program.)

Note that {\tt AL}, {\tt AE} and {\tt AN} represent the symmetric diagonals
of {\tt AU}, {\tt AW}, and {\tt AS} in the $\A$ matrix and are therefore
identical arrays, shifted by one element.  In the Id version, we use one
array to represent both diagonals because we assume a shared memory model
where accessing an element and its neighbor cost the same amount.  For CM
Fortran, if the machine configuration has enough memory, we keep separate
versions of the diagonals to avoid additional {\tt CSHIFT}s, because {\tt
CSHIFT}s can cause a large amount of additional 
communication.

Although the stencil computation does not require creation of full
temporaries to hold shifted arrays, the CM Fortran compiler does not do
analysis to determine that most of the elements required by a processor are
already local to the machine, and accessible with some address calculation
-- rather, temporary shifted arrays are usually created because the vector
units pay little penalty for the memory accesses.

\subsubsection{Computational masks}

Some calculations require different behavior at zones that have a land
boundary than at zones that are completely surrounded by water.  Whenever
possible, we handle these cases by setting land or border zones to special
``benign'' values that make the array computation uniform for land zones
and water zones.  For some computations, this is not possible, and we must
handle the borders differently and explicitly as shown by the following
example.

The $\A$ matrix (consisting of the arrays {\tt AC}, {\tt AN}, etc.) used in
the stencil computation is calculated once at the beginning of the program
from data on the size of each zone and a few other physical constants.
Zones for the different components of the $\A$ matrix that are on the
border of water and land must be calculated in a slightly different manner
than water zones that are completely surrounded by water.  For example, to
compute {\tt AC} and {\tt AN}, the central and the north diagonal of the
$\A$ array respectively, we first compute these quantities as if all zones
were water zones and were surrounded by water on all sides.  Then, we fix
the zones that are adjacent to land zones.  In the following code, we make
the correction when a water zone has a land zone as its northern neighbor.
({\tt WLMASK} is an array that contains 1's on land zones, and 0's on
water zones.)
\begin{verbatim}
   where ((CSHIFT(WLMASK,+1,2)-WLMASK)
  &       .eq.-1)
     AC = AC + AN
     AN = 0.d0
   endwhere
\end{verbatim}
The statements within the {\tt where} block are evaluated only on the
points where the predicate is true; on other points, values of the
variables remain unchanged.  The expression
\begin{code}
\begin{verbatim}
  (CSHIFT(WLMASK,+1,2)-WLMASK).eq.-1
\end{verbatim}
\end{code}
describes those water zones that have a land zone as a northern neighbor.

This manner of handling conditional execution in the data parallel
programming style is inefficient because only a tiny fraction of the
elements of an array are on the border, but the entire array must be tested
with the mask to determine which elements should be handled.  Furthermore,
each of the north, south, east, west, up and down borders must be handled
separately.  The {\tt CSHIFT} is also expensive but, because the geometry of
the ocean does not change significantly, the masks can be computed once and
reused if there is sufficient memory to hold arrays determining the
borders.  For the GCM code, however, this sort of computational masking is
used only during initialization, and is not a serious overhead.

\subsubsection{Computing on array subsections}
\label{subsection:array-subsections}

As discussed in Section \ref{section:ocean-model}, the preconditioning step
of the PCG actually performs a forward-backward substitution using
precalculated $\Lh$ and $\Uh$ arrays for each column of the ocean
independently.  Note that for our data layout, the preconditioning step
does not require any communication between nodes because each column lies
in the same node.  The preconditioner and array layout were chosen
explicitly for this reason; although we could use a different and more
efficient preconditioner for the multithreaded version, we use the same
preconditioner chosen for the data parallel version to give a better
comparison between the two.

To exploit data parallelism in CM Fortran, the expression of the
preconditioning step requires that all of the linear solvers for each
column be executed in lock-step.  That is, rather than expressing the
computation in the most natural fashion, as $N_{x} \times N_{y}$
independent forward-backward substitutions, we perform the first step of
the forward substitution on every element of the top layer of the ocean
then continue with each layer of the ocean down to the bottom of the ocean.

The backward substitution happens in the reverse manner, starting from the
bottom layer, and then substituting up for each layer to the top.  In the
following code, {\tt UD}, {\tt MD} and {\tt LD} represent, respectively,
the upper, middle and lower diagonals of the $\Lh$ and $\Uh$ factors of the
$\D$ matrices, and {\tt q(K,:,:)} refers to the horizontal slice of the
array {\tt q} at depth {\tt K} of the geometry.  (The $\D$ matrices
represent the diagonal block elements of the $\A$ matrix.)
\begin{code}
\begin{verbatim}
  C Forward Substitution
   q(1,:,:) = MD(1,:,:)*r(1,:,:)
   DO K = 2, NZ
     q(K,:,:) = MD(K,:,:)*
  &    (r(K,:,:)-LD(K,:,:)*q(K-1,:,:))
   ENDDO
  C Backward Substitution
   DO K = NZ-1, 1, -1
     q(K,:,:) = q(K,:,:) -
  &     UD(K,:,:)*q(K+1,:,:)
   ENDDO
\end{verbatim}
\end{code}

Because the ocean has different depths at different places, the linear
systems for each column of water may be of different sizes, but the data
parallel programming model forces us to treat each column as if it were the
same depth as the deepest column in the ocean, padding land zones
representations with null equations.

\subsubsection{Putting it together}

With the pieces we have described so far, we can implement the
preconditioned conjugate gradient solver, which has four basic operations:
inner product, daxpy, 7-point stencil and the preconditioner.  Figure
\ref{figure:pcg-fortran} shows the PCG loop expressed in CM Fortran.  For
clarity, we have omitted array declarations and layout directives -- all of
the arrays in this loop have identical extents and layouts.  Also, in the
actual code, the procedure calls to {\tt multiply\_by\_A} and {\tt
precondition} are written in-line, rather than implemented as separate
procedures, in order to eliminate procedure call overhead.

\begin{fig}{figure:pcg-fortran}{The preconditioned conjugate
gradient loop written in CM Fortran is very concise because of the array
syntax.  Note that the results of the two subroutine calls to {\tt
multiply\_by\_A} and {\tt precondition} are returned through arrays passed
to those routines, namely {\tt Ab} and {\tt xi}, respectively.}
\begin{verbatim}
  DO WHILE (loopflag)
    gamma  = sum(r*xi)
    CALL multiply_by_A(b,Ab)
    alpha  = gamma / sum(b*Ab)
    p      = p+alpha*b
    r      = r-alpha*Ab
    CALL precondition(r,xi)
    beta   = sum(r*xi) / gamma
    b      = xi+beta*b
    if (sum(r*r).le.epsilon)
 &    loopflag=.FALSE.
  ENDDO
\end{verbatim}
\end{fig}

Because PCG is the key inner loop for the entire application, we have taken
care to eliminate expensive operations; there are no mask operations in
this loop, and there is no use of the data network, except for the {\tt
CSHIFT}'s in {\tt multiply\_by\_A}.  The {\tt sum} reduction operations use
the control network, which is relatively inexpensive.

As our analysis in Section \ref{ss:computational-characteristics}
indicated, for extremely high-resolution problems,
the vast majority of the time in GCM is spent in the PCG kernel,
so the overall GCM performance is almost completely dependent upon PCG.  We
should note, however, that the issues important for executing PCG
efficiently are the same ones that come up in optimizing the rest of GCM.

\subsection{Data parallel GCM performance}
\label{subsection:pcg-fortran}

\begin{fig}{figure:pcg-performance-cm5}{VU cycles spent per floating point
operations on the CM-5, best case.  Times are for a 32 node machine,
running a problem size of $256 \times 256 \times 32$.}
\illustrationx{3.2in}{figures/cm5-performance.ps}
\end{fig}

Figure \ref{figure:pcg-performance-cm5} shows the cycles spent for each
floating point operation for the constituent parts of the preconditioned
conjugate gradient.  This measure of performance allows one to factor out
machine size, and we have chosen a problem size ($256 \times 256 \times
32$) that does not produce any garbage padding and is also large
enough to factor out idling due to insufficient parallelism and vector
startup.

These times were obtained from the most highly optimized versions of
the PCG components.  Note that in our calculation of overhead
cycles for floating point operations here, it is assumed that every
floating point operation executed by the CM-5 for these operations
is necessary.  In fact, in the GCM code, many of the operations counted
could be occur over land and, therfore, are redundant.

Regardless of these caveats, the performance shown by the CM-5 and CM
Fortran compiler is impressive; if is is assumed
 that each vector unit has a
peak performance of 32 MFlop/s (assuming a multiply-add on every cycle)
then GCM achieves approximately 1/6 of the peak performance.  A more
realistic assumption of 16 MFlop/s as peak would give GCM 1/3 of peak
performance.

Although the compiler estimates for the inner loop from Figure
\ref{figure:vector-assembly} seemed to show that the daxpy computation
would require only 1.75 cycles per floating point operation, there is
additional overhead due to looping code surrounding the inner loop shown in
Figure \ref{figure:vector-assembly}, which brings up the cycle count to a
little more than 2.5 cycles.  This same overhead is seen in each of the
four operations.

The reduction operation in inner product takes a bit more than 0.5 cycles,
which should be the minimum, and the CSHIFT operation in the 7-point
stencil is fairly expensive, as expected.  For the most part, the code
executed by the front end was insubstantial or overlapped by parallel
computation, except in the forward-backward substitution, where the looping
control overhead was higher to handle the forward and backward
substitutions in the preconditioner.

The distribution of time in the PCG is a weighted average of the
distribution of time for the components of the PCG.  The stencil is most
heavily weighted because it accounts for 13 of the 32 total floating point
operations in the PCG.  An additional 8 floating point operations per zone
come from the inner product, 6 from the daxpy, and 5 from the
preconditioner.

We measured the performance of the entire GCM code by running the program
for several hundred time steps to eliminate the effects of initialization
of the system. For the cases we focus on here,
the overall distribution of time in GCM reflects PCG, where
most of the time in GCM is spent, and this is clear from the almost
identical profiles for PCG and GCM.

Note that these timings are the absolute best case for the CM-5 because
there is no garbage padding, the ghost zones overhead is negligible, and
the problem size is sufficiently large to allow the VU's to work at full
speed.  Section \ref{section:performance}, shows how to compare
these best-case numbers against the performance statistics for the Id and
Monsoon version.
