\documentclass[a4paper,12pt]{article}
\newcommand{\ds}{\displaystyle}
\newcommand{\pl}{\partial}
\parindent=0pt
\begin{document}


{\bf Question}

Define the terms equilibrium distribution and stationary
distribution for a Markov chain.  Explain how they are related for
a finite Markov chain.

Consider the following experiment.  Initially 6 fair coins are
tossed and $X_0$ is the total number of heads obtained.  One coin
is then selected at random and turned over and $X_1$ is the total
number of heads now showing.  A coin is again selected  at random
and turned over giving $X_2$ heads, and so on.

Discuss briefly why $\{X_k\},\ \ k = 0,1,2,...,$ forms a Markov
chain on the states $0, 1, 2, \ldots, 6.$ Write down the initial
probabilities of occupying the states and the transition
probability matrix.

Obtain the stationary distribution of the Markov chain.  Hence
find the probability distribution of $X_k\ \  (k = 1, 2, 3,
\ldots).$

If the number of heads showing initially is known to be 2,
calculate the probability distribution for the number of heads
showing after 2 coins have been turned over.


\vspace{.25in}

{\bf Answer}

A Markov chain with transition matrix $P$ has an equilibrium
distribution if $\ds {\bf p}^{(n)} = {\bf p}^0P^n \to${\boldmath
$\pi$} as $\ds n \to \infty$, independently of the initial
distribution ${\bf p}^{(0)}.$

{\boldmath$\pi$} $^*$ is a stationary distribution if
{\boldmath$\pi$}$^*P =${\boldmath$\pi$}$^*$.

If {\boldmath$\pi$} is an equilibrium distribution then it it is a
stationary distribution, but not conversely.  An irreducible
finite Markov chain with aperiodic states has a unique stationary
distribution which is also its equilibrium distribution.

\bigskip

$X_n$ depends only on $X_{n-1}$ i.e. how many heads there are at
that stage, and not how $X_{n-1}$ has been arrived at.  The
initial probabilities are $$p_0 = p_6 = \frac{1}{64} \hspace{.2in}
p_1 = p_5 = \frac{6}{64} \hspace{.2in} p_2 = p_4 = \frac{15}{64}
\hspace{.2in}p_3= \frac{20}{64} - {\rm binomial}$$

The transition matrix is:

$$\bordermatrix{& 0 &  1 & 2 & 3 & 4 & 5 & 6 \cr 0 & 0 & 1 & 0 & 0
& 0 & 0 & 0 \cr 1 & \frac{1}{6} & 0 & \frac{5}{6} & 0 & 0  & 0 & 0
\cr 2 & 0 & \frac{2}{6} & 0 & \frac{4}{6} & 0 & 0 & 0 \cr 3 & 0 &
0 & \frac{3}{6} & 0 & \frac{3}{6} & 0 & 0 \cr 4 & 0 & 0 & 0 &
\frac{4}{6} & 0 & \frac{2}{6} & 0 \cr 5 & 0 & 0& 0 & 0 &
\frac{5}{6} & 0 & \frac{1}{6} \cr 6 & 0 & 0 &0 & 0 & 0 & 1 & 0 }$$

Note: The Markov chain is irreducible.  All states are positive
recurrent. All are periodic, with period 2.

Suppose the stationary distribution is $(\pi_0 , ..., \pi_6)$.
Solving {\boldmath$\pi$} = {\boldmath$\pi$}$P$ gives:

{\boldmath$\pi$}$\ds ^* = \left( \frac{1}{64}, \frac{6}{64},
\frac{15}{64}, \frac{20}{64}, \frac{15}{64}, \frac{6}{64},
\frac{1}{64}\right)$

 $\begin{array}{rll}{\rm If\ \ } {\bf p}^0 & = & (0, \,  0, \, 1, \,  0, \,
 0,
\, 0, \, 0) \\ \\ {\bf p}^{(1)} & = & (0, \frac{2}{6}, \, 0, \,
\frac{4}{6}, \,  0, \,  0,\,  0 ) \\ \\ {\bf p}^{(2)} & = &
(\frac{2}{36}, \, 0, \, \frac{22}{36},\,  0, \, \frac{12}{36}, \,
0, \, 0 )\end{array}$

There is no equilibrium distribution since we have periodicity.

The vector of initial probabilities is the same as
{\boldmath$\pi$}$^*$.  Thus the probability distribution of $X_k$
is {\boldmath$\pi$}$^* P^k =${\boldmath$\pi$}$^*$.



\end{document}