\documentclass[a4paper,12pt]{article}
\newcommand{\ds}{\displaystyle}
\newcommand{\pl}{\partial}
\parindent=0pt
\begin{document}
{\bf Question}
Define the terms equilibrium distribution and stationary
distribution for a Markov chain. Explain how they are related for
a finite Markov chain.
Consider the following experiment. Initially 6 fair coins are
tossed and $X_0$ is the total number of heads obtained. One coin
is then selected at random and turned over and $X_1$ is the total
number of heads now showing. A coin is again selected at random
and turned over giving $X_2$ heads, and so on.
Discuss briefly why $\{X_k\},\ \ k = 0,1,2,...,$ forms a Markov
chain on the states $0, 1, 2, \ldots, 6.$ Write down the initial
probabilities of occupying the states and the transition
probability matrix.
Obtain the stationary distribution of the Markov chain. Hence
find the probability distribution of $X_k\ \ (k = 1, 2, 3,
\ldots).$
If the number of heads showing initially is known to be 2,
calculate the probability distribution for the number of heads
showing after 2 coins have been turned over.
\vspace{.25in}
{\bf Answer}
A Markov chain with transition matrix $P$ has an equilibrium
distribution if $\ds {\bf p}^{(n)} = {\bf p}^0P^n \to${\boldmath
$\pi$} as $\ds n \to \infty$, independently of the initial
distribution ${\bf p}^{(0)}.$
{\boldmath$\pi$} $^*$ is a stationary distribution if
{\boldmath$\pi$}$^*P =${\boldmath$\pi$}$^*$.
If {\boldmath$\pi$} is an equilibrium distribution then it it is a
stationary distribution, but not conversely. An irreducible
finite Markov chain with aperiodic states has a unique stationary
distribution which is also its equilibrium distribution.
\bigskip
$X_n$ depends only on $X_{n-1}$ i.e. how many heads there are at
that stage, and not how $X_{n-1}$ has been arrived at. The
initial probabilities are $$p_0 = p_6 = \frac{1}{64} \hspace{.2in}
p_1 = p_5 = \frac{6}{64} \hspace{.2in} p_2 = p_4 = \frac{15}{64}
\hspace{.2in}p_3= \frac{20}{64} - {\rm binomial}$$
The transition matrix is:
$$\bordermatrix{& 0 & 1 & 2 & 3 & 4 & 5 & 6 \cr 0 & 0 & 1 & 0 & 0
& 0 & 0 & 0 \cr 1 & \frac{1}{6} & 0 & \frac{5}{6} & 0 & 0 & 0 & 0
\cr 2 & 0 & \frac{2}{6} & 0 & \frac{4}{6} & 0 & 0 & 0 \cr 3 & 0 &
0 & \frac{3}{6} & 0 & \frac{3}{6} & 0 & 0 \cr 4 & 0 & 0 & 0 &
\frac{4}{6} & 0 & \frac{2}{6} & 0 \cr 5 & 0 & 0& 0 & 0 &
\frac{5}{6} & 0 & \frac{1}{6} \cr 6 & 0 & 0 &0 & 0 & 0 & 1 & 0 }$$
Note: The Markov chain is irreducible. All states are positive
recurrent. All are periodic, with period 2.
Suppose the stationary distribution is $(\pi_0 , ..., \pi_6)$.
Solving {\boldmath$\pi$} = {\boldmath$\pi$}$P$ gives:
{\boldmath$\pi$}$\ds ^* = \left( \frac{1}{64}, \frac{6}{64},
\frac{15}{64}, \frac{20}{64}, \frac{15}{64}, \frac{6}{64},
\frac{1}{64}\right)$
$\begin{array}{rll}{\rm If\ \ } {\bf p}^0 & = & (0, \, 0, \, 1, \, 0, \,
0,
\, 0, \, 0) \\ \\ {\bf p}^{(1)} & = & (0, \frac{2}{6}, \, 0, \,
\frac{4}{6}, \, 0, \, 0,\, 0 ) \\ \\ {\bf p}^{(2)} & = &
(\frac{2}{36}, \, 0, \, \frac{22}{36},\, 0, \, \frac{12}{36}, \,
0, \, 0 )\end{array}$
There is no equilibrium distribution since we have periodicity.
The vector of initial probabilities is the same as
{\boldmath$\pi$}$^*$. Thus the probability distribution of $X_k$
is {\boldmath$\pi$}$^* P^k =${\boldmath$\pi$}$^*$.
\end{document}