\documentclass[a4paper,12pt]{article}
\newcommand{\ds}{\displaystyle}
\newcommand{\pl}{\partial}
\newcommand{\un}{\underline}
\newcommand{\undb}{\underbrace}
\newcommand{\df}{\ds\frac}
\parindent=0pt
\begin{document}


{\bf Question}

\begin{description}
\item[(a)]
A Markov chain has three states, and transition probability matrix

$$P=\left(\begin{array}{ccc} 0 & 1 & 0\\ 1-p & 0 & p\\ 0 & 1 & 0
\end{array}\right)$$

where $0<p<1$. Find the probability distribution for state
occupancy at the nth step $(n \geq 1)$ if initially all the states
are equally likely to be occupied.

\item[(b)]
A Markov chain has the transition probability matrix given below.
Classify the states and find the mean recurrence times for all
recurrent states. (Label the states 1,2,3,4,5,6,7 in order)

$$P=\left(\begin{array}{ccccccc} 0 & 0 & \frac{1}{4} & \frac{1}{2}
& \frac{1}{8} & \frac{1}{8} & 0 \\ 0 & \frac{1}{4} & \frac{3}{4} &
0 & 0 & 0 & 0 \\ 0 & \frac{1}{3} & \frac{2}{3} & 0 & 0 & 0 & 0 \\
0 & 0 & 0 & \frac{1}{2} & 0 & \frac{1}{2} & 0\\ \frac{1}{3} & 0 &
\frac{1}{3} & 0 & \frac{1}{6} & \frac{1}{6} & 0\\ 0 & 0 & 0 &
\frac{1}{4} & 0 & \frac{3}{4} & 0\\ 1 & 0 & 0 & 0 & 0 & 0 & 0
\end{array}\right)$$
\end{description}



\vspace{.25in}

{\bf Answer}

\begin{description}
\item[(a)]
$P^2=\left(\begin{array}{ccc} 1-p & 0 & p\\ 0 & 1 & 0\\ 1-p & 0 &
p\end{array}\right)$ and $P^3=P$

so $P^n=P$ if $n$ is odd and $P^n=P^2$ if $n$ is even. So if the
initial distribution is $\left(\df{1}{3},\ \df{1}{3},\
\df{1}{3}\right)$ the distribution at step $n$ is:

$\left(\df{1}{3}(1-p),\ \df{2}{3},\ \df{1}{3}p\right)$ if $n$ is
odd.

$\left(\df{2}{3}(1-p),\ \df{1}{3},\ \df{2}{3}p\right)$ if $n$ is
even.

\item[(b)]
The transition diagram is as follows:

PICTURE \vspace{2in}

$\{2,3\}$ and $\{4,6\}$ both form irreducible closed aperiodic
sets of states, and so are ergodic.

States 1,\ 5 intercommunicate and are of the same type.

The probability of return to state 1 is

$\begin{array}{rcl} f_{11} & = & \df{1}{8} \cdot
\df{1}{3}+\df{1}{8} \cdot \df{1}{6} \cdot \df{1}{3}+\df{1}{8}
\cdot \left(\df{1}{6}\right)^2 \cdot \df{1}{3}+\cdots\\ & = &
\df{1}{8} \cdot
\df{1}{3}\left(1+\df{1}{6}+\df{1}{6^2}+\cdots\right)\\ & = &
\df{1}{8}\cdot \df{1}{3}\cdot \df{1}{1-\frac{1}{6}}\\ & = &
\df{1}{8}\cdot \df{1}{3}\cdot \df{6}{5}=\df{1}{20}<1 \end{array}$

Hence states 1 and 5 are transient.

OR:

The probability of leaving state 1 initially to a state other than
5 is $\df{7}{8}$. Return is only possible from state 5. So the
probability of return is at most $\df{1}{8}<1$.

State 7 is transient since the probability of return is zero.

To calculate mean recurrence times

$\begin{array}{rcl} \mu_2 & = & \df{1}{4}+2 \cdot \df{3}{4}\cdot
\df{1}{3}+3 \cdot \df{3}{4} \cdot \df{2}{3} \cdot \df{1}{3}+4
\cdot \df{3}{4} \cdot \left(\df{2}{3}\right)^2 \cdot
\df{1}{3}+\cdots\\ & = & \df{1}{4}+\df{3}{4} \cdot \df{1}{3}
\left[2+3 \cdot \df{2}{3}+4\left(\df{2}{3}\right)^2+\cdots\right]
\end{array}$

${}$

$\begin{array}{rcrcrcrcl} s & = & a & + & (a+1)r & + & (a+2)r^2 &
+ & \cdots\\ rs & = & & & ar & + & (a+1)r^2& + & \cdots\\ s-rs & =
& a & + & r & + & r^2 & +& \cdots \\ & = & a & + &
\df{r}{1-r}\end{array}$

$$s = \df{a}{1-r} + \df{r}{(1-r)^2}$$

${}$

so $\mu_2=\df{1}{4}+\df{3}{4} \cdot \df{1}{3} \left[3 \cdot 2+9
\cdot \df{2}{3} \right]=3\df{1}{4}$

${}$

$\begin{array}{rcl}\mu_3 & = & \df{2}{3}+\df{1}{3}\cdot \df{3}{4}
\left[2+3\cdot \df{1}{4}+4 \cdot \left(\df{1}{4}\right)^2+\cdots
\right]\\ & = & \df{2}{3}+\df{1}{3}\cdot \df{3}{4}\left[2 \cdot
\df{4}{3}+\df{1}{4}\cdot \left(\df{4}{3}\right)^2\right]\\ & = &
\df{2}{3}+\left(\df{2}{3}+\df{1}{9}\right]\\ & = & 1\df{4}{9}
\end{array}$

${}$

$\begin{array}{rcl}\mu_4 & = & \df{1}{2}+\df{1}{2}\cdot \df{1}{4}
\left[2+3\cdot \df{3}{4}+4 \cdot \left(\df{3}{4}\right)^2+\cdots
\right]\\ & = & \df{1}{2}+\df{1}{2}\cdot \df{1}{4}\left[2 \cdot
4+\df{3}{4}\cdot 4^2\right]\\ & = & \df{1}{2}+\df{1}{2}\cdot 5=3
\end{array}$

${}$

$\begin{array}{rcl}\mu_6 & = & \df{3}{4}+\df{1}{2}\cdot \df{1}{4}
\left[2+3\cdot \df{1}{2}+4 \cdot \left(\df{1}{2}\right)^2+\cdots
\right]\\ & = & \df{3}{4}+\df{1}{2}\cdot \df{1}{4}\left[2 \cdot
2+\df{1}{2}\cdot 2^2\right]\\ & = &
\df{3}{4}+\left(\df{1}{4}+3\right]\\ & = & \df{3}{2}\end{array}$
\end{description}



\end{document}