\documentclass[a4paper,12pt]{article}
\newcommand{\ds}{\displaystyle}
\newcommand{\pl}{\partial}
\parindent=0pt
\begin{document}


{\bf Question}

What is meant by a \lq\lq closed set of states\rq\rq\  of a Markov
chain?  Explain the use of closed sets in classifying states as
\begin{description}
\item[(i)] positive-recurrent,null-recurrent or transient,
\item[(ii)] period or aperiodic,
\end{description}
giving definitions of these terms.

The following transition matrix is a Markov chain with states $1,
2, 3,\ldots, 10.$

$$\left( \begin{array}{cccccccccc} \frac{1}{3} & 0 & 0 & 0 &
\frac{2}{3} & 0 & 0 & 0 & 0 & 0 \\ 0 & \frac{1}{2}& 0 &
\frac{1}{2} & 0 & 0 & 0 & 0  & 0 & 0 \\ 0 & 0 & \frac{1}{2} & 0 &
\frac{1}{2} & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 1 &
0 & 0 \\ 1 & 0 & 0 &  0 & 0 &0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 &
0 & 1 &0 & 0 & 0 &0 \\ \frac{1}{4} & 0 & \frac{1}{4} & 0 &
\frac{1}{4} & \frac{1}{4} & 0 & 0 &0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0
& 0 &0 & 1 & 0
\\ 0 & 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ \frac{1}{10} &
\frac{1}{10} & \frac{1}{10} & \frac{1}{10} & \frac{1}{10} &
\frac{1}{10} & \frac{1}{10} & \frac{1}{10} & \frac{1}{10} &
\frac{1}{10} \end{array} \right)$$ Use a transition diagram to
identify the closed sets of states and write the matrix in blocked
form.

Carry out the above classification of the states.  For each
positive recurrent state calculate the mean recurrence time.



\vspace{.25in}

{\bf Answer}

A set of states of a Markov chain is closed if no state outside C
can be reached from a state in C.   We can treat each closed set
as a sub-chain in classifying states.


In classifying states we use the following results

\begin{description}
\item[1.] If two states intercommunicate, they are of the same
type.
\item[2.] The states space of a Markov chain decomposes into
disjoint sets
\begin{description}
\item[(i)] $T$ - consisting of all transient states
\item[(ii)]$C_1, C_2, ...$ irreducible closed sets of recurrent
states, therefore of the same type.
\end{description}
\end{description}



\setlength{\unitlength}{.25in}
\begin{picture}(14,15)
\put(4,7){\circle{1}}

\put(4,7){\makebox(0,0){10}}

\put(3.1,7){\circle{.75}}

\put(2.7,6.9){\vector(0,-1){0}}

\put(2.4,7){\makebox(0,0){$\frac{1}{10}$}}

\put(4,0.5){\circle{1}}

\put(4,0.5){\makebox(0,0){7}}

\put(4,6.5){\line(0,-1){5.5}}

\put(4,6.5){\vector(0,-1){2.25}}

\put(3.6,4.7){\makebox(0,0){$\frac{1}{10}$}}

\put(1.4,3.5){\circle{1}}

\put(1.4,3.5){\makebox(0,0){6}}

\put(0.5,3.5){\circle{.75}}

\put(0.1,3.3){\vector(0,-1){0}}

\put(-0.1,3.5){\makebox(0,0){1}}

\put(3.5,0.75){\line(-1,1){2.25}}

\put(3.5,0.75){\vector(-1,1){1}}

\put(2.5,1){\makebox(0,0){$\frac{1}{4}$}}

\put(1.3,4.05){\line(1,1){2.5}}

\put(3.5,6.22){\vector(-1,-1){1}}

\put(2,5.5){\makebox(0,0){$\frac{1}{10}$}}

\put(6.5,3.5){\circle{1}}

\put(6.5,3.5){\makebox(0,0){1}}

\put(4,6.5){\line(3,-4){2}}

\put(4,6.5){\vector(3,-4){1}}

\put(4.8,4.6){\makebox(0,0){$\frac{1}{10}$}}

\put(4.3,1){\line(1,1){2}}

\put(4.3,1){\vector(1,1){1}}

\put(5,2.5){\makebox(0,0){$\frac{1}{4}$}}

\put(5.6,3.5){\circle{0.75}}

\put(5.2,3.4){\vector(0,-1){0}}

\put(5,3.5){\makebox(0,0){$\frac{1}{3}$}}

\put(9.5,3.5){\circle{1}}

\put(9.5,3.5){\makebox(0,0){5}}

\put(4,6.5){\line(2,-1){5.1}}

\put(4,6.5){\vector(2,-1){2.5}}

\put(6,5){\makebox(0,0){$\frac{1}{10}$}}

\put(4.5,0.5){\line(2,1){5}}

\put(4.5,0.5){\vector(2,1){2.5}}

\put(6.75,2.25){\makebox(0,0){$\frac{1}{4}$}}

\put(7,3.2){\line(1,0){2}}

\put(9,3.2){\vector(-1,0){1}}

\put(8,2.75){\makebox(0,0){$1$}}

\put(7,3.8){\line(1,0){2}}

\put(7,3.8){\vector(1,0){1}}

\put(7.4,4.3){\makebox(0,0){$\frac{2}{3}$}}

\put(12.5,3.5){\circle{1}}

\put(12.5,3.5){\makebox(0,0){3}}

\put(4,6.5){\line(3,-1){8}}

\put(4,6.5){\vector(3,-1){4}}

\put(9,5.5){\makebox(0,0){$\frac{1}{10}$}}

\put(4.5,0.3){\line(3,1){8}}

\put(4.5,0.3){\vector(3,1){4}}

\put(9,1.25){\makebox(0,0){$\frac{1}{4}$}}

\put(10,3.5){\line(1,0){2}}

\put(12,3.5){\vector(-1,0){1}}

\put(11,3){\makebox(0,0){$\frac{1}{2}$}}

\put(13.4,3.5){\circle{0.75}}

\put(13.8,3.4){\vector(0,-1){0}}

\put(14,3.5){\makebox(0,0){$\frac{1}{2}$}}

\put(0.5,14.5){\circle{1}}

\put(0.5,14.5){\makebox(0,0){2}}

\put(3.8,7.5){\line(-1,2){3.25}}

\put(3.8,7.5){\vector(-1,2){1.5}}

\put(2,9.75){\makebox(0,0){$\frac{1}{10}$}}

\put(0.5,15.4){\circle{.75}}

\put(.9,15.3){\vector(0,-1){0}}

\put(1.2,15.5){\makebox(0,0){$\frac{1}{2}$}}

\put(8.5,14.5){\circle{1}}

\put(8.5,14.5){\makebox(0,0){9}}

\put(4.3,7.5){\line(2,3){4.3}}

\put(4.3,7.5){\vector(2,3){2.3}}

\put(6.5,10){\makebox(0,0){$\frac{1}{10}$}}

\put(8,14.5){\line(-1,0){7}}

\put(8,14.5){\vector(-1,0){3.5}}

\put(4.5,15){\makebox(0,0){$1$}}

\put(3,12.5){\circle{1}}

\put(3,12.5){\makebox(0,0){4}}

\put(4,7.5){\line(-1,4){1.15}}

\put(4,7.5){\vector(-1,4){.5}}

\put(3.8,10){\makebox(0,0){$\frac{1}{10}$}}

\put(2.5,12.5){\line(-1,1){1.6}}

\put(1,14){\vector(1,-1){.75}}

\put(2,13.8){\makebox(0,0){$\frac{1}{2}$}}

\put(3.5,12.5){\line(1,0){2}}

\put(3.5,12.5){\vector(1,0){1}}

\put(4.5,13){\makebox(0,0){$1$}}

\put(6,12.5){\circle{1}}

\put(6,12.5){\makebox(0,0){8}}

\put(4,7.5){\line(1,2){2.3}}

\put(4,7.5){\vector(1,2){1.}}

\put(4.7,10){\makebox(0,0){$\frac{1}{10}$}}

\put(6.5,12.5){\line(1,1){1.6}}

\put(6.5,12.5){\vector(1,1){.8}}

\put(7,13.5){\makebox(0,0){$1$}}

\end{picture}

\bigskip

$$ \begin{tabular}{r|ccccccccccccc}  & 3 & 7 & 10 & & 1 & 5 & & 2
& 4 & 8 & 9 & & 6 \\ \hline \\  3 & $\frac{1}{2}$ & 0 & 0 &
$\vdots$ & 0 & $\frac{1}{2}$ & $\vdots$ & 0 & 0 & 0 & 0 & $\vdots$
& 0 \\ 7 & $\frac{1}{4}$ & 0 & 0 & $\vdots$ & $\frac{1}{4}$ &
$\frac{1}{4}$ & $\vdots$ & 0 & 0 & 0 & 0 &  $\vdots$ &
$\frac{1}{4}$ \\ 10 & $\frac{1}{10}$ & $\frac{1}{10}$ &
$\frac{1}{10}$ & $\vdots$ & $\frac{1}{10}$ & $\frac{1}{10}$ &
$\vdots$  & $\frac{1}{10}$ & $\frac{1}{10}$ & $\frac{1}{10}$ &
$\frac{1}{10}$ & $\vdots$ & $\frac{1}{10}$ \\ & $\cdots$ &
$\cdots$ & $\cdots$ & $\cdots$ & $\cdots$ & $\cdots$ & $\cdots$ &
$\cdots$ & $\cdots$ & $\cdots$ & $\cdots$ & $\cdots$ & $\cdots$
\\ 1 & 0 & 0 & 0 & $\vdots$ & $\frac{1}{3}$ & $\frac{2}{3}$ & $\vdots$
& 0 & 0 & 0 & 0 &$\vdots$ & 0 \\ 5 & 0 & 0 & 0 & $\vdots$ & 1 & 0
& $\vdots$ & 0 & 0 & 0 & 0 &$\vdots$ & 0 \\ & $\cdots$ & $\cdots$
& $\cdots$ & $\cdots$ & $\cdots$ & $\cdots$ & $\cdots$ & $\cdots$
& $\cdots$ & $\cdots$ & $\cdots$ & $\cdots$ & $\cdots$\\ 2 & 0 & 0
& 0 & $\vdots$ & 0 & 0 & $\vdots$ & $\frac{1}{2}$ & $\frac{1}{2}$
& 0 & 0 &$\vdots$ & 0 \\ 4 & 0 & 0 & 0 & $\vdots$ & 0 & 0 &
$\vdots$ & 0 & 0 & 1 & 0 &$\vdots$ & 0
\\ 8 & 0 & 0 & 0 & $\vdots$ & 0 & 0 & $\vdots$ & 0 & 0 & 0 & 1 &$\vdots$
& 0 \\ 9 & 0 & 0 & 0 & $\vdots$ & 0 & 0 & $\vdots$ & 1 & 0 & 0 & 0
&$\vdots$ & 0 \\ & $\cdots$ & $\cdots$ & $\cdots$ & $\cdots$ &
$\cdots$ & $\cdots$ & $\cdots$ & $\cdots$ & $\cdots$ & $\cdots$ &
$\cdots$ & $\cdots$ & $\cdots$ \\ 6 & 0 & 0 & 0 & $\vdots$ & 0 & 0
& $\vdots$ & 0 & 0 & 0 & 0 &$\vdots$ & 1 \end{tabular} $$


States 3, 7, 10 are transient.

For states $\{1, 5\},$ $\ds f_{11} = \frac{1}{3} + \frac{2}{3}
\cdot 1 = 1$ so they are both recurrent.

Since the closed set is finite they are both positive recurrent.
\begin{eqnarray*} \mu_1 & = & \frac{1}{3} \times 1 + \frac{2}{3}
\times 2 = \frac{5}{3} \\ \mu_5 & = & \sum_{r=2}^\infty r
\frac{2}{3} \left( \frac{1}{3} \right) ^{r-2} = \frac{5}{2}
\end{eqnarray*}

Note that $\mu_5$ could be calculated from the relation
$\ds\frac{3}{5}+\frac{2}{5}=1.$

For states $\{2, 4, 8, 9\}$ $\ds f_{22} = \frac{1}{2} +
\frac{1}{2} \cdot 1 \cdot 1 \cdot 1 = 1$ so all are positive
recurrent.

\begin{eqnarray*} \mu_2 & = & \frac{1}{2} \times 1 + \frac{1}{2}
\times 4 = \frac{5}{2} \\ \mu_4 & = & \sum_{r=4}^\infty r \left(
\frac{1}{2} \right)^{r-4} \cdot \frac{1}{2} = 5 (= \mu_8 = \mu_9)
\end{eqnarray*}

State 6 is absorbing ($\mu_6 = 1$).

All states are aperiodic.



\end{document}