\documentclass[a4paper,12pt]{article}
\newcommand{\ds}{\displaystyle}
\newcommand{\pl}{\partial}
\parindent=0pt
\begin{document}


{\bf Question}

Classify the states of the infinite Markov chain with probability
transformation matrix $P$ as transient, null-recurrent or
positive-recurrent, and periodic or aperiodic where $$P = \left(
\begin{array}{ccccccccc} p & 0 & 1-p & 0 & \cdot & \cdot & \cdot
& \cdot & \cdot \\ 0 & 1 & 0 & 0 & \cdot & \cdot & \cdot & \cdot &
\cdot \\ 1 - p & 0 & p & 0 & \cdot & \cdot & \cdot & \cdot & \cdot
\\ 0 & 0 & 0 & p & 1-p & 0 &  \cdot & \cdot & \cdot  \\ \cdot &
\cdot & 0 & p & 0 & 1-p & 0  & \cdot & \cdot\\ \cdot & \cdot & 0 &
p & 0 & 0 & 1-p & 0  & \cdot \\  \cdot & \cdot & 0 & p & 0 & 0 & 0
& 1-p & \cdot \\ \cdot & \cdot & 0 & \cdot & \cdot & \cdot & \cdot
& \cdot & \cdot \end{array} \right)$$



\vspace{.25in}

{\bf Answer}

\begin{description}
\item[(i)] $p=1:$ States $1, 2, 3, 4$ are all absorbing states. States $5, 6,
7, 8,\ldots$ are ephemeral, i.e. $p_{jj} = 0,\ \  j = 5, 6,
\ldots$

\item[(ii)]$p = 0:$  States $\{1,3\}$ form a 2-state Markov chain with
transition matrix $\left(\begin{array}{cc} 0 & 1 \\ 1 & 0
\end{array} \right)$ so both states are periodic with period 2 and
are positive recurrent with mean recurrence time 2.

State 2 is absorbing.

States $4, 5, 6,\ldots$ are transient $(p_{jj} = 0)$.  Infact the
Markov chain follows the route $4\rightarrow 5\rightarrow
6\rightarrow 7\rightarrow 8 \ldots$ with probability 1.


\item[(iii)] $0<p<1$

States $\{1,3\}$ form a 2-state Markov chain, both states
aperiodic. From general results in lectures for a $2\times2$
Markov chain both states are positive recurrent with $\mu_1 = 2 =
\mu_3.$

State 2 is absorbing.

$\{4,5,6,...\}$ is a closed irreducible set of states.  The first
return to state 4 in $n$ steps follows only the path
$4\rightarrow5\rightarrow6\rightarrow\ldots(n+2)\rightarrow(n+3)\rightarrow4.$

So P(1st return to 4 at step\ $n) = f_{44}^{(n)} = p(1-p)^{n-1}$

So $\ds f_{44}= p \sum_{n=1}^\infty(1-p)^{n-1} = 1: \ \
\mu_4=p\sum_{n=1}^\infty n(1-p)^{n-1}= \frac{1}{p}$

So state 4 is positive recurrent and aperiodic.  State 4
intercommunicates with $5, 6, 7,\ldots$ so they are all positive
recurrent and aperiodic.

In fact it can be seen directly from the matrix that states $5, 6,
7,\ldots$ will all be similar to state 4 in their behaviour.

\end{description}




\end{document}