\documentclass[a4paper,12pt]{article}
\newcommand{\ds}{\displaystyle}
\newcommand{\pl}{\partial}
\newcommand{\df}{\ds\frac}
\parindent=0pt
\begin{document}


{\bf Question}

Explain what is meant by an irreducible Markov chain.

The transition probability matrices given below are for 4-state.
5-state and infinite Markov chains respectively, each with states
labelled $1,2,3,\cdots$ . In each case, determine if the Markov
chain is irreducible and classify its states as
positive-recurrent, null-recurrent or transient. Give the periods
of any periodic states. For (ii) find the mean recurrence times of
any ergodic states. State, but do not prove, any general results
you use.

\begin{description}
\item[(i)]
$\left(\begin{array}{cccc} 0 & 0 & \frac{1}{2} & \frac{1}{2}\\ 1 &
0 & 0 & 0\\ 0 & 1 & 0 & 0\\ 0 & 1 & 0 & 0 \end{array}\right)$

\item[(ii)]
$\left(\begin{array}{ccccc} \frac{1}{2} & 0 & \frac{1}{2} & 0 & 0
\\ 0 & \frac{1}{2} & 0 & \frac{1}{2} & 0\\ \frac{1}{2} & 0 &
\frac{1}{2} & 0 & 0\\ 0 & \frac{1}{2} & 0 & \frac{1}{2} & 0\\
\frac{1}{4} & 0 & \frac{1}{4} & 0 & \frac{1}{4}
\end{array}\right)$

\item[(iii)]
$\left(\begin{array}{cccccccc} p & 0 & 1-p & 0 & 0 & 0 & 0 &
\cdots\\ 0 & 1 & 0 & 0 & 0 & 0 & 0 & \cdots\\ p & 0 & 0 & 1-p & 0
& 0 & 0 & \cdots\\ p & 0 & 0 & 0 & 1-p & 0 & 0 & \cdots\\ p & 0 &
0 & 0 & 0 & 1-p & 0 & \cdots\\ . & . & . & . & . & . & . & \cdots
\end{array}\right),\ 0<p<1.$
\end{description}



\vspace{.25in}

{\bf Answer}

A Markov chain is a sequence of discrete (integer-valued) random
variables $(X_n)$ with the property that

$$P(X_{n+1}=i\ |\ X_0=a_0,\ x_1=a_1,\ \cdots\ X_n=j)=P(x_{n+1}=i\
|\ X_n=j)$$

A Markov chain is irreducible if all its states intercommunicate,
i.e. if it is possible to pass between each pair of states in a
finite number of steps with positive probability.

\begin{description}
\item[(i)]
Transition diagram

PICTURE \vspace{2in}

Irreducible, finite, so all states are positive recurrent. All
have period 3.

\item[(ii)]
Transition diagram

PICTURE \vspace{2in}

\newpage
Not irreducible closed sets are $\{1,\ 3\}\ \{2,\ 4\}$ all
aperiodic and positive recurrent. State 5 is transient.

$\mu_1=1 \cdot
\df{1}{2}+2\left(\df{1}{2}\right)^2+3\left(\df{1}{2}\right)^3+\cdots=2$

From symmetry $\mu_2=\mu_3\mu_4=2$

\item[(iii)]
Transition diagram

PICTURE \vspace{2in}

Not irreducible (2) is absorbing $\{1,\ 3,\ 4,\ cdots\}$ is a
closed set of states and so are all of the same type. (1) is
aperiodic, so they all are

$p_{11}=p+p(1-p)+p(1-p)^2+\cdots=p\ds\sum_{n=0}^\infty (1-p)^n=1$

so each state is recurrent

$\mu_1=p+2p(1-p)+3p(1-p)^2+\cdots=\df{1}{p}<\infty$

so each state is positive recurrent.
\end{description}



\end{document}