Introduction
This notes confines to one of the ways of analyzing the quality of a point estimator for a parameter
Suggested Reading: [CABE] Casella, G., & Berger, R. L. (2002). Statistical inference (Vol. 2). Pacific Grove, CA: Duxbury; specifically, Chapter 7
Keywords:
- Estimator
- Estimate
- Likelihood function
- Method of Moments (MOM)
- Maximum Likelihood Estimates (MLE)
- Bias, Variance and Mean Squared Error (MSE) of an estimator
Objectives
- Method of finding estimators
- Criteria to find a “best” estimator
- Assessing tools – goodness of estimator
Symbols followed:
- $\theta:$ Parameter
- $f(x_i|\theta):$ Probability density/mass function
- $L(\theta|X):$ Likelihood function
- $l(\theta|X)= \ln L(\theta|X):$ Log Likelihood function
- $T$: An Estimator for a parameter
Definitions
1. An estimator of $\tau(\boldsymbol\theta)$, a function of parameter is any function $T=W(X_1,X_2,\cdots, X_n)$ of a sample; that is any statistic is a point estimator
Here, $\boldsymbol{\theta}=(\theta_1, \theta_2, \cdots, \theta_k)$
2. Bias of T : E(T) – $\theta$
so if E(T) = $\theta$, T is called an $\textbf{unbiased estimator}$ for $\theta$
3. MSE of T : $E[T-\theta]^2$
$$E[T-\theta]^2 = E[T-E(T)+E(T)-\theta]^2$$
$$=E[T-E(T)]^2+E[E(T)-\theta]^2+2*E[(T-E(T))(E(T)-\theta)]$$
T is a statistic, a function of $x_i$ (samples) and hence E(T) is independent of $x_i$
$$= V(T)+(E(T)-\theta)^2 + 2(E(T)-\theta) E(T-E(T))$$
since $E(T-E(T)) = 0$
$$=V(T)+(E(T)-\theta)^2$$
$$MSE_T = Var(T) + (Bias(T))^2$$
Following can be recalled from Sampling Distribution of a statistic
$X_1,\cdots \cdots \cdots, X_n$ be random sample from a distribution $f(x_i|\theta)$ having mean $\mu$ and variance $\sigma^2$
Mean and Variance of sample mean $\bar{X}$
$E(\overline {X}) = \mu$, an unbiased estimator for $\mu$
$V(\overline {X} = \frac{\sigma^2}{n})$
$\Rightarrow MSE~(\overline {X}) = \frac{\sigma^2}{n}$
Mean and Variance of Sample Variance $S^2$
$S^2 = \frac{\sum{(X_i-\overline {X})^2}}{n-1}$
$E[S^2] = \sigma^2$, an unbiased estimator for $\sigma^2$
However, Variance of sample variance $S^2$ requires following approach.
1. If $X_1,X_2,\cdots,X_n \sim f(X~|~\theta)$, then
$V(S^2) = \frac{1}{n} [\mu_4 – \frac{n-3}{n-1} \mu_2^2]$
2. In particular, if $X_1,X_2,\cdots,X_n \sim \text{Normal}~(\mu,~\sigma^2)$
$V(S^2) = V(\frac{n-1}{\sigma^2}\frac{\sigma^2}{n-1}S^2)$
$= \frac{\sigma^4}{(n-1)^2} V(\frac{n-1}{\sigma^2} S^2)$
$=\frac{\sigma^4}{(n-1)^2} V(\chi^2_{n-1})$
$= \frac{\sigma^4}{(n-1)^2} 2(n-1)$
$= 2\frac{\sigma^4}{n-1}$
Variance of MLE for $\sigma^2$
$V(\hat\sigma^2_{ML}) = V(\frac{n-1}{n} S^2)$
$= (\frac{n-1}{n})^2~ V(S^2)$
$= \frac{(n-1)^2}{n^2}~ \frac{2\sigma^4}{n-1}$
$= 2\frac{n-1}{n^2}~ \sigma^4$
Comparison
$V(S^2) = \frac{2}{n-1}~ \sigma^4$
$2 \sigma^4 = (n-1)~ V(S^2)$
$\Rightarrow V(\hat\sigma^2_{ML}) = \frac{(n-1)}{n^2} ~(n-1)~ V(S^2)$
$=(\frac{n-1}{n})^2~ V(S^2)$
$V(\hat\sigma^2_{ML}) = (1-\frac{1}{n})^2~ V(S^2)$
$V(\hat\sigma^2_{ML}) \leq V(S^2)$
It can easily be observed that $n \rightarrow \infty ~~~V(\sigma^2) = V(S^2)$
Note on MLE of Gamma($\alpha,\beta$), $\alpha$ known
$X1,X2,\cdots,Xn \sim \text{Gamma}~(\alpha,\beta)$ and let $\lambda = \frac{1}{\beta}$
$\alpha:$ Shape parameter; $\beta:$ Scale parameter; $\lambda:$ rate parameter
$f(x_i,\theta) = \frac{\lambda^{\alpha}}{\Gamma(\alpha)} x_i^{\alpha -1} \exp^{-\lambda x_i} ~~~~x_i>0,~ \alpha>0,~\lambda>0$
$\Rightarrow L(\theta|X) = \prod_{i=1}^n \frac{\lambda^{\alpha}}{\Gamma(\alpha)} x_i^{\alpha -1} e^{-\lambda x_i}$
$= (\frac{\lambda^\alpha}{\Gamma \alpha})^n \prod_{i=1}^n (x_i)^{\alpha – 1} e^{-\lambda \sum{x_i}}$
$l(\theta|X) = \alpha~n~\ln\lambda – n~\ln\Gamma(\alpha) + (\alpha-1) \sum{\ln x_i} – \lambda \sum{x_i}$
$\frac{dl}{d\theta} =0 \Rightarrow \alpha~n \frac{1}{\lambda} – \sum{x_i}= 0$
$\Rightarrow \frac{\alpha ~n}{\lambda} = \sum{x_i}$
$\lambda_{ML} = \frac{\alpha n}{\sum{x_i}} = \frac{\alpha}{\bar{X}}$
Hence, if $\alpha$ is known
$\hat\beta_{ML} = \frac{\overline {X}}{\alpha}$
$\hat\lambda_{ML} = \frac{\alpha}{\overline {X}}$
Exponential distribution from a Gamma distribution with shape parameter $\alpha$ = 1
If $X ~\sim \text{Gamma}(1,\beta)$, then $X \sim \text{Exponential}~(\beta)$ with mean $\beta$ and $f(x|\beta)=\frac{1}{\beta}~\exp(-\frac{x}{\beta})$
If $X_1,X_2,\cdots,X_n ~\sim \text{Exponential}(\beta)$ then $\sum X_i \sim \text{Gamma}(n,\beta)$
Then $E[\hat\beta_{ML}] = \frac{E(\overline {X})}{\alpha}$
$= E(\bar X)$ since $\alpha = 1$
$=\frac{n~\beta}{n}=\beta$
$E[\hat\beta_{ML}]=\beta$
On the other hand, if
$X ~\sim \text{Gamma}(1,\lambda)$, then $X \sim \text{Exponential}~(\lambda)$ with mean $~\frac{1}{\lambda}$and $f(x|\lambda)=\lambda~\exp(\lambda~x)$
$\bar X \sim \text{Gamma}~(n,n\lambda)$
Hence, $T=\frac{\alpha}{\bar X} ~= \frac{1}{\overline X} \sim \text{Inverse} ~\text{Gamma} (n, ~n\lambda)$
$\Rightarrow$ $E(T) =\frac{n}{n-1}~\lambda$ and
$V(T) =\frac{n^2}{(n-1)^2~(n-2)}~\lambda^2$
Beta Distribution
$X_1,X_2,\cdots ,X_n \sim \text{Beta}(1,\theta)$
$f(x_i,\theta) = \theta (1-x)^{\theta -1} ~~~ 0 < x < 1$
$L(\theta|X) = \prod \theta (1-x_i)^{\theta -1 }$
$L(\theta|X) = \theta^n \prod(1-x_i)^{\theta -1 }$
$l(\theta|X) = n ~\ln\theta + (\theta -1) \sum \ln(1-x_i)$
$\frac{\partial~l}{\partial~\theta} = 0$
$\Rightarrow \frac{n}{\theta} + \sum \ln(1-x_i)$
$T= \hat\theta_{ML} = \frac{-n}{\sum \log(1-x_i)} = \frac{n}{u}$
It can be noted that $Y = 1-X ~\sim \text{Beta}(\theta,1)$
PDF of Y is $g(Y|\theta) = \theta y^{\theta – 1} ~~~ y>0, ~~~0<\theta<1$
Let Z = $-\ln Y$
$\frac{dz}{dy} = \frac{-1}{y}$
$\Rightarrow J = |\frac{dy}{dz}| = y = e^{-z}$
Hence, PDF of Z is
$h(Z|\theta) = \theta (e^{-z})^{\theta – 1} e^z = \theta e^{-\theta z}$
$\Rightarrow Z \sim \text{Exponential}(\theta) = \text{Gamma}(1,\theta)$ and $\theta: ~\text{rate~ parameter}$
$\Rightarrow U = \sum Z = -n \sum \ln(1-X_i) = \text{Gamma}(n,\theta)$
$\Rightarrow \frac{1}{U} \sim ~\text{Inverse~ gamma} (n,\theta) ~~\theta = \frac{1}{\beta} ~~~\beta : \text{Scale~parameter}$
$\Rightarrow E(T) =\frac{n}{n-1}~\lambda=\frac{n}{n-1}~\theta$ and hence T is not an unbiased estimator for $\theta$
It can also be observed that $V(T) =\frac{n^2}{(n-1)^2~(n-2)}~\lambda^2=\frac{n^2}{(n-1)^2~(n-2)}~\theta^2$
This shows that mean and variance of an estimator may not always be straight forward
Also to note that E(T) or V(T) may be very difficult in certain cases. For example, estimator of proportion parameter in negative binomial distribution is
$T = \hat \theta_{ML} = \frac{r}{\overline {X}}$
Comparing estimators of Poisson parameter
$X_1,\cdots,X_n \sim \text{Poisson}~(\theta)$
MLE
$T_1=\hat\theta_{ML} = \overline {X}$
$E(T_1) = \mu = \theta$ and hence, $T_1$ is unbiased
$V(T_1) = \frac{\sigma^2}{n} = \frac{\theta}{n}$
For any distribution $E(S^2) = \sigma^2$ where $S^2$ is sample variance.
$\Rightarrow$ if $T_2 = S^2$ then ~$T_2$ is also unbiased.
$V(S^2) = \frac{1}{n} [\mu_4 – \frac{n-3}{n-1} \mu_2^2]$
$= \frac{1}{n} [(\lambda+3\lambda^2) – \frac{n-3}{n-1} \lambda^2]$
$=\frac{\lambda^2}{n}~\Big[2+\frac {2}{n-1}+\frac {1}{\lambda}\Big]$
$\Rightarrow V(T_1) < V(T_2)$
Cramer Rao Inequality
Let $X_1,X_2,\cdots,X_n$ be a random sample from a distribution with pdf $f(x~|~\theta)$ and $T(x) = W(x_1,x_2,\cdots,x_n)$ be any estimator satisfying Leibnitz Rule
$\frac{d}{d\theta}[E(T)] = \int \frac{d}{d\theta}[T.f(x|\theta)] dx$ and $V(x) < \infty$ then
$$V[T] \geq \frac{[\frac{d}{d\theta}E(T)]^2}{E[\frac{d}{d\theta} \log f(x|\theta)]^2}= \frac{[\frac{d}{d\theta}E(T)]^2}{E[\frac{dl}{d\theta}]^2}$$
Note on CRI
1. Estimator Based
if T is an unbiased estimator, then E(T) = $\theta$
$\Rightarrow$ $\frac{d}{d\theta} E(T) = 1$
Hence CRI is
$$V[T] \geq \frac{1}{E\Big[\frac{dl}{d\theta}\Big]^2}$$
2. Likelihood based or Data based
If $X_1,…,X_n \overset{iid}\sim f(X|\theta)$ then
$$E[\frac{dl}{d\theta}]^2 = n E[\frac{dlnf}{d\theta}]^2$$
3.Fisher Information Matrix – FIM
$E\Big[\frac{d~l}{d\theta}\Big]^2 = -E\Big[\frac{d^2~l}{d\theta^2}\Big]$
It may be possible to use all these results for a given situation. (i.e. a random sample, has an unbiased estimator and use FIM)
$\Rightarrow$ CRI is $V(T) \geq \frac{1}{-n E\Big[\frac{d^2~l}{d~\theta^2}\Big]}$
FIM has to be applied for likelihoods (pdfs) that satisfy $E[\frac{dl}{d\theta}] = 0$ (necessary condition)
Let $X \sim \text{Uniform}(0,\theta)$
$f(x|\theta) = \frac{1}{\theta} ~~~ 0 < x < \theta$
$\frac{dl}{d\theta} = -\frac{1}{\theta}$ and $(\frac{dl}{d\theta})^2 = \frac{1}{\theta^2}$
$E[\frac{dl}{d\theta}]^2 = \frac{1}{\theta^2}$
$\frac{d^2~l}{d~\theta^2}$ = $\frac{1}{\theta^2}$
$E[\frac{d^2l}{d\theta^2}] = \frac{1}{\theta^2}$ and $-E[\frac{d^2 l}{d\theta^2}] = \frac{-1}{\theta~^2}$
Hence, $E[\frac{dl}{d\theta}]^2 \neq -E[\frac{d^2 l}{d\theta^2}]$
Parameter in the range of x is the reason for such a result.
CRI and Bernoulli Distribution
$X_1,…,X_n \overset{iid}\sim \text{Bernoulli} ~(\theta)$
$\Rightarrow f(x|\theta) = \theta^x (1-\theta)^{1-x}$
$\ln f(x~|~\theta) = x \ln\theta + (1-x)~ \ln(1-\theta)$
$\frac{d~l}{d~\theta} = \frac{x}{\theta} + \frac{1-x}{1-\theta}(-1)$
$= \frac{x}{\theta} – \frac{1-x}{1-\theta}$
$\frac{d^2~l}{d\theta^2} = -\frac{x}{\theta^2} – \frac{1-x}{(1-\theta)^2}$
$E[\frac{d^2~l}{d\theta^2}] = -[\frac{1}{\theta^2} E(x) + \frac{1}{(1-\theta)^2} E(1-x)]$
$= -[\frac{1}{\theta^2} \theta + \frac{1}{(1-\theta)^2} (1-\theta)]$
$E[\frac{d^2~l}{d\theta^2}] = -[\frac{1}{\theta (1-\theta)}]$
$-E[\frac{d^2~l}{d\theta^2}] = \frac{1}{\theta (1-\theta)}$
$\Rightarrow$ CRI $\Rightarrow$
$V(T) \geq \frac{[\frac{d}{d\theta}E(T)]^2}{-n E[\frac{d^2~l}{d\theta^2}]}$
For the class of unbiased estimators
$V(T) \geq \frac{1}{-n E[\frac{d^2}{d\theta^2} lnf]}$
$\Rightarrow$ CRLB is,
$\frac{1}{-n E[\frac{d^2}{d\theta^2} lnf]} = \frac{1}{n[\theta (1-\theta)]^{-1}}$
$= \frac{\theta (1-\theta)}{n}$
Now, $T = \overline {X}$, an unbiased estimator for $\theta$ and
$V(\bar X) = \frac{\sigma^2}{n} = \frac{\theta (1-\theta)}{n}$
$\Rightarrow \overline {X}$ $\textbf{attains}$ CRLB.
CRI and Poisson Distribution:
$X_1,\cdots,X_n \overset{iid}\sim \text{Poisson} (\theta$)
$P(X=x) = e^{-\theta}~ \frac{\theta~^x}{x!} ~~~ x = 0,1,2,…$
$l(\theta|X) = -\theta + x \ln\theta +k$
$\frac{dl}{d\theta} = -1 +\frac{x}{\theta}$
$\frac{d^2 l}{d\theta^2} = -\frac{x}{\theta^2}$
$E[\frac{d^2 l}{d\theta^2}] = -\frac{1}{\theta^2} E(X)$
$= -\frac{1}{\theta^2} \theta = -\frac{1}{\theta}$
$\Rightarrow -E[\frac{d^2 l}{d\theta^2}] = \frac{1}{\theta}$
CRLB: $-\frac{1}{-n E[\frac{d^2 l}{d\theta^2}]}$
$= \frac{1}{n\frac{1}{\theta}} = \frac{\theta}{n}$
Consider $\hat\theta_{ML}=\overline {X}$
$V(\overline {X}) =\frac{\sigma^2}{n} = \frac{\theta}{n}$
$\hat\theta_{ML} = \overline {X}$ $\textbf{attains}$ CRLB.
CRI and Normal Distribution
$X_1,\cdots,X_n \overset{iid}\sim \text{Normal}(\mu,\sigma^2)$
$\textbf{CRLB:}$
$\frac{1}{-nE[\frac{d^2l}{d\theta^2}]}$
$= \frac{1}{-n(\frac{-1}{2\sigma^4})}$
$= \frac{2\sigma^4}{n}$
If $T_1 = S^2 = \frac{\sum (x_i – \overline {X})^2}{n-1}$ then $V(T_1) = \frac{2\sigma^4}{n-1}$
Now, $n-1 < n \Rightarrow \frac{1}{n-1} > \frac{1}{n}$
$\Rightarrow \frac{2\sigma^4}{n-1} > \frac{2\sigma^4}{n}$
$\Rightarrow S^2$ does not attain CRLB
Also, if
$T_2 = \hat\sigma^2_{ML} = \frac{\sum(x_i – \overline {X})^2}{n}$ then,
$V(T_2) = \frac{n-1}{n^2}~ 2\sigma^4$
$= \frac{n-1}{n} \frac{2\sigma^4}{n}$
But $T_2$ is not unbiased and hence the numerator term of CRI is
$E(T_2) = \frac{n-1}{n} \sigma^2$
$\frac{d}{d\sigma^2} E(T_2) = \frac{n-1}{n}$
$\Big[\frac{d}{d\sigma^2} E(T_2)\Big]^2 = (\frac{n-1}{n})^2$
$\Rightarrow$ CRLB is $\frac{(\frac{n-1}{n})^2}{-n(\frac{-1}{2\sigma^4})}$
$= \frac{(n-1)^2}{n^3} 2\sigma^4$
$\Rightarrow \hat\sigma^2_{MLE}$ does not attain CRLB$
CRI and Gamma Distribution
Scale parameter $\beta$ is unknown
shape parameter$(\alpha)$ is assumed to be known
$X_1,\cdots,X_n \overset{iid} \sim \text{Gamma}~(\alpha,\beta)$
$f(x|\theta) = \frac{1}{\Gamma\alpha ~\beta^\alpha} x^{\alpha-1} e^{-\frac{x}{\beta}}$
$l = \ln f(x|\theta) = -ln \Gamma \alpha -~\alpha \ln \beta +~ (\alpha-1) \ln n – \frac{x}{\beta}$
$\frac{dl}{d\beta} = -\frac{\alpha}{\beta} + \frac{x}{\beta^2}$
$\frac{d^2l}{d\beta^2} = \frac{\alpha}{\beta^2} – \frac{2x}{\beta^3}$
$E[\frac{d^2l}{d\beta^2}] =\frac{\alpha}{\beta^2} – \frac{2}{\beta^3} \alpha \beta$
$= -\frac{\alpha}{\beta^2}$
Now, an estimator of $\beta$ is $T = \frac{\overline {X}}{\alpha}$ so that
$E(T) = \frac{1}{\alpha} E(\overline {X})$
$= \frac{1}{\alpha} \mu = \beta$
$\Rightarrow ~T$ is unbiased estimator of $\beta$
Also
$V(T) = \frac{1}{\alpha^2} V(\bar X) = \frac{1}{\alpha^2} \frac{\sigma^2}{n}$
$= \frac{1}{\alpha^2} \frac{\alpha \beta^2}{n}$
$= \frac{\beta^2}{n\alpha}$
$\Rightarrow$ CRLB is $\frac{1}{-nE[\frac{d^2l}{d\theta^2}]}$
$= \frac{1}{n \frac{\alpha}{\beta^2}}$
$= \frac{\beta^2}{n\alpha}$
$\Rightarrow$ MLE for $\beta$ attains CRLB.
CRI and Beta Distribution:
$X_1,\cdots,X_n \overset{iid}\sim \text{Beta}(1,\beta)$ one shape parameter $\alpha = 1$ and another shape parameter $\beta>0$ is unknown
$f(x|\theta) = \beta (1-x)^{(\beta -1)}$
$l(\theta~|~X) = \ln\beta + (\beta -1) \ln(1-x)$
$\frac{dl}{d\theta} = ln(1-x) + \frac{1}{\beta}$
$\frac{d^2l}{d\theta^2} = -\frac{1}{\beta^2}$
$E[\frac{d^2l}{d\theta}] = -\frac{1}{\beta^2}$
Now,
$T = \hat\beta_{ML} = \frac{n}{y} = -\frac{n}{\sum log(1-x_i)}$
$E(T) = \frac{n}{n-1}\beta$
$\Rightarrow \textbf{T is not unbiased}$
$\Rightarrow \frac{d}{d\theta} E(T) = \frac{n}{n-1}$
$[\frac{d}{d\theta} E(T)]^2 = \frac{n^2}{(n-1)^2}$
$\textbf{CRLB} $$= \frac{\frac{n^2}{(n-1)^2}}{\frac{n}{\beta^2}}$
$=\frac{n}{(n-1)^2} \beta^2$
But $V(T) = \frac{n}{(n-1)^2} ~\beta^2 \frac{n}{n-2}$
$\Rightarrow T$ $\textbf{does not attain}$ CRLB
Let $T_1 = \frac{n-1}{n} T$
$E(T_1) = (\frac{n-1}{n}) E(T)$
$= \frac{n-1}{n}\frac{n}{n-1} \beta = \beta$
$V(T_1) = (\frac{n-1}{n})^2~V(T) $
$= (\frac{n-1}{n})^2 \frac{n^2}{(n-1)^2 (n-2)} \beta^2 = \frac{\beta^2}{n-2}$
CRLB is $\frac{1}{-n[\frac{d^2 l}{d\beta^2}]} = \frac{1}{n(\frac{1}{\beta^2})} = \frac{\beta^2}{n}$
So,$V(T_1) > CRLB$
CRI and Exponential distribution
$X_1,\cdots,X_n \overset{iid}\sim \text{Exponential}~(\beta)~~~~ \beta$: Scale Parameter
Equivalently, $X\sim \text{Gamma}~(1,\beta)$.
$f(x|\beta) = \frac{1}{\beta} e^{\frac{-~x}{\beta}}$
$l(\theta|X) = -\ln \beta -\frac{x}{\beta}$
$\frac{dl}{d\theta} = -\frac{1}{\beta} + \frac{x}{\beta^2}$ and $\frac{d^2l}{d\beta^2} = \frac{1}{\beta^2}-\frac{2x}{\beta^3}$
$E[\frac{d^2l}{d\beta^2}] = \frac{1}{\beta^2} – \frac{2}{\beta^3} E(X)$
$= \frac{1}{\beta^2} – \frac{2}{\beta^3} \beta$
$= -\frac{1}{\beta^2}$
$\Rightarrow$ denominator of CRLB is
$-n E[\frac{d^2l}{d\beta^2}] = \frac{n}{\beta^2}$
Consider MLE of $\beta$
$T =\hat \beta_{ML} = \overline {X}$
$E(T) = E(\overline {X}) =\mu $
$= \frac{1}{\theta} = \beta $
$\Rightarrow$ T is unbiased
$V(T) = V(\overline {X})$
$V(T) =\frac{\sigma^2}{n}$
$V(T) =\frac{\beta^2}{n}$
CRLB is $ \frac{1}{-nE[\frac{d^2l}{d\theta^2}]} = \frac{1}{(\frac{n}{\beta^2})} = \frac{\beta^2}{n}$
$\Rightarrow V(T) = CRLB$
$\Rightarrow \overline {X} \textbf{attains}$ CRLB.
Reparameterization of Exponential distribution
$X_1,\cdots,X_n \overset{iid}\sim \text{Exponential}~(\theta)~~~~ \theta = \frac {1}{\beta}$: rate Parameter
Goal is to check MLE $T_1$ of $\theta$ = $\frac{1}{\bar X}$ for CRLB.
$T_1 \sim IG(n,\frac{\beta}{n})$
$\Rightarrow E(T_1) = \frac{1}{(n-1)\frac{\beta}{n}} = \frac{n}{(n-1)} \frac{1}{\beta} = \frac{n}{n-1} \theta$
$V(T_1) = \frac{1}{(n-1)^2 (n-2) \frac{\beta^2}{n^2}} = \frac{n^2}{(n-1)^2 (n-2)} \theta^2$
Let $T = \frac{n-1}{n} T_1$
$E(T) = \frac{n-1}{n} E(T_1) = \theta$
T is an unbiased estimator of $\theta$
$V(T) = (\frac{n-1}{n})^2 ~~~V(T_1)$
$= (\frac{n-1}{n})^2 \frac{n^2}{(n-1)^2 (n-2)} \theta^2 = \frac{1}{n-2} \theta^2$
Now,
$l = lnf(X|\theta) = ln \theta -\theta x$
$\frac{dl}{d\theta} = \frac{1}{\theta} – x$
$\frac{d^2l}{d\theta^2} = \frac{-1}{\theta}$
$-E[\frac{d^2l}{d\theta^2}] = \frac{1}{\theta^2}$
$\Rightarrow$ CRLB is
$\frac{1}{-nE[\frac{d^2l}{d\theta^2}]}$
$= \frac{1}{n\frac{1}{\theta^2}} = \frac{\theta^2}{n}$
But
$V(T) = \frac{1}{n-2} \theta^2 \geq \frac{\theta^2}{n}$
$\Rightarrow$ T does not attain CRLB.
Cramer-Rao Inequality should be handled carefully when reparameterization is desired
It is easy to verify that, for $X \sim \text{Uniform} (0,\theta)$ ($\theta$ is the parameter), CRI cannot be applied. However, MLE for $\theta$ can be studied further.
$\hat \theta_{ML}= Max\{x_{i}\} = y$
$f(y|\theta) = \frac{n y^{n-1}}{\theta^n} ~~~ 0 < y < \theta$
$E(Y) = \int_{0}^{\theta} \frac{n y^{n-1}}{\theta^n} y dy = \frac{n\theta}{n+1}$
Let $T_2 = \frac{n+1}{n} y$
$\Rightarrow E(T_2) = \frac{n+1}{n} E(y) = \theta$
$\Rightarrow T_2$ is an unbiased estimator of $\theta$
Now $E(Y^2) = \frac{n}{\theta^n} \int_{0}^{\theta} y^2 y^{n-1} dy$
$= \frac{n}{\theta^n}\int_{0}^{\theta} y^{n+1} dy$
$= \frac{n}{\theta^n} [\frac{y^{n+2}}{n+2}]_{0}^{\theta} = \frac{n}{n+2} \theta^2$
$\Rightarrow V(Y) = \frac{n}{n+2} ~\theta^2 – \frac{n^2}{(n+1)^2}~ \theta^2$
$= n \theta^2 [\frac{1}{n+2} – \frac{n}{(n+1)^2}]$
$= n \theta^2 [\frac{(n+1)^2 – n(n+2)}{(n+1)^2(n+2)}]$
$= n\theta^2 [\frac{1}{(n+1)^2(n+2)}]$
$\Rightarrow V(T) = V(\frac{n+1}{n} y) = \frac{(n+1)^2}{n}~ V(Y)$
$V(T) = (\frac{n+1}{n})^2 \frac{n\theta~^2}{(n+1)^2(n+2)}$
$V(T) = \frac{\theta^2}{n(n+2)}$