---
title: Asymptotic & Maximum Likelihood Estimation
date: 2026-07-21 18:18:00
tags:
  - Statistics
cover: https://img.yulecna.top/posts/yn-background.jpg
description: ""
---
### Review

1. Score function: $S_\theta(X) = \nabla_\theta l(\theta, X)$
	1. Local sufficient statistics: 
	   $$
	   \frac{p_{\theta_0+\eta}(X)}{p_{\theta_0}(X)} \approx \exp\{\eta^TS_{\theta_0}(X)\}
	   $$
	2. Statistic property: 
	   $$
	   \begin{aligned}
	   \mathbb{E}[S_\theta(X)] &= \int S_\theta(x) dP_\theta(x) \\
	    &= \int S_\theta(x)p_\theta(x)\,d\mu(x) \\
	    &= \nabla_\theta \int p_\theta(x)\,d\mu(x) \\
	    &= \nabla_\theta\mathbb{E}_\theta[X] = 0
	   \end{aligned}
	   $$
	   **Notation**: This property holds true in any direction (any dimension of $\theta$)
2. Fisher Information: $J(\theta) = Var_\theta(S_\theta(X)) = \mathbb{E}_\theta[-\nabla^2l(\theta; X)]$
   $$
   \begin{aligned}
   & \mathbb{E}[\nabla_\theta l(\theta, X)] = 0 \\
   \Longrightarrow 
   0 &= \dfrac{\partial}{\partial \theta_j}\dfrac{\partial}{\partial \theta_i}\mathbb{E}[e^l] \\
   &= \int \left(\dfrac{\partial^2l}{\partial\theta_j\partial\theta_i} + \dfrac{\partial l}{\partial \theta_j}\dfrac{\partial l}{\partial\theta_i}e^l\right) \,d\mu \\
   &= \mathbb{E}_\theta[\nabla^2l] + \mathbb{E}_\theta[SS^T] \\
   \Longrightarrow
   \mathbb{E}_\theta[SS^T] &= -\mathbb{E}_\theta[\nabla^2l]
   \end{aligned}
   $$
	1. $J(\theta) = Var(S_\theta)$
	2. $J(\theta) = -\mathbb{E}[\nabla^2l]$

## Asymptotic

2 kinds of convergence:
1. cvg. in probability
   cip to **constant** $c\in\mathbb{R}^d$ if $\mathbb{P}(||X_n-c||>\epsilon)\rightarrow 0, \forall \epsilon>0$
2. cvg. in distribution
   cid to **random variable** $X$ if $\mathbb{E}f(X_n)\longrightarrow\mathbb{E}f(X)$ for all bdd,cts $f: x\rightarrow\mathbb{R}$

### Limit Theorems

Let $X_1, X_2, \dots \overset{iid}{\sim} X$ be random vectors, $\bar{X_n} = \dfrac{1}{n}\sum_{i=1}^{n}X_i$, 
1. **LLN**:  If $\mathbb{E}|X| < \infty$, $\mathbb{E}[X] = \mu$, then: 
	1. $\bar{X_n} \overset{p}{\longrightarrow}\mu$  ($\bar{X_n}\overset{a.s.}{\longrightarrow}\mu$)
2. **CLT**: If $\mathbb{E}[X] = \mu \in \mathbb{R}^d$, $Var(X) = \Sigma$, then: 
	1. $\sqrt{n} (\bar{X_n}-\mu) \overset{d}{\longrightarrow} N(0, \Sigma)$

### Continuous Mapping

$g$ cts, $X_1, X_2, \dots$ r.v.s
- If $X_n \overset{d}{\longrightarrow}X$, then $g(X_n)\overset{d}{\longrightarrow}g(X)$
- If $X_n \overset{p}{\longrightarrow}c$, then $g(X_n)\overset{p}{\longrightarrow}g(c)$

### Delta Method

If 
$$
\begin{cases}
	\sqrt{n}(X_n-\mu)\overset{d}{\longrightarrow}N(0, \sigma^2) \\ 
	f(x)\text{ differentiable at }x=\mu
\end{cases}
$$
Then $\sqrt{n}(f(X_n)-f(\mu))\overset{d}{\longrightarrow}N(0, \dot{f}(\mu)^2\sigma^2)$
( i.e. $X_n\approx N(\mu, \sigma^2/n) \overset{d}{\longrightarrow} f(X_n)\approx N(f(\mu), \dot{f}(\mu)\sigma^2/n)$ )

## Maximum Likelihood Estimation

Setup: 
1. Given: $X_1, \dots, X_n \overset{iid}{\sim}P_{\theta_0}$
2. Model: $p_\theta(x), \theta\in\Theta$
3. Log likelihood: $l_n(\theta)=\sum_{i=1}^{n}\log p_\theta(X_i)$
4. MLE: $\hat{\theta}_n \in \operatorname{arg max}_{\theta\in\Theta}l_n(\theta)$
5. Problem: $\hat{\theta}_n \longrightarrow \theta_0 ?$, $\sqrt{n}(\hat{\theta}_n - \theta_0) \longrightarrow ?$ 

![MLE](https://img.yulecna.top/posts/06_MLE/MLE_score_FisherI.png)
 
### Consistency of MLE

$\hat{\theta}_n \overset{p}{\rightarrow}\theta_0$ if $\hat{\theta}_n \in \operatorname*{arg max}_{\theta\in\Theta}l_n(\theta; X)$

**proof sketch:** 
$$
\begin{gather}
	W_i(\theta) = l(\theta; X_i) - l(\theta_0; X_i) \\
	\bar{W}_n = \dfrac{1}{n}\sum W_i \\
	\mu(\theta) = \mathbb{E}[W_i] = -D_{KL}(\theta_0 ||\theta) \\
	\delta_n = |\bar{W}_n(\theta) - \mu(\theta)|
\end{gather} 
$$
$$
\{\hat{\theta}_n \in \Theta_\epsilon\} \subset \{\bar{W}_n(\theta_0)<\bar{W}_n(\hat{\theta}_n^\epsilon)\}
$$
$$
\begin{gather}
	\Longrightarrow P(\hat{\theta}_n \in \Theta_\epsilon) < P(\bar{W}_n(\theta_0)<\bar{W}_n(\hat{\theta}_n^\epsilon)) \\
	\Longrightarrow P(\{\hat{\theta}_n \in \Theta_\epsilon\}) < P(-\delta_n<\mu(\hat{\theta}_n^\epsilon)+\delta_n)
\end{gather}
$$
moreover: 
$$
\begin{gather}
	\delta_n\overset{p}{\longrightarrow}0 \\
	\mu(\hat{\theta}_n^\epsilon) < 0
\end{gather}
$$
thus: 
$$
P(\{\hat{\theta}_n \in \Theta_\epsilon\}) \overset{p}{\longrightarrow} 0
$$

### Asymptotic Distribution of  MLE

$\sigma_n(\hat{\theta}_n-\theta_0)\Rightarrow N(0, J_1(\theta_0)^{-1})$