Review
Score function: S θ ( X ) = ∇ θ l ( θ , X ) S_\theta(X) = \nabla_\theta l(\theta, X) S θ ( X ) = ∇ θ l ( θ , X )
Local sufficient statistics:
p θ 0 + η ( X ) p θ 0 ( X ) ≈ exp { η T S θ 0 ( X ) } \frac{p_{\theta_0+\eta}(X)}{p_{\theta_0}(X)} \approx \exp\{\eta^TS_{\theta_0}(X)\} p θ 0 ( X ) p θ 0 + η ( X ) ≈ exp { η T S θ 0 ( X )}
Statistic property:
E [ S θ ( X ) ] = ∫ S θ ( x ) d P θ ( x ) = ∫ S θ ( x ) p θ ( x ) d μ ( x ) = ∇ θ ∫ p θ ( x ) d μ ( x ) = ∇ θ E θ [ X ] = 0 \begin{aligned}
\mathbb{E}[S_\theta(X)] &= \int S_\theta(x) dP_\theta(x) \\
&= \int S_\theta(x)p_\theta(x)\,d\mu(x) \\
&= \nabla_\theta \int p_\theta(x)\,d\mu(x) \\
&= \nabla_\theta\mathbb{E}_\theta[X] = 0
\end{aligned} E [ S θ ( X )] = ∫ S θ ( x ) d P θ ( x ) = ∫ S θ ( x ) p θ ( x ) d μ ( x ) = ∇ θ ∫ p θ ( x ) d μ ( x ) = ∇ θ E θ [ X ] = 0 Notation : This property holds true in any direction (any dimension of θ \theta θ
Fisher Information: J ( θ ) = V a r θ ( S θ ( X ) ) = E θ [ − ∇ 2 l ( θ ; X ) ] J(\theta) = Var_\theta(S_\theta(X)) = \mathbb{E}_\theta[-\nabla^2l(\theta; X)] J ( θ ) = V a r θ ( S θ ( X )) = E θ [ − ∇ 2 l ( θ ; X )] E [ ∇ θ l ( θ , X ) ] = 0 ⟹ 0 = ∂ ∂ θ j ∂ ∂ θ i E [ e l ] = ∫ ( ∂ 2 l ∂ θ j ∂ θ i + ∂ l ∂ θ j ∂ l ∂ θ i e l ) d μ = E θ [ ∇ 2 l ] + E θ [ S S T ] ⟹ E θ [ S S T ] = − E θ [ ∇ 2 l ] \begin{aligned}
& \mathbb{E}[\nabla_\theta l(\theta, X)] = 0 \\
\Longrightarrow
0 &= \dfrac{\partial}{\partial \theta_j}\dfrac{\partial}{\partial \theta_i}\mathbb{E}[e^l] \\
&= \int \left(\dfrac{\partial^2l}{\partial\theta_j\partial\theta_i} + \dfrac{\partial l}{\partial \theta_j}\dfrac{\partial l}{\partial\theta_i}e^l\right) \,d\mu \\
&= \mathbb{E}_\theta[\nabla^2l] + \mathbb{E}_\theta[SS^T] \\
\Longrightarrow
\mathbb{E}_\theta[SS^T] &= -\mathbb{E}_\theta[\nabla^2l]
\end{aligned} ⟹ 0 ⟹ E θ [ S S T ] E [ ∇ θ l ( θ , X )] = 0 = ∂ θ j ∂ ∂ θ i ∂ E [ e l ] = ∫ ( ∂ θ j ∂ θ i ∂ 2 l + ∂ θ j ∂ l ∂ θ i ∂ l e l ) d μ = E θ [ ∇ 2 l ] + E θ [ S S T ] = − E θ [ ∇ 2 l ]
J ( θ ) = V a r ( S θ ) J(\theta) = Var(S_\theta) J ( θ ) = V a r ( S θ ) J ( θ ) = − E [ ∇ 2 l ] J(\theta) = -\mathbb{E}[\nabla^2l] J ( θ ) = − E [ ∇ 2 l ]
Asymptotic
2 kinds of convergence:
cvg. in probability
cip to constant c ∈ R d c\in\mathbb{R}^d c ∈ R d P ( ∣ ∣ X n − c ∣ ∣ > ϵ ) → 0 , ∀ ϵ > 0 \mathbb{P}(||X_n-c||>\epsilon)\rightarrow 0, \forall \epsilon>0 P ( ∣∣ X n − c ∣∣ > ϵ ) → 0 , ∀ ϵ > 0
cvg. in distribution
cid to random variable X X X E f ( X n ) ⟶ E f ( X ) \mathbb{E}f(X_n)\longrightarrow\mathbb{E}f(X) E f ( X n ) ⟶ E f ( X ) f : x → R f: x\rightarrow\mathbb{R} f : x → R
Limit Theorems
Let X 1 , X 2 , … ∼ i i d X X_1, X_2, \dots \overset{iid}{\sim} X X 1 , X 2 , … ∼ ii d X X n ˉ = 1 n ∑ i = 1 n X i \bar{X_n} = \dfrac{1}{n}\sum_{i=1}^{n}X_i X n ˉ = n 1 ∑ i = 1 n X i
LLN : If E ∣ X ∣ < ∞ \mathbb{E}|X| < \infty E ∣ X ∣ < ∞ E [ X ] = μ \mathbb{E}[X] = \mu E [ X ] = μ
X n ˉ ⟶ p μ \bar{X_n} \overset{p}{\longrightarrow}\mu X n ˉ ⟶ p μ X n ˉ ⟶ a . s . μ \bar{X_n}\overset{a.s.}{\longrightarrow}\mu X n ˉ ⟶ a . s . μ
CLT : If E [ X ] = μ ∈ R d \mathbb{E}[X] = \mu \in \mathbb{R}^d E [ X ] = μ ∈ R d V a r ( X ) = Σ Var(X) = \Sigma V a r ( X ) = Σ
n ( X n ˉ − μ ) ⟶ d N ( 0 , Σ ) \sqrt{n} (\bar{X_n}-\mu) \overset{d}{\longrightarrow} N(0, \Sigma) n ( X n ˉ − μ ) ⟶ d N ( 0 , Σ )
Continuous Mapping
g g g X 1 , X 2 , … X_1, X_2, \dots X 1 , X 2 , …
If X n ⟶ d X X_n \overset{d}{\longrightarrow}X X n ⟶ d X g ( X n ) ⟶ d g ( X ) g(X_n)\overset{d}{\longrightarrow}g(X) g ( X n ) ⟶ d g ( X )
If X n ⟶ p c X_n \overset{p}{\longrightarrow}c X n ⟶ p c g ( X n ) ⟶ p g ( c ) g(X_n)\overset{p}{\longrightarrow}g(c) g ( X n ) ⟶ p g ( c )
Delta Method
If
{ n ( X n − μ ) ⟶ d N ( 0 , σ 2 ) f ( x ) differentiable at x = μ \begin{cases}
\sqrt{n}(X_n-\mu)\overset{d}{\longrightarrow}N(0, \sigma^2) \\
f(x)\text{ differentiable at }x=\mu
\end{cases} { n ( X n − μ ) ⟶ d N ( 0 , σ 2 ) f ( x ) differentiable at x = μ Then n ( f ( X n ) − f ( μ ) ) ⟶ d N ( 0 , f ˙ ( μ ) 2 σ 2 ) \sqrt{n}(f(X_n)-f(\mu))\overset{d}{\longrightarrow}N(0, \dot{f}(\mu)^2\sigma^2) n ( f ( X n ) − f ( μ )) ⟶ d N ( 0 , f ˙ ( μ ) 2 σ 2 ) X n ≈ N ( μ , σ 2 / n ) ⟶ d f ( X n ) ≈ N ( f ( μ ) , f ˙ ( μ ) σ 2 / n ) X_n\approx N(\mu, \sigma^2/n) \overset{d}{\longrightarrow} f(X_n)\approx N(f(\mu), \dot{f}(\mu)\sigma^2/n) X n ≈ N ( μ , σ 2 / n ) ⟶ d f ( X n ) ≈ N ( f ( μ ) , f ˙ ( μ ) σ 2 / n )
Maximum Likelihood Estimation
Setup:
Given: X 1 , … , X n ∼ i i d P θ 0 X_1, \dots, X_n \overset{iid}{\sim}P_{\theta_0} X 1 , … , X n ∼ ii d P θ 0
Model: p θ ( x ) , θ ∈ Θ p_\theta(x), \theta\in\Theta p θ ( x ) , θ ∈ Θ
Log likelihood: l n ( θ ) = ∑ i = 1 n log p θ ( X i ) l_n(\theta)=\sum_{i=1}^{n}\log p_\theta(X_i) l n ( θ ) = ∑ i = 1 n log p θ ( X i )
MLE: θ ^ n ∈ argmax θ ∈ Θ l n ( θ ) \hat{\theta}_n \in \operatorname{arg max}_{\theta\in\Theta}l_n(\theta) θ ^ n ∈ argmax θ ∈ Θ l n ( θ )
Problem: θ ^ n ⟶ θ 0 ? \hat{\theta}_n \longrightarrow \theta_0 ? θ ^ n ⟶ θ 0 ? n ( θ ^ n − θ 0 ) ⟶ ? \sqrt{n}(\hat{\theta}_n - \theta_0) \longrightarrow ? n ( θ ^ n − θ 0 ) ⟶ ?
Consistency of MLE
θ ^ n → p θ 0 \hat{\theta}_n \overset{p}{\rightarrow}\theta_0 θ ^ n → p θ 0 θ ^ n ∈ argmax θ ∈ Θ l n ( θ ; X ) \hat{\theta}_n \in \operatorname*{arg max}_{\theta\in\Theta}l_n(\theta; X) θ ^ n ∈ argmax θ ∈ Θ l n ( θ ; X )
proof sketch:
W i ( θ ) = l ( θ ; X i ) − l ( θ 0 ; X i ) W ˉ n = 1 n ∑ W i μ ( θ ) = E [ W i ] = − D K L ( θ 0 ∣ ∣ θ ) δ n = ∣ W ˉ n ( θ ) − μ ( θ ) ∣ \begin{gather}
W_i(\theta) = l(\theta; X_i) - l(\theta_0; X_i) \\
\bar{W}_n = \dfrac{1}{n}\sum W_i \\
\mu(\theta) = \mathbb{E}[W_i] = -D_{KL}(\theta_0 ||\theta) \\
\delta_n = |\bar{W}_n(\theta) - \mu(\theta)|
\end{gather} W i ( θ ) = l ( θ ; X i ) − l ( θ 0 ; X i ) W ˉ n = n 1 ∑ W i μ ( θ ) = E [ W i ] = − D K L ( θ 0 ∣∣ θ ) δ n = ∣ W ˉ n ( θ ) − μ ( θ ) ∣ { θ ^ n ∈ Θ ϵ } ⊂ { W ˉ n ( θ 0 ) < W ˉ n ( θ ^ n ϵ ) } \{\hat{\theta}_n \in \Theta_\epsilon\} \subset \{\bar{W}_n(\theta_0)<\bar{W}_n(\hat{\theta}_n^\epsilon)\} { θ ^ n ∈ Θ ϵ } ⊂ { W ˉ n ( θ 0 ) < W ˉ n ( θ ^ n ϵ )} ⟹ P ( θ ^ n ∈ Θ ϵ ) < P ( W ˉ n ( θ 0 ) < W ˉ n ( θ ^ n ϵ ) ) ⟹ P ( { θ ^ n ∈ Θ ϵ } ) < P ( − δ n < μ ( θ ^ n ϵ ) + δ n ) \begin{gather}
\Longrightarrow P(\hat{\theta}_n \in \Theta_\epsilon) < P(\bar{W}_n(\theta_0)<\bar{W}_n(\hat{\theta}_n^\epsilon)) \\
\Longrightarrow P(\{\hat{\theta}_n \in \Theta_\epsilon\}) < P(-\delta_n<\mu(\hat{\theta}_n^\epsilon)+\delta_n)
\end{gather} ⟹ P ( θ ^ n ∈ Θ ϵ ) < P ( W ˉ n ( θ 0 ) < W ˉ n ( θ ^ n ϵ )) ⟹ P ({ θ ^ n ∈ Θ ϵ }) < P ( − δ n < μ ( θ ^ n ϵ ) + δ n ) moreover:
δ n ⟶ p 0 μ ( θ ^ n ϵ ) < 0 \begin{gather}
\delta_n\overset{p}{\longrightarrow}0 \\
\mu(\hat{\theta}_n^\epsilon) < 0
\end{gather} δ n ⟶ p 0 μ ( θ ^ n ϵ ) < 0 thus:
P ( { θ ^ n ∈ Θ ϵ } ) ⟶ p 0 P(\{\hat{\theta}_n \in \Theta_\epsilon\}) \overset{p}{\longrightarrow} 0 P ({ θ ^ n ∈ Θ ϵ }) ⟶ p 0 Asymptotic Distribution of MLE
σ n ( θ ^ n − θ 0 ) ⇒ N ( 0 , J 1 ( θ 0 ) − 1 ) \sigma_n(\hat{\theta}_n-\theta_0)\Rightarrow N(0, J_1(\theta_0)^{-1}) σ n ( θ ^ n − θ 0 ) ⇒ N ( 0 , J 1 ( θ 0 ) − 1 )
