Hypothesis Testing & Linear Model

• simple test: Fully specify a distribution $\longleftrightarrow$ composite test
• one-sided hypotheses
• two-sided hypotheses
• two-sample nonparametric testing
• fail to reject

Usually, our goal is to minimize the type II error while controlling type I error bellow a prespecified rate $\alpha\in[0, 1]$ (called significance).

Significance level and power

Critical function

$$\phi(x)=\begin{cases} 0 & accept\,H_0\\ \gamma\in(0, 1) & reject\,w.p. \gamma \\ 1 & reject\,H_0 \end{cases}$$

or more commonly used, choosing a test statistic $T(X)$ and a critical threshold:

$$\phi(x)=\begin{cases} 0 & T(X)<c\\ \gamma\in(0, 1) & T(X)=c \\ 1 & T(X)>c \end{cases}$$

Power function $\beta_\phi(\theta)=\mathbb{E}_\theta[\phi(X)]=P_\theta(Reject \ H_0)$. Our goal can be expressed as

$$\operatorname*{arg\,max}_{\phi} \,\mathbb{E}_\theta[\phi(X)] \, for \, \theta\in\Theta_1 \ \ subject \, to \, \mathbb{E}_\theta[\phi(X)] \leq \alpha \, for \, \theta\in\Theta_0$$

Likelihood Ratio Test

Neyman-Pearson Lemma:

UMP Test:

p-values, confidence regions, and (mis-)interpreting Tests

Definition: P value (Informal): The probability of a test statistic T(X) to be at least as big as its realized value T(x), under the assumption that $H_0$ is true.

$$p(x)=\operatorname*{sup}_{\theta\in\Theta_0}\mathbb{P}_\theta(T(X)\geq T(x))$$

Definition: P value (formal) :

$$p(x)=\sup\{\alpha: \phi_\alpha(x) < 1\}=\inf\{\alpha: \phi_\alpha(x)=1\}$$

in terms of the rejection regions:

$$p(x)=\sup\{\alpha: x\notin R_\alpha\}=\inf\{\alpha: x\in R_\alpha\}$$

Confidence Interval/Region: $C(X)$ is a $1-\alpha$ confidence region for $g(\theta)$ if:

$$P_\theta(C(X))\ni g(\theta))\geq1-\alpha \quad \text{for all}\, \theta\in\Theta$$

Duality with Test:

Suppose we have a level-$\alpha$ test $\phi(X, a)$ of $H_0: g(\theta)=a\text{ v.s. }H_1: g(\theta)\neq a$ for every $a$, then we can use these tests to construct a (non-randomized) confidence region for $g(\theta)$ as follows:

$$C(X)=\{a: \phi(X, a)<1\}$$

$C(X)$ is all non-rejected values of $a$, and it is a valid confidence region because

$$\mathbb{P}_\theta(C(X)\ni g(\theta)) = P_\theta(\phi(X;g(\theta))<1)\geq1-\alpha$$

How to derive a level-$\alpha$ test $\phi_\alpha$ from a confidence region?

$$\phi_\alpha(x)=\begin{cases} 0 & C(x)\ni a \\ 1 & C(x) \not\ni a \end{cases}$$

Testing in Linear Model

$\chi, \,t, \, F$ distributions

1. If $z_1,\dots,z_d\overset{\text{i.i.d.}}{\sim}\mathcal{N}(0, 1)$, then $$V=\sum_d Z_i^2\sim\chi_d^2=Gamma(d/2, 2)$$
   1. $\mathbb{E}V=d$, $Var(V)=2d$
   2. CLT: $\dfrac{V-d}{\sqrt{2d}}\Rightarrow N(0, 1)$
2. If $z\sim\mathcal{N}(0, \sigma^2)$ and $V\sim\sigma^2\chi_d^2$, $z\perp V$, then $$\frac{z}{\sqrt{v/d}}\sim t_d\Rightarrow \mathcal{N}(0, 1)\,as\,d\rightarrow\infty$$
3. If $V_1\sim\sigma^2\chi_{d_1}^2$ and $V_2\sim\sigma^2\chi_{d_2}^2$, $V_1\perp V_2$, then $$\frac{V_1/d_1}{V_2/d_2}\sim F_{d_1,d_2}\Rightarrow\dfrac{1}{d_1}\chi_{d_1}^2 \,as\, d_2\rightarrow\infty$$
   1. If $T\sim t_d$, then $T^2\sim F_{1, d}$
4. If $T\sim t_d$ then $T^2\sim F_{1,d}$

$z_d\longrightarrow \chi_d^2\longrightarrow t_d$ $\chi_{d_1}^2, \chi_{d_2}^2\longrightarrow F_{d_1, d_2}$

Change Basis

Assume $X_i \sim N(\mu, \sigma^2)$, construct

$$Q=\begin{bmatrix} q_1 & q_2 & \dots & q_n \end{bmatrix}$$

Where

$$q_1=\frac{1}{\sqrt{n}}1_n$$

$Q$ is a Orthogonal Unit Matrix:

$$Q=\begin{bmatrix} q_1 & Q_r \end{bmatrix}$$

with $Q'Q=I_n$.

We have

$$Z = Q'X \sim N_n\left( \begin{bmatrix} \sqrt{n}\mu \\ 0 \\ \vdots \\ 0 \end{bmatrix}, \sigma^2 I_n \right)$$ $$Z_r = Q_r'X \sim N(0, \sigma^2 I_{n-1}) \Rightarrow S^2= \dfrac{1}{n-1} ||Z_r||^2 \sim \frac{\sigma^2}{n-1}\chi_{n-1}^{2}$$

and $S^2 \perp Z_1$.

Canonical Linear Model

1. $\sigma^2$ known, $d_1=1$
2. $\sigma^2$ known, $d_1\geq1$
3. $\sigma^2$ unknown, $d_1=1$
4. $\sigma^2$ unknown, $d_1\geq1$

General Linear Model

Basic Setup:

1. Observe $Y\sim N_n(\theta, \sigma^2I_n)$, $\sigma^2>0$.
2. Test $\theta\in\Theta_0$ vs. $\theta\in\Theta \backslash \Theta_0$.
3. $\Theta_0\subset\Theta$, with $dim(\Theta_0)=d_0$, $dim(\Theta)=d=d_0+d_1$.

We rotate it into canonical form:

$$Z = Q'Y\sim N_n\left(\begin{bmatrix} Q_0'\theta \\ Q_1'\theta \\ 0 \end{bmatrix}, \sigma^2I_n\right)$$