Bayes Estimation

A Frequentist Motivation

Bayes Estimation = A strategy for selecting a estimator

Prior: Use the average-case risk to reduce the risk function to a scalar summary. The average will be taken with respect to some measure $\Lambda$ on the parameter space $\Theta$, $\Lambda$ will be called prior.

i.e., for an estimator $\delta$, will define its Bayes risk with respect to $\Lambda$:

$$r_\Lambda(\delta)=\int_\Theta R(\theta; \delta) d\Lambda(\theta)$$

Bayes estimator: The estimator $\delta_\Lambda$ that will minimize bayes risk.

If $\Lambda$ is a probability measure, we call it proper, otherwise, $\Lambda(\Theta)=\infty$, and we call it improper. If $\Lambda$ is proper, the integral can be written as an expectation:

$$r_\Lambda(\delta) = \mathbb{E}_{\theta\sim\Lambda}[R(\theta; \delta)] = \mathbb{E}[L(\theta, \delta(X))]$$

The last expectation is taken wrt the joint distribution where:

$$\theta\sim\Lambda; \ X | \theta \sim P(\theta)$$

Posterior: conditional distribution of $\theta$ given X. i.e., giving desity $\lambda(\theta)$, joint distribution $\lambda(\theta)P_\theta(x)$, then the marginal distribution of x: $q(x) = \int_\Theta\lambda(\theta)P_\theta(x)d\theta$; The posterior is given by bayes' rule:

$$\lambda(\theta|x)=\frac{P_\theta(x)\cdot\lambda(\theta)}{q(x)}$$

Bayes Estimator

Theorem: Bayes Estimation: $\delta_\Lambda$ is bayes estimator iff $\delta_\Lambda \in \operatorname{argmin}(\mathbb{E}[L(\theta, d) | X=x])$, for a.e. $x$. The way to minimize the conditional expectation that defines $r_\Lambda$ is to find an estimator that minimize the conditional expectation of the loss given X.