---

title: Bayes Estimation

date: '2026-07-20 18:18:00'

tags:

- Statistics

cover: https://img.yulecna.top/posts/yn-background.jpg

description: ''

---
## A Frequentist Motivation

Bayes Estimation = A strategy for selecting a estimator

**Prior**: Use the average-case risk to reduce the risk function to a scalar summary. The average will be taken with respect to some measure $\Lambda$ on the parameter space $\Theta$, $\Lambda$ will be called prior. 

i.e., for an estimator $\delta$, will define its **Bayes risk** with respect to $\Lambda$: 
$$
r_\Lambda(\delta)=\int_\Theta R(\theta; \delta) d\Lambda(\theta)
$$
**Bayes estimator**: The estimator $\delta_\Lambda$ that will minimize bayes risk. 

If $\Lambda$ is a probability measure, we call it proper, otherwise, $\Lambda(\Theta)=\infty$, and we call it improper. If $\Lambda$ is proper, the integral can be written as an expectation:
$$
r_\Lambda(\delta) = \mathbb{E}_{\theta\sim\Lambda}[R(\theta; \delta)] = \mathbb{E}[L(\theta, \delta(X))]
$$
The last expectation is taken wrt the joint distribution where: 
$$
\theta\sim\Lambda; \ X | \theta \sim P(\theta)
$$
**Posterior**: conditional distribution of $\theta$ given X. i.e., giving desity $\lambda(\theta)$, joint distribution $\lambda(\theta)P_\theta(x)$, then the marginal distribution of x: $q(x) = \int_\Theta\lambda(\theta)P_\theta(x)d\theta$; The posterior is given by bayes' rule: 
$$
\lambda(\theta|x)=\frac{P_\theta(x)\cdot\lambda(\theta)}{q(x)}
$$

## Bayes Estimator

**Theorem: Bayes Estimation:** $\delta_\Lambda$ is bayes estimator iff $\delta_\Lambda \in \operatorname{argmin}(\mathbb{E}[L(\theta, d) | X=x])$,  for a.e. $x$. 
The way to minimize the conditional expectation that defines $r_\Lambda$ is to find an estimator that minimize the conditional expectation of the loss given X.