Data 145: Evidence and Uncertainty

Compact Final Review and Test-Selection Guide - Spring 2026
Instructors: Ani Adhikari, William Fithian

How to use this page: start with the course map, use the decision tree to identify a testing family, then check the compact tables for assumptions, calibration, and common traps. This is a final-pass reference rather than a replacement for the lecture guides.

Big Picture: The Course in Eight Jobs
Test Decision Map
Estimation and Decision Theory
Bayesian Inference, Robustness, and Bootstrap
Hypothesis Testing Foundations
Compact Test Reference
Multiple Testing and Linear Models
Hierarchical and Empirical Bayes
Theorems, Derivation Skeletons, and Exam Traps

Big Picture: The Course in Eight Jobs

Data 145 methods look less like a list when they are organized by the job they perform. Name the job first; then identify the random object that calibrates the answer.

Estimate

MLE, plug-in, Fisher information, delta method. Maximize likelihood, attach standard errors, and transform estimates.

Judge estimators

Risk, MSE, admissibility, shrinkage. “Good” depends on the loss function, not only on unbiasedness.

Update beliefs

Bayesian inference. Posterior is proportional to likelihood times prior; the estimator depends on the loss.

Survive wrong models

KL projection, sandwich variance, bootstrap. Separate the target of estimation from its sampling uncertainty.

Test hypotheses

NP, MLR, UMP, GLRT, and p-values. Control Type I error while comparing or maximizing power.

Handle many tests

Bonferroni, Sidak, and BH. Decide whether the goal is family-wise error control or false-discovery control.

Use Gaussian geometry

z, t, χ², F, regression, ANOVA. Rotate data into nuisance, tested, and residual directions.

Borrow strength

Hierarchical and empirical Bayes. Learn shrinkage from many parallel noisy problems.

First classify the structure of the data, then name the target or null question, and only then choose the test statistic and its calibration.

Test Decision Map

Begin at the top and classify the problem before reaching for a formula. The branches separate sample structure, model restrictions, linear-model questions, and multiplicity goals.

Sample structure Test or reduction Model-based method Multiple-testing method

Start: What test should I use? Classify the structure first, then scroll across the complete map.

The full map is horizontally scrollable so the labels stay readable on smaller screens.

$X^2$, $G^2$, KS, GLRT, ANOVA $F$, and permutation discrepancy tests reject for large values. Two-sided mean tests reject for large absolute values; sign and binomial tests use both tails for two-sided alternatives.

Estimation and Decision Theory

Regular MLE workflow

Write $L(\theta)=\prod_i f_\theta(X_i)$ and $\ell_n(\theta)=\sum_i\log f_\theta(X_i)$.
Differentiate: $S_n(\theta)=\ell_n'(\theta)$; solve $S_n(\hat\theta)=0$ or inspect the boundary and support.
Compute $I(\theta)=E_\theta[S_1(\theta;X)^2]$; under a correct regular model, $I(\theta)=-E_\theta[\ell_1''(\theta;X)]$.
Use asymptotic normality and plug in $\hat\theta$ for the standard error.

$$\sqrt n(\hat\theta-\theta_0)\Rightarrow N\!\left(0,\frac{1}{I(\theta_0)}\right),\qquad \widehat{\operatorname{SE}}(\hat\theta)=\frac{1}{\sqrt{nI(\hat\theta)}}.$$

Delta method

$$\sqrt n\{g(\hat\theta)-g(\theta_0)\}\Rightarrow N\!\left(0,\frac{[g'(\theta_0)]^2}{I(\theta_0)}\right).$$

Use for smooth transformed targets. If $g'(\theta_0)=0$ or the model is nonregular, first-order theory can fail.

Cramér–Rao lower bound

$$E_\theta[T]=g(\theta)\quad\Longrightarrow\quad \Var_\theta(T)\ge \frac{[g'(\theta)]^2}{nI(\theta)}.$$

This is a finite-sample lower bound for unbiased estimators under regularity conditions.

Common models

Model	MLE	$I(\theta)$ per observation	Key note
$\operatorname{Bernoulli}(p)$	$\hat p=\bar X$	$1/[p(1-p)]$	For logit targets, $g'(p)=1/[p(1-p)]$.
$\operatorname{Pois}(\lambda)$	$\hat\lambda=\bar X$	$1/\lambda$	$\bar X$ is finite-sample efficient for $\lambda$.
$\operatorname{Exp}(\lambda)$, rate	$\hat\lambda=1/\bar X$	$1/\lambda^2$	Do not confuse rate with the mean parameterization.
$N(\mu,\sigma^2)$, $\sigma^2$ known	$\hat\mu=\bar X$	$1/\sigma^2$	Exact normal sampling distribution.
$\operatorname{Unif}(0,\theta)$	$\hat\theta=X_{(n)}$	Regular identity fails	The support depends on $\theta$; regular MLE theory does not apply.

Decision theory

Risk and MSE

$$R(\theta;T)=E_\theta[L(\theta,T(X))],\qquad \operatorname{MSE}_\theta(T)=\Var_\theta(T)+\operatorname{Bias}_\theta(T)^2.$$

An estimator dominates another if its risk is never larger and is strictly smaller somewhere.

Bias can help

Shrinkage may add bias while reducing variance enough to lower total risk. Unbiasedness is a constraint, not a universal objective.

Admissible: no other estimator dominates it. Minimax: minimizes the worst-case risk.

Bayesian Inference, Robustness, and Bootstrap

Posterior workflow and conjugacy

Posterior

Write the likelihood kernel in $\theta$.
Multiply by the prior kernel.
Collect powers and exponential terms.
Recognize the posterior family.
Summarize according to the loss.

$$\pi(\theta\mid x)\propto_\theta f_\theta(x)\pi(\theta).$$

Bayes estimator by loss

Loss	Estimator
Squared error	Posterior mean
Absolute error	Posterior median
0–1 / MAP-like	Posterior mode

Likelihood	Prior	Posterior pattern
Binomial / Bernoulli	$\operatorname{Beta}(a,b)$	$\operatorname{Beta}(a+x,b+n-x)$
Poisson	$\operatorname{Gamma}(a,b)$ on rate	Gamma with shape $a+\sum x_i$ and rate $b+n$
Normal mean, variance known	Normal	Posterior precision = prior precision + data precision

Bernstein–von Mises: for a regular one-parameter model and large $n$, the posterior is approximately $N(\hat\theta_{\mathrm{MLE}},1/[nI(\hat\theta_{\mathrm{MLE}})])$. This is different from the many-parameter asymptotics used in empirical Bayes.

Misspecification and resampling

Wrong-model target

$$D_{\mathrm{KL}}(g\|f)=E_g\!\left[\log\frac{g(X)}{f(X)}\right],\qquad \hat\theta\xrightarrow{P}\arg\min_\theta D_{\mathrm{KL}}(g\|f_\theta).$$

Under misspecification, the MLE targets the closest model member—not a literal “true” parameter.

Sandwich variance

$$I=-E_g[\ell_1''(\theta^*;X)],\quad J=E_g[\ell_1'(\theta^*;X)^2],$$ $$\sqrt n(\hat\theta-\theta^*)\Rightarrow N(0,I^{-1}JI^{-1}).$$

The information identity $I=J$ requires correct specification.

Bootstrap choice	Rule	Use or trap
Parametric	Sample $X^*\sim f_{\hat\theta}$.	Efficient when the model is trusted; inherits misspecification.
Nonparametric	Sample with replacement from the observed data.	Fewer modeling assumptions.
Normal interval	$T\pm z_{1-\alpha/2}\widehat{\operatorname{sd}}(T^*)$	Needs approximate normality and low bias.
Percentile interval	$[q^_{\alpha/2},q^_{1-\alpha/2}]$	Simple, but can be bias-sensitive.
Basic interval	$[2T-q^_{1-\alpha/2},2T-q^_{\alpha/2}]$	Reflects the bootstrap error distribution.

Nonparametric methods trade model-based efficiency for robustness. They are not automatically assumption-free: permutation tests require exchangeability, rank tests target rank or distributional features, and bootstrap validity still depends on a stable sampling problem.

Hypothesis Testing Foundations

Neyman–Pearson and MLR

$$\operatorname{LR}(X)=\frac{f_1(X)}{f_0(X)};\qquad \text{reject for large LR}.$$

The NP lemma gives the most powerful level-$\alpha$ test for a fixed simple alternative.

If an exponential family has a monotone likelihood ratio in $T(X)$, thresholding $T$ can give a one-sided UMP test.

p-values and test–CI duality

$$p(x)=\sup_{\theta\in\Theta_0}P_\theta\{T(X)\ge T(x)\}.$$

A valid p-value satisfies $P_\theta\{p(X)\le t\}\le t$ under every null parameter.

A $1-\alpha$ confidence region contains the parameter values whose level-$\alpha$ tests do not reject.

GLRT

$$G^2=2\{\ell_n(\hat\theta)-\ell_n(\hat\theta_0)\} \Rightarrow \chi^2_{\dim(\Theta)-\dim(\Theta_0)}.$$

Use for regular nested parametric hypotheses. Unlike NP, this is not a finite-sample most-powerful guarantee.

Goodness of fit

$$X^2=\sum_j\frac{(O_j-E_j)^2}{E_j},\qquad G^2=2\sum_j O_j\log\frac{O_j}{E_j}.$$

With $k$ cells and $r$ parameters estimated from the data, the asymptotic degrees of freedom are $k-1-r$.

Likelihood-ratio simplification directions

Model / alternative	LR monotone in	Reject direction
$N(\mu,\sigma^2)$ known, $\mu_1>\mu_0$	$\bar X$	Large $\bar X$
$\operatorname{Exp}(\lambda)$ rate, $\lambda_1>\lambda_0$	$\sum X_i$ decreasing	Small $\sum X_i$
$\operatorname{Binom}(n,p)$, $p_1>p_0$	$X$	Large $X$
Pareto shape $\theta_1<\theta_0$	$\sum\log X_i$	Large sum for the smaller, heavier-tail shape

A p-value is a tail probability under the null; it is not $P(H_0\mid X)$. UMP claims usually require a one-sided alternative and monotone likelihood ratio structure. Wald and score tests are outside this final-review scope; the score function is still used for MLE and Fisher information.

Compact Test Reference

Use this table after the decision map. The last column is as important as the formula: it says when the calibration is justified.

Situation	Method	Statistic or rule	Calibration	Assumptions / traps
One mean, $\sigma$ known	$z$-test	$Z=(\bar X-\mu_0)/(\sigma/\sqrt n)$	$N(0,1)$	Normal model or justified large-$n$ approximation.
One normal mean, $\sigma$ unknown	One-sample $t$	$T=(\bar X-\mu_0)/(S/\sqrt n)$	$t_{n-1}$	IID normal for exact finite-sample calibration.
Paired before/after	Paired $t$	One-sample $t$ on $D_i=X_i-Y_i$	$t_{n-1}$	Preserve the pairing; do not treat samples as independent.
Paired, small or non-normal	Sign test	$S=\#\{D_i>0\}$ after dropping ties	$\operatorname{Binom}(k,1/2)$	Positive and negative signs are equally likely under $H_0$.
Two independent means	Welch $t$ by default	$(\bar X-\bar Y)/\widehat{\operatorname{SE}}$	Welch–Satterthwaite $t$	Use pooled $t$ only when equal variances are justified.
One fully specified continuous CDF	One-sample KS	$D_n=\sup_x\|F_n(x)-F_0(x)\|$	KS distribution	If parameters are estimated, refit inside a parametric bootstrap.
Two full distributions	Two-sample KS	$D_{m,n}=\sup_x\|F_m(x)-G_n(x)\|$	Exact, permutation, or asymptotic	Independent samples; cleanest for continuous distributions without ties.
Two samples, rank / location comparison	Wilcoxon rank-sum	Rank sum for one group	Permutation over labels	Targets ranks or a distributional shift, not automatically the mean.
Categorical goodness of fit	Pearson $X^2$ or $G^2$	$\sum(O-E)^2/E$ or $2\sum O\log(O/E)$	$\chi^2_{k-1-r}$	Expected counts must support the asymptotic approximation.
Three or more normal group means	One-way ANOVA	Between-group / within-group variation	$F_{g-1,n-g}$	Normal equal-variance model.
Three or more groups, no model form	$k$-sample permutation	Recompute a chosen discrepancy after relabeling	Permutation distribution	Exchangeability under the null; keep group sizes fixed.
Simple vs. simple hypotheses	NP likelihood ratio	Reject for large $f_1(X)/f_0(X)$	Choose cutoff under $H_0$	Optimal for the specified simple alternative only.
Composite nested parametric model	GLRT	$2\{\ell(\hat\theta)-\ell(\hat\theta_0)\}$	Asymptotic $\chi^2$	Regular nested models; degrees of freedom = dimension difference.
One regression coefficient	Coefficient $t$	$\hat\beta_j/\widehat{\operatorname{SE}}(\hat\beta_j)$	$t_{n-d}$	Normal linear model; $d$ is full-model dimension.
$d_1$ regression restrictions	Regression $F$	$[(\operatorname{RSS}_0-\operatorname{RSS})/d_1]/[\operatorname{RSS}/(n-d)]$	$F_{d_1,n-d}$	The denominator uses full-model residual variance.
Many hypotheses, FWER goal	Bonferroni / Sidak	$p_i\le\alpha/m$ or $p_i\le1-(1-\alpha)^{1/m}$	FWER control	Bonferroni allows any dependence; exact Sidak needs independence.
Many hypotheses, discovery goal	Benjamini–Hochberg	Largest $k$ with $p_{(k)}\le\alpha k/m$	FDR control	Independence or suitable positive dependence conditions.

Multiple Testing and Linear Models

Multiple testing

Quantities to separate

$$\operatorname{FWER}=P(V>0),\qquad \operatorname{FDP}=\frac{V}{R\vee1},\qquad \operatorname{FDR}=E[\operatorname{FDP}].$$

$R$ is the number of discoveries and $V$ is the number of false discoveries. Under the global null, FDR = FWER.

Procedure choice

Bonferroni: valid p-values, any dependence; conservative FWER control.
Sidak: independent true-null p-values; sharper FWER threshold.
BH: screening and discovery; controls expected false-discovery proportion under its dependence conditions.

Gaussian geometry

$$\sum_{i=1}^d Z_i^2\sim\chi^2_d,\qquad T=\frac{Z}{\sqrt{V/d}}\sim t_d,\qquad F=\frac{V_1/d_1}{V_2/d_2}\sim F_{d_1,d_2},\qquad t_d^2\sim F_{1,d}.$$

Independence of the signal and residual projections is the engine behind the exact $t$ and $F$ calibrations.

Canonical split

Rotate $Y\sim N_n(\theta,\sigma^2I)$ into independent blocks $Z=(Z_0,Z_1,Z_r)$.

$Z_0$: nuisance directions, ignored by the test.
$Z_1$: tested signal, with dimension $d_1$.
$Z_r$: residual noise, with dimension $d_r=n-\dim(\Theta)$.

Regression and ANOVA

$$F=\frac{(\operatorname{RSS}_0-\operatorname{RSS})/d_1}{\operatorname{RSS}/(n-d)}\sim F_{d_1,n-d}.$$

For one-way ANOVA, $d_1=g-1$ and the full-model residual degrees of freedom are $n-g$.

For a linear-model problem, identify the null subspace, the added signal directions, and the residual directions before writing the statistic. Degrees of freedom are dimensions, not numbers to memorize in isolation.

Hierarchical and Empirical Bayes

The many-means problem is where the course’s estimation, decision, and Bayesian threads meet. Each coordinate is noisy, but the collection teaches us how strongly to shrink.

$$X_i\mid\theta_i\sim N(\theta_i,1),\qquad \theta_i\stackrel{\mathrm{iid}}{\sim}N(\mu,\tau^2),\qquad \zeta=\frac{1}{1+\tau^2}.$$

Estimator	Form	Interpretation
Coordinatewise MLE	$\hat\theta_i=X_i$	No sharing across coordinates.
Oracle Bayes	$\zeta\mu+(1-\zeta)X_i$	Uses known population shrinkage.
Hierarchical Bayes	$E[\zeta\mu\mid X]+\{1-E[\zeta\mid X]\}X_i$	Integrates over hyperparameter uncertainty.
Empirical Bayes	Plug estimated $\mu$ and $\tau^2$ into the oracle rule	Learns the shrinkage coefficient from the ensemble.

Small $\tau^2$ means the group-level effects are similar, so shrink strongly toward $\mu$. Large $\tau^2$ means the effects are heterogeneous, so trust each $X_i$ more. Here the useful asymptotic is large dimension $d$, not many observations for each $\theta_i$.

Theorems, Derivation Skeletons, and Exam Traps

One-line theorem index

Result	Statement	Use / trap
Cramér–Rao	$\Var_\theta(T)\ge[g'(\theta)]^2/[nI(\theta)]$ for unbiased $T$.	Regular finite-sample lower bound.
MLE asymptotic normality	$\sqrt n(\hat\theta-\theta_0)\Rightarrow N(0,1/I(\theta_0))$.	Standard errors and large-sample intervals.
Delta method	Multiply asymptotic variance by $[g'(\theta_0)]^2$.	Smooth transformed targets.
Bernstein–von Mises	Posterior $\approx N(\hat\theta_{\mathrm{MLE}},1/[nI(\hat\theta)])$.	Regular large-$n$ one-parameter Bayes.
Neyman–Pearson	LR test is most powerful for simple vs. simple hypotheses.	Optimality is for a fixed alternative.
MLR $\Rightarrow$ UMP	Threshold a monotone statistic for one-sided alternatives.	Usually not a two-sided result.
Wilks / GLRT	$2\Delta\ell\Rightarrow\chi^2_{\text{dimension difference}}$.	Regular nested composite tests.
Rotational invariance	Orthogonal Gaussian projections are independent and chi-squared by dimension.	Foundation of $t$, $F$, regression, and ANOVA.
Bonferroni	$P(\bigcup_i A_i)\le\sum_iP(A_i)$.	FWER control without independence.
Benjamini–Hochberg	Under its conditions, FDR $\le\alpha m_0/m\le\alpha$.	Expected false-discovery proportion, not FWER.

Top derivation skeletons

MLE asymptotic: Taylor-expand $0=S_n(\hat\theta)$ around $\theta_0$.
Exponential rate: use $\sum X_i\sim\operatorname{Gamma}(n,\lambda)$ and inverse moments.
German tank: start from $P(X_{(n)}\le x)=(x/\theta)^n$ and bias-correct the maximum.
Normal–normal posterior: posterior precision = prior precision + data precision.
Bayes estimator: decompose posterior squared loss into posterior variance plus squared distance from the posterior mean.
NP proof: compare where two rejection regions disagree; the LR region buys more power per Type I error cost.
Normal GLRT: decompose total squared distance into residual variation plus $n(\bar X-\mu_0)^2$.
GOF degrees of freedom: begin with $k-1$ and subtract fitted parameters.
$t/F$: numerator is a signal projection; denominator is an independent residual-noise projection.
BH: sort p-values and find the largest crossing of $p_{(k)}\le\alpha k/m$.

Common traps

Exponential rate vs. mean: $\hat\lambda=1/\bar X$ but $\hat\theta=\bar X$.
Parameter-dependent support: regular Fisher and MLE asymptotics can fail.
Wrong degrees of freedom: GOF uses $k-1-r$; linear-model residual df is $n-\dim(\Theta)$.
Paired data: reduce to differences before testing.
Mean vs. distribution: a mean test and a KS test answer different questions.
Pooled variance: use only when the equal-variance model is justified.
KS after fitting: the textbook null is invalid; use a parametric bootstrap with refitting.

p-value: not the probability that the null is true.
Credible vs. confidence interval: different probability statements.
Bonferroni: does not require independence.
Sidak: exact product calibration does require independence.
BH: controls an expected false fraction, not the chance of any false discovery.
Bootstrap: estimates uncertainty; it does not automatically improve a point estimator.
Empirical Bayes: large $d$ learns shrinkage; it is not large $n$ for every coordinate.

For every problem, write this chain before computing: model and assumptions → target or null → statistic or estimator → calibration distribution, posterior, or asymptotic law → degrees of freedom or standard error → interpretation.