Bayesian Statistics

Bayesian statistics is an approach to inference that treats probability as a degree of belief, updated in light of new evidence. While the frequentist approach (covered in earlier lessons) interprets probability as a long-run frequency, the Bayesian approach offers a principled way to combine prior knowledge with data.

---

Bayes' Theorem

At the heart of Bayesian statistics is Bayes' Theorem:

P(A | B) = P(B | A) × P(A) / P(B)

In the context of statistical inference:

P(θ | data) = P(data | θ) × P(θ) / P(data)

Term	Name	Meaning
P(θ | data)	Posterior	Updated belief about parameter θ after seeing data
P(data | θ)	Likelihood	How probable the data is, given θ
P(θ)	Prior	Belief about θ before seeing data
P(data)	Marginal likelihood (Evidence)	Total probability of the data across all possible θ values

---

A Classic Example: Medical Testing

A disease affects 1% of the population. A test has:

• Sensitivity (true positive rate): 99%
• Specificity (true negative rate): 95%

If you test positive, what is the probability you actually have the disease?

P(disease | positive) = P(positive | disease) × P(disease) / P(positive)

P(positive) = P(positive | disease)P(disease) + P(positive | no disease)P(no disease)
            = 0.99 × 0.01 + 0.05 × 0.99
            = 0.0099 + 0.0495
            = 0.0594

P(disease | positive) = 0.0099 / 0.0594 ≈ 0.167 ≈ 16.7%

Despite a 99% accurate test, a positive result only gives a 16.7% chance of actually having the disease — because the disease is rare (low prior).

---

Bayesian vs. Frequentist

Aspect	Frequentist	Bayesian
Definition of probability	Long-run frequency	Degree of belief
Parameters	Fixed but unknown constants	Random variables with distributions
Prior information	Not formally included	Incorporated via prior distributions
Result	p-values, confidence intervals	Posterior distributions, credible intervals
Interpretation of intervals	"If we repeat the experiment, 95% of intervals would contain the true value"	"There is a 95% probability that the parameter lies in this interval"

---

Prior Distributions

The prior reflects your beliefs about the parameter before collecting data.

Types of Priors

Type	Description	Example
Informative prior	Encodes specific prior knowledge	Previous studies suggest μ ≈ 50 ± 5
Weakly informative	Mildly constrains the parameter space	We know μ is positive
Non-informative (vague)	Attempts to let data dominate	Uniform prior over a wide range
Conjugate prior	A prior that, when combined with the likelihood, yields a posterior of the same family	Beta prior + Binomial likelihood → Beta posterior