Normal-Inverse Gamma Model for Variance
Assumptions
- We observe $n$ independent data points $y_1, y_2, \dots, y_n$
- Each $y_i \sim N(\mu, \sigma^2)$ where $\mu$ is known and fixed
- The unknown parameter of interest is $\sigma^2$ (variance) alone
- The sufficient statistic for $\sigma^2$ is $S_\mu = \sum_{i=1}^n (y_i – \mu)^2$
- We present two approaches. Approach A places an Inverse Gamma prior directly on $\sigma^2$. Approach B reparameterises to precision $\phi = 1/\sigma^2$ and places a Gamma prior on $\phi$. Both lead to identical posterior inference.
Approach A: Normal — Inverse Gamma
Step 1. The Likelihood
$$y_i \mid \sigma^2 \sim N(\mu, \sigma^2), \quad i = 1, \dots, n, \quad \text{independently}$$
The joint likelihood is:
$$\mathscr{L}(\sigma^2 \mid \mathbf{y}) = \prod_{i=1}^n \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(y_i-\mu)^2}{2\sigma^2}\right)$$
$$= \frac{1}{(2\pi\sigma^2)^{n/2}} \exp\left(-\frac{\sum_{i=1}^n(y_i-\mu)^2}{2\sigma^2}\right)$$
$$= \frac{1}{(2\pi)^{n/2}(\sigma^2)^{n/2}} \exp\left(-\frac{S_\mu}{2\sigma^2}\right)$$
where $S_\mu = \displaystyle\sum_{i=1}^n(y_i – \mu)^2$.
Step 2. The Prior
$$\sigma^2 \sim \text{IG}(\alpha, \beta)$$
$$p(\sigma^2) = \frac{\beta^\alpha}{\Gamma(\alpha)}\,(\sigma^2)^{-(\alpha+1)}\,e^{-\beta/\sigma^2}, \quad \sigma^2 > 0$$
Step 3. Bayes’ Theorem from First Principles
From the definition of conditional probability:
$$p(\sigma^2,\, \mathbf{y}) = p(\mathbf{y} \mid \sigma^2)\cdot p(\sigma^2)$$
$$p(\sigma^2,\, \mathbf{y}) = p(\sigma^2 \mid \mathbf{y})\cdot p(\mathbf{y})$$
Setting equal and dividing by $p(\mathbf{y})$:
$$\boxed{p(\sigma^2 \mid \mathbf{y}) = \frac{p(\mathbf{y} \mid \sigma^2)\cdot p(\sigma^2)}{p(\mathbf{y})}}$$
where:
$$p(\mathbf{y}) = \int_0^\infty p(\mathbf{y} \mid \sigma^2)\, p(\sigma^2)\, d\sigma^2$$
Step 4. The Marginal Likelihood — Full Derivation
Substituting:
$$p(\mathbf{y}) = \int_0^\infty \frac{1}{(2\pi)^{n/2}(\sigma^2)^{n/2}} \exp\left(-\frac{S_\mu}{2\sigma^2}\right) \cdot \frac{\beta^\alpha}{\Gamma(\alpha)}\,(\sigma^2)^{-(\alpha+1)}\,e^{-\beta/\sigma^2}\, d\sigma^2$$
Pulling constants out:
$$p(\mathbf{y}) = \frac{\beta^\alpha}{(2\pi)^{n/2}\,\Gamma(\alpha)} \int_0^\infty (\sigma^2)^{-(\alpha + n/2 + 1)} \exp\left(-\frac{S_\mu/2 + \beta}{\sigma^2}\right) d\sigma^2$$
Now we recognise this as an Inverse Gamma integral. Recall:
$$\int_0^\infty (\sigma^2)^{-(a+1)}\,e^{-b/\sigma^2}\, d\sigma^2 = \frac{\Gamma(a)}{b^a}$$
Matching with $a = \alpha + n/2$ and $b = \beta + S_\mu/2$:
$$\int_0^\infty (\sigma^2)^{-(\alpha+n/2+1)} \exp\left(-\frac{\beta + S_\mu/2}{\sigma^2}\right) d\sigma^2 = \frac{\Gamma(\alpha + n/2)}{(\beta + S_\mu/2)^{\alpha+n/2}}$$
Substituting back:
$$\boxed{p(\mathbf{y}) = \frac{\beta^\alpha}{(2\pi)^{n/2}\,\Gamma(\alpha)} \cdot \frac{\Gamma\left(\alpha + \dfrac{n}{2}\right)}{\left(\beta + \dfrac{S_\mu}{2}\right)^{\alpha + n/2}}}$$
Step 5. Substituting into Bayes’ Theorem
Numerator:
$$p(\mathbf{y} \mid \sigma^2)\cdot p(\sigma^2) = \frac{\beta^\alpha}{(2\pi)^{n/2}\,\Gamma(\alpha)}\,(\sigma^2)^{-(\alpha+n/2+1)}\exp\left(-\frac{\beta + S_\mu/2}{\sigma^2}\right)$$
Denominator:
$$p(\mathbf{y}) = \frac{\beta^\alpha}{(2\pi)^{n/2}\,\Gamma(\alpha)} \cdot \frac{\Gamma\left(\alpha + \dfrac{n}{2}\right)}{\left(\beta + \dfrac{S_\mu}{2}\right)^{\alpha+n/2}}$$
Dividing:
$$p(\sigma^2 \mid \mathbf{y}) = \frac{\dfrac{\beta^\alpha}{(2\pi)^{n/2}\Gamma(\alpha)}\,(\sigma^2)^{-(\alpha+n/2+1)}\exp\left(-\dfrac{\beta+S_\mu/2}{\sigma^2}\right)}{\dfrac{\beta^\alpha}{(2\pi)^{n/2}\Gamma(\alpha)}\cdot\dfrac{\Gamma(\alpha+n/2)}{(\beta+S_\mu/2)^{\alpha+n/2}}}$$
The term $\dfrac{\beta^\alpha}{(2\pi)^{n/2}\,\Gamma(\alpha)}$ cancels:
$$p(\sigma^2 \mid \mathbf{y}) = \frac{(\beta + S_\mu/2)^{\alpha+n/2}}{\Gamma(\alpha + n/2)}\,(\sigma^2)^{-(\alpha+n/2+1)}\exp\left(-\frac{\beta + S_\mu/2}{\sigma^2}\right)$$
This is exactly the Inverse Gamma density with updated hyperparameters $\alpha_n = \alpha + n/2$ and $\beta_n = \beta + S_\mu/2$:
$$\boxed{p(\sigma^2 \mid \mathbf{y}) = \frac{\beta_n^{\alpha_n}}{\Gamma(\alpha_n)}\,(\sigma^2)^{-(\alpha_n+1)}\,e^{-\beta_n/\sigma^2}}$$
$$\therefore \quad \sigma^2 \mid \mathbf{y} \sim \text{IG}\left(\alpha + \frac{n}{2},\ \beta + \frac{S_\mu}{2}\right)$$
Normal-Gamma Model for Precision
Approach B: Normal — Gamma (Precision)
Step 1. The Likelihood
Reparameterising with $\phi = 1/\sigma^2$, so $\sigma^2 = 1/\phi$:
$$p(y_i \mid \phi) = \sqrt{\frac{\phi}{2\pi}}\exp\left(-\frac{\phi(y_i-\mu)^2}{2}\right)$$
The joint likelihood for $n$ observations is:
$$\mathscr{L}(\phi \mid \mathbf{y}) = \prod_{i=1}^n \sqrt{\frac{\phi}{2\pi}}\exp\left(-\frac{\phi(y_i-\mu)^2}{2}\right)$$
$$= \left(\frac{\phi}{2\pi}\right)^{n/2} \exp\left(-\frac{\phi}{2}\sum_{i=1}^n(y_i-\mu)^2\right)$$
$$= \frac{\phi^{n/2}}{(2\pi)^{n/2}} \exp\left(-\frac{\phi\, S_\mu}{2}\right)$$
Step 2. The Prior
$$\phi \sim \text{Gamma}(\alpha, \beta)$$
$$p(\phi) = \frac{\beta^\alpha}{\Gamma(\alpha)}\,\phi^{\alpha-1}\,e^{-\beta\phi}, \quad \phi > 0$$
Step 3. Bayes’ Theorem from First Principles
$$p(\phi,\, \mathbf{y}) = p(\mathbf{y} \mid \phi)\cdot p(\phi)$$
$$p(\phi,\, \mathbf{y}) = p(\phi \mid \mathbf{y})\cdot p(\mathbf{y})$$
Setting equal and dividing by $p(\mathbf{y})$:
$$\boxed{p(\phi \mid \mathbf{y}) = \frac{p(\mathbf{y} \mid \phi)\cdot p(\phi)}{p(\mathbf{y})}}$$
where:
$$p(\mathbf{y}) = \int_0^\infty p(\mathbf{y} \mid \phi)\, p(\phi)\, d\phi$$
Step 4. The Marginal Likelihood — Full Derivation
Substituting:
$$p(\mathbf{y}) = \int_0^\infty \frac{\phi^{n/2}}{(2\pi)^{n/2}} \exp\left(-\frac{\phi\, S_\mu}{2}\right) \cdot \frac{\beta^\alpha}{\Gamma(\alpha)}\,\phi^{\alpha-1}\,e^{-\beta\phi}\, d\phi$$
Pulling constants out:
$$p(\mathbf{y}) = \frac{\beta^\alpha}{(2\pi)^{n/2}\,\Gamma(\alpha)} \int_0^\infty \phi^{\alpha + n/2 – 1}\,\exp\left(-\phi\left(\beta + \frac{S_\mu}{2}\right)\right) d\phi$$
Recognising the Gamma integral:
$$\int_0^\infty \phi^{a-1}\,e^{-b\phi}\, d\phi = \frac{\Gamma(a)}{b^a}$$
Matching with $a = \alpha + n/2$ and $b = \beta + S_\mu/2$:
$$\int_0^\infty \phi^{\alpha+n/2-1}\exp\left(-\phi\left(\beta+\frac{S_\mu}{2}\right)\right)d\phi = \frac{\Gamma(\alpha + n/2)}{\left(\beta + S_\mu/2\right)^{\alpha+n/2}}$$
Substituting back:
$$\boxed{p(\mathbf{y}) = \frac{\beta^\alpha}{(2\pi)^{n/2}\,\Gamma(\alpha)} \cdot \frac{\Gamma\left(\alpha + \dfrac{n}{2}\right)}{\left(\beta + \dfrac{S_\mu}{2}\right)^{\alpha + n/2}}}$$
Step 5. Substituting into Bayes’ Theorem
Numerator:
$$p(\mathbf{y} \mid \phi)\cdot p(\phi) = \frac{\beta^\alpha}{(2\pi)^{n/2}\,\Gamma(\alpha)}\,\phi^{\alpha+n/2-1}\exp\left(-\phi\left(\beta + \frac{S_\mu}{2}\right)\right)$$
Denominator:
$$p(\mathbf{y}) = \frac{\beta^\alpha}{(2\pi)^{n/2}\,\Gamma(\alpha)} \cdot \frac{\Gamma\left(\alpha+\dfrac{n}{2}\right)}{\left(\beta+\dfrac{S_\mu}{2}\right)^{\alpha+n/2}}$$
Dividing:
$$p(\phi \mid \mathbf{y}) = \frac{\dfrac{\beta^\alpha}{(2\pi)^{n/2}\Gamma(\alpha)}\,\phi^{\alpha+n/2-1}\exp\left(-\phi\left(\beta+\dfrac{S_\mu}{2}\right)\right)}{\dfrac{\beta^\alpha}{(2\pi)^{n/2}\Gamma(\alpha)}\cdot\dfrac{\Gamma(\alpha+n/2)}{\left(\beta+S_\mu/2\right)^{\alpha+n/2}}}$$
The term $\dfrac{\beta^\alpha}{(2\pi)^{n/2}\,\Gamma(\alpha)}$ cancels:
$$p(\phi \mid \mathbf{y}) = \frac{\left(\beta + S_\mu/2\right)^{\alpha+n/2}}{\Gamma(\alpha+n/2)}\,\phi^{\alpha+n/2-1}\exp\left(-\phi\left(\beta+\frac{S_\mu}{2}\right)\right)$$
This is exactly the Gamma density with updated hyperparameters $\alpha_n = \alpha + n/2$ and $\beta_n = \beta + S_\mu/2$:
$$\boxed{p(\phi \mid \mathbf{y}) = \frac{\beta_n^{\alpha_n}}{\Gamma(\alpha_n)}\,\phi^{\alpha_n – 1}\,e^{-\beta_n \phi}}$$
$$\therefore \quad \phi \mid \mathbf{y} \sim \text{Gamma}\left(\alpha + \frac{n}{2},\ \beta + \frac{S_\mu}{2}\right)$$
Conclusion
Both approaches yield posteriors with identical updated hyperparameters:
$$\alpha_n = \alpha + \frac{n}{2} \qquad \beta_n = \beta + \frac{S_\mu}{2} = \beta + \frac{\sum_{i=1}^n(y_i-\mu)^2}{2}$$
$$\boxed{\sigma^2 \mid \mathbf{y} \sim \text{IG}\left(\alpha + \frac{n}{2},\ \beta + \frac{S_\mu}{2}\right)}$$
$$\boxed{\phi \mid \mathbf{y} \sim \text{Gamma}\left(\alpha + \frac{n}{2},\ \beta + \frac{S_\mu}{2}\right)}$$
Approach A worked directly with $\sigma^2$, requiring recognition of the Inverse Gamma integral $\int_0^\infty (\sigma^2)^{-(a+1)} e^{-b/\sigma^2} d\sigma^2 = \Gamma(a)/b^a$.
Approach B reparameterised to $\phi = 1/\sigma^2$, turning the integral directly into a standard Gamma integral $\int_0^\infty \phi^{a-1} e^{-b\phi} d\phi = \Gamma(a)/b^a$ — no substitution required.
Both marginal likelihoods are identical in closed form, confirming full consistency.
The two posteriors are related simply by $\phi = 1/\sigma^2$, as expected from the transformation between variance and precision.