Introduction
A random variable X is completly described by its CDF, PDF, or PMF. But summarizing a distribution would be convenient. This can be acheived by deriving a moments of a distribution. Also, if a procedure helps to derive the moments upto the required order then it will be helpful in avoiding the redundency in computing the individual moments. Three such generating functions are discussed in this notes
It is covered by three functions:
- The moment generating function
- The cumulant generating function
- The characteristic function.
All of these follows same form that is ‘the expectation of an exponential of X’.
MOMENT GENERATING FUNCTION (MGF)
Definition
The MGF of $X$ is defined as an expectation:
$$M_X(t) = E[e^{tX}]$$
For a discrete random variable with PMF $p_X(x)$:
$$M_X(t) = \sum_x e^{tx}\, p_X(x)$$
For a continuous random variable with PDF $f_X(x)$:
$$M_X(t) = \int_{-\infty}^{\infty} e^{tx}\, f_X(x)\, dx$$
The MGF only exists if this sum or integral is finite for $t$ in some open interval around $0$. If it diverges for every $t$ other than $0$, the MGF simply does not exist for that random variable — it happens for any distribution with a heavy enough tail, as shown later.
Why we refer MGF as Moment Generating Function
Expanding the exponential as a power series and taking expectation term by term shows why this function is called “moment generating”:
$$e^{tX} = 1 + tX + \frac{t^2X^2}{2!} + \frac{t^3X^3}{3!} + \cdots$$
$$M_X(t) = 1 + t\,E[X] + \frac{t^2}{2!}E[X^2] + \frac{t^3}{3!}E[X^3] + \cdots$$
Approach 1:
The coefficient of $t^n/n!$ is $E[X^n]$, the n-th raw moment of $X$.
Approach 2:
Differentiating the above series $n$ times and evaluating at $t = 0$ gives the nth raw moment:
$$E[X^n] = \left[ \frac{d^n}{dt^n} M_X(t) \right]_{t=0}$$
$$E[X^n] = M_X^{(n)}(0)$$
specifically,
$$E[X] = \left[ \frac{d}{dt} M_X(t) \right]_{t=0}$$
$$E[X^2] = \left[ \frac{d^2}{dt^2} M_X(t) \right]_{t=0}$$
Key Properties
If $Y = aX + b$, the MGF transforms as:
$$M_Y(t) = e^{tb}\, M_X(at)$$
If $X$ and $Y$ are independent, the MGF of their sum is the product of their individual MGFs:
$$M_{X+Y}(t) = M_X(t)\, M_Y(t)$$
And if two random variables share the same MGF on an interval around $0$, they must have the same distribution — this makes the MGF to identifying a distribution.
Example 1: Discrete Random Variable
Let $X$ be a random variable with range $\mathscr{A}_x=\{0, 1, 2\}$ with probabilities $p_0, p_1, p_2$, where $p_0 + p_1 + p_2 = 1$.
therefore,
$$M_X(t) = E[e^{tX}] = p_0 e^{t \cdot 0} + p_1 e^{t \cdot 1} + p_2 e^{t \cdot 2}$$
$$M_X(t) = p_0 + p_1 e^{t} + p_2 e^{2t}$$
Differentiate once with respect to t.
$$M_X'(t) = p_1 e^{t} + 2p_2 e^{2t}$$
Evaluate at $t = 0$.
$$M_X'(0) = p_1 + 2p_2$$
since,
$E[X] = 0 \cdot p_0 + 1 \cdot p_1 + 2 \cdot p_2 = p_1 + 2p_2$
Differentiate a second time and evaluate again at t = 0.
$$M_X”(t) = p_1 e^{t} + 4p_2 e^{2t}$$
$$M_X”(0) = p_1 + 4p_2 = E[X^2]$$
Example 2: Continuous Random Variable
If the pdf of a random variable $X$ is
$$ f(x) = \begin{cases} \dfrac{1}{2} & |x| < 1 \\ 0 & \text{elsewhere} \end{cases}$$
Find its MGF and hence its mean and variance.
$$M_X(t) = E(e^{tx}) = \int_{-\infty}^{\infty} e^{tx} f(x) \, dx$$
$$= \int_{-1}^{1} \frac{1}{2} (e^{tx}) \, dx = \left( \frac{e^{tx}}{2t} \right)_{-1}^{1}$$
$$M_X(t) = \frac{e^t – e^{-t}}{2t}$$
$$= \frac{1}{2t}\left[ \left(1 + \frac{t}{1!} + \frac{t^2}{2!} + \frac{t^3}{3!} + \cdots \right) – \left(1 – \frac{t}{1!} + \frac{t^2}{2!} – \frac{t^3}{3!} + \cdots \right) \right]$$
$$= \frac{1}{2t}\left[ \frac{2t}{1!} + \frac{2t^3}{3!} + \frac{2t^5}{5!} + \cdots \right]$$
$$= 1 + \frac{1}{3!}t^2 + \frac{1}{5!}t^4 + \cdots$$
$$E(X) = \text{coefficient of } \left( \frac{t}{1!} \right) \text{ in } M_X(t) = 0$$
$$E(X^2) = \text{coefficient of } \left( \frac{t^2}{2!} \right) \text{ in } M_X(t) = \frac{1}{3}$$
$$\therefore \quad \text{Mean} = 0 \text{ and variance} = \frac{1}{3}$$
In this example differentiating $M_X(t)$ with respect to $t$ and assigning $t = 0$ will not bring a finite value, so it is necessary to use the expansion principle to obtain mean and variance of $X$.
CUMULANT GENERATING FUNCTION (CGF)
Definition
Let $K_X(t) = \log_e M_X(t)$, provided the right-hand side can be expanded as a convergent series in power of $t$. The coefficient of $\left(\dfrac{t^r}{r!}\right)$ in this series is called the $r^{th}$ cumulant and it is denoted by $K_r$.
This cumulants are helpful in finding central moments, infact without finding non-central moments.
$$K_X(t) = \log[M_X(t)]$$
$$= \log\left[1 + \frac{t}{1!}E(X) + \frac{t^2}{2!}E(X^2) + \frac{t^3}{3!}E(X^3) + \cdots \right]$$
$$= \left[ \left(\frac{t}{1!}\right)E(X) + \frac{t^2}{2!}E(X^2) + \frac{t^3}{3!}E(X^3) + \cdots \right] – \frac{1}{2}\left[ \frac{t}{1!}E(X) + \frac{t^2}{2!}E(X^2) + \cdots \right]^2 +$$
$$\frac{1}{3}\left[ \frac{t}{1!}E(X) + \frac{t^2}{2!}E(X^3) + \frac{t^3}{3!}E(X^3) + \cdots \right]^3 – \cdots$$
$$= \frac{t}{1!}E(X) + \frac{t^2}{2!}\left[E(X^2) – E(X)^2\right] + \frac{t^3}{3!}\left[E(X^3) – 3E(X)E(X^2) + \cdots + 2E(X^3)\right]$$
That is
$$K_1 t + K_2\left(\frac{t^2}{2!}\right) + K_3\left(\frac{t^3}{3!}\right) + \cdots = \left(\frac{t}{1!}\right)E(X) + \left(\frac{t^2}{2!}\right)(\mu_2) + \left(\frac{t^3}{3!}\right)(\mu_3) + \cdots$$
Comparing the coefficients of $\dfrac{t^r}{r!}$, we have
$$K_1 = E(X) = \text{Mean}; \quad K_2 = V(X); \quad K_3 = \mu_3; \quad K_4 = \mu_4 – 3k_2^2 \cdots$$
Remark
If we differentiate $K_X(t)$ with respect to $t$, `$r$’ times and then putting $t = 0$, we get the cumulant of $r^{th}$ order.
That is
$$K_r = \left[ \frac{d^r}{dt^r} K_X(t) \right]_{t=0}$$
Since if $r = 1$,
$$K_1 = \left[ \frac{d}{dt} K_X(t) \right]_{t=0}$$
$$= \left[ \frac{d}{dt} \log M_X(t) \right]_{t=0}$$
$$= \left[ \frac{1}{M_X(t)} M_X'(t) \right]_{t=0} = \frac{\mu_1′}{1} \quad \text{Since } M_X(0) = E(e^0) = 1$$
$$\therefore \quad K_1 = \mu_1′ = E(X), \text{ Mean of } X$$
Similarly if $r = 2$,
$$K_2 = \left[ \frac{d^2}{dt^2} K_X(t) \right]_{t=0}$$
$$= \left[ \frac{M_X(t)M_X”(t) – (M_X'(t))^2}{[M_X(t)]^2} \right]_{t=0}$$
$$= \frac{\mu_2 – (\mu_1′)^2}{1}$$
$$= V(X)$$
This idea can be extended to a general `$r$’. That is
$$K_r = \left[ \frac{d^r}{dt^r} K_X(t) \right]_{t=0}$$
Example 3
Find the $r^{th}$ cumulant of the random variable $X$ if its pdf is
$$f(x) = \begin{cases} Ce^{-Cx} & C > 0 \text{ and } x > 0 \\ 0 & \text{elsewhere} \end{cases} $$
$$M_X(t) = E(e^{tx})$$
$$= \int_0^{\infty} Ce^{-ex} e^{tx} \, dx$$
$$= C \int_0^{\infty} e^{-(C-t)x} \, dx$$
$$= C \left[ \frac{e^{-(C-t)x}}{-(C-t)} \right]_0^{\infty}$$
$$= \frac{C}{C-t}$$
$$K_X(t) = \log M_X(t)$$
$$= \log\left( \frac{C}{C-t} \right) = \log C – \log(C-t)$$
$$K_X'(t) = \frac{1}{C-t}; \quad K_X”(t) = \frac{1}{(C-t)^2};$$
$$K_X^{\prime\prime\prime}(t) = \frac{2}{(C-t)^3}, \quad K_X^{(iv)}(t) = \frac{3!}{(C-t)^4} \quad \text{(and so on)}$$
$$K_X^{(r)}(t) = \frac{(r-1)!}{(C-t)^r}$$
Now $K^{th}$ cumulant $K_r = \left[ \frac{d^r}{dt^r} k_X(t) \right]_{t=0}$
$$= \frac{(r-1)!}{C^r}$$
CHARACTERISTIC FUNCTION (CF)
Definition
$$\varphi_X(t) = E[e^{itX}]$$
where i is the imaginary unit. For a discrete random variable:
$$\varphi_X(t) = \sum_x e^{itx}\,p(x)$$
For a continuous random variable:
$$\varphi_X(t) = \int_{-\infty}^{\infty} e^{itx}\,f(x)\,dx$$
Why It Always Exists
By Euler’s formula, $e^{itx} = \cos(tx) + i\sin(tx)$, so its modulus is exactly $1$ for every real $x$ and every real $t$.
$$|\varphi_X(t)| = \big|E[e^{itX}]\big| \leq E\big[|e^{itX}|\big] = E[1] = 1$$
The defining expectation is bounded by $1$ in absolute value no matter what distribution $X$ has, so it is always finite. Unlike the MGF, there is no restriction to a neighborhood around $0$ — the CF exists everywhere, for every random variable.
Key Properties
Moments are recovered through differentiation, with factors of $i$ appearing:
$$E[X^n] = \frac{1}{i^n}\varphi_X^{(n)}(0)$$
Under independence, the CF of a sum is the product of the individual CFs, exactly parallel to the MGF:
$$\varphi_{X+Y}(t) = \varphi_X(t)\,\varphi_Y(t)$$
Taking logs shows the log-CF adds across independent sums, exactly like the CGF. And the CF uniquely determines the distribution unconditionally, with no existence caveat attached — a stronger uniqueness statement than the MGF’s version.
Example 4: Discrete Random Variable
Let $X$ be a random variable with range $\mathscr{A}_x=\{0, 1, 2\}$ with probabilities $p_0, p_1, p_2$, where $p_0 + p_1 + p_2 = 1$.
$$\varphi_X(t) = p_0\, e^{it\cdot 0} + p_1\, e^{it\cdot 1} + p_2\, e^{it\cdot 2}$$
$$\varphi_X(t) = p_0 + p_1 e^{it} + p_2 e^{2it}$$
This is finite for every real $t$ automatically, since each exponential term has modulus exactly $1$ — no convergence condition needs to be checked at all. Compare this to the MGF version $p_0 + p_1 e^{t} + p_2 e^{2t}$, which happens to be fine here only because there are finitely many terms; with infinitely many terms in the sum an MGF could diverge where a CF could not.
Example 5: Continuous Random Variable
Let $X$ be a random variable have the PDF
$$f(x) = 2x, \qquad 0 < x < 1$$ start from the definition.
$$\varphi_X(t) = \int_0^1 e^{itx}(2x)\,dx$$
The integration by parts,
$$\varphi_X(t) = \frac{2e^{it}}{it} + \frac{2(e^{it}-1)}{t^2}$$
which implies,
$$|\varphi_X(t)| \leq \int_0^1 |e^{itx}|(2x)\,dx = \int_0^1 2x\,dx = 1$$
So convergence is guaranteed for every real $t$, matching the general argument and this holds no matter which valid PDF is used.
Remark — Existence of CF
Consider a continuous random variable with a heavy-tailed density that decays only like $1/x^2$ for large $|x|$:
$$f(x) = \frac{1}{\pi(1+x^2)}, \qquad -\infty < x < \infty$$
Check the MGF first.
$$M_X(t) = \int_{-\infty}^{\infty} \frac{e^{tx}}{\pi(1+x^2)}\,dx$$
For any $t \neq 0$, $e^{tx}$ grows exponentially as x moves toward either positive or negative infinity (depending on the sign of t), while the denominator only grows like $x^2$. The integrand never decays, so the integral diverges. The MGF fails to exist for every $t \neq 0$.
Now check the CF.
$$\varphi_X(t) = \int_{-\infty}^{\infty} \frac{e^{itx}}{\pi(1+x^2)}\,dx$$
Here $|e^{itx}| = 1$ regardless of $X$, so the integrand is bounded by $1/(\pi(1+x^2))$, which integrates to $1$ over the whole real line.
$$|\varphi_X(t)| \leq \int_{-\infty}^{\infty} \frac{1}{\pi(1+x^2)}\,dx = 1$$
The integral converges for every real t (its exact value works out to $e^{-|t|}$).
SUMMARY
- MGF: $M_X(t)=E[e^{tX}]$. Differentiating at $t=0$ yields raw moments. Multiplies across sums of independent variables. Can fail to exist for heavy-tailed distributions.
- CGF: $K_X(t)=\ln M_X(t)$. Differentiating at $t=0$ yields cumulants. Adds across sums of independent variables. Inherits the MGF’s existence problems, since it is built directly from the MGF.
- CF: $\varphi_X(t)=E[e^{itX}]$. Differentiating at $t=0$ yields moments (with factors of $i$). Multiplies across sums of independent variables, and its logarithm adds just like the CGF. Always exists for every random variable and every real $t$, because $|e^{itX}|=1$.