Introduction
We now prove several useful properties of the expectation of a random variable $X$, which is applied in univatiate & Bivariate Distributions. These properties help in determining the expectations of functions of a random variable(s) from known or easily computed expectations.
The derivations are presented for the continuous case. The discrete case follows in a similar manner, with summations replacing integrals where appropriate.
Property 1:
If $a$ is a constant then $E(aX) = aE(X)$. In particular $E(a) = a$.
$\textit{Proof:}$
$$E(aX) = \int_{-\infty}^{\infty} ax f_{X}(x) \, dx$$
$$= a \int_{-\infty}^{\infty} x f_{X}(x) \, dx = aE(X)$$
Also, $$E(a) = \int_{-\infty}^{\infty} a f_{X}(x) \, dx = a \int_{-\infty}^{\infty} f_{X}(x) dx = a(1) \quad (\because f_{X}(x) \text{ is the pdf of } X)$$
$$E(a) = a$$
Property 2:
If $a$ and $b$ are constants, then $E(aX + b) = aE(X) + b$.
$\textit{Proof:}$
$$E(aX + b) = \int_{-\infty}^{\infty} (ax + b) f_{X}(x) \, dx$$
$$= \int_{-\infty}^{\infty} [ax f_{X}(x) \, + bf_{X}(x)] \, dx$$
$$= a \int_{-\infty}^{\infty} xf_{X}(x) \, dx + b \int_{-\infty}^{\infty} f_{X}(x) \, dx$$
$$= aE(X) + b.$$
Property 3:
If $C_1, C_2, \cdots, C_n$ are constants and $g_1(x), g_2(x), \cdots, g_n(x)$ are any functions of $X$, then
$$E\left[ \sum_{i=1}^{n} C_i g_i(X) \right] = \sum_{i=1}^{n} C_i E[g_i(X)]$$
$\textit{Proof:}$
LHS:$$E\left[ \sum_{i=1}^{n} C_i g_i(X) \right] = E[C_1 g_1(X) + C_2 g_2(X) + \cdots + C_n g_n(X)]$$
$$= \int_{-\infty}^{\infty} [C_1 g_1(x) + C_2 g_2(x) + \cdots + C_n g_n(x)] f_{X}(x) \, dx$$
$$= C_1 \int_{-\infty}^{\infty} g_1(x) f_{X}(x) \, dx + C_2 \int_{-\infty}^{\infty} g_2(x) f_{X}(x) \, dx + \cdots +$$
$$C_n \int_{-\infty}^{\infty} g_n(x) f_{X}(x) \, dx$$
$$= \sum_{i=1}^{n} C_i E[g_i(X)]$$
Property 4:
If $X$ and $Y$ are random variables then $E(X+Y) = E(X) + E(Y)$.
$\textit{Proof:}$
Let $f_{XY}(x, y)$ be the joint pdf of $X$ and $Y$ and $f_X(x)$ and $f_Y(y)$ be their respective marginal pdfs. Then by definition,
$$E(X+Y) = \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} (x+y) f_{XY}(x, y) \, dx dy$$
$$= \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} x f_{XY}(x, y) \, dx dy + \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} y f_{XY}(x, y) \, dx dy$$
$$= \int_{-\infty}^{\infty} x \left[ \int_{-\infty}^{\infty} f_{XY}(x, y) \, dy \right] dx + \int_{-\infty}^{\infty} y \left[ \int_{-\infty}^{\infty} f_{XY}(x, y) \, dx \right] dy$$
$$= \int_{-\infty}^{\infty} x f_X(x) \, dx + \int_{-\infty}^{\infty} y f_Y(y) \, dy$$
$$= E(X) + E(Y).$$
Note:
Property 4 can be extended to $n$ variables as
$$E(X_1 + X_2 + \cdots + X_n) = E(X_1) + E(X_2) + \cdots + E(X_n)$$
Property 5:
If $X$ and $Y$ are independent random variables, then $E(XY) = E(X) \cdot E(Y)$.
$\textit{Proof:}$
Let $X$ and $Y$ be two random variables with joint pdf $f_{XY}(x, y)$ and the respective marginal pdfs be $f_X(x)$ and $f_Y(y)$.
$$E(XY) = \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} xy f_{XY}(x, y) \, dx dy$$
$$= \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} xy f_X(x) \cdot f_Y(y) \, dx dy$$
$$= \int_{-\infty}^{\infty} x f_X(x) \, dx \cdot \int_{-\infty}^{\infty} y f_Y(y) \, dy$$
$$= E(X) \cdot E(Y)$$
Note: Property 5 can be extended to $n$ independent variables as
$$E(X_1, X_2, \cdots, X_n) = E(X_1) \cdot E(X_2) \cdots E(X_n)$$
Property 6: Telescope formula.
For a random variable $X$ taking its values in natural number system that is $\{1, 2, 3 \cdots\}$ then
$$E(X) = \sum_{n=1}^{\infty} P(X \geq n)$$
$\textit{Proof:}$
By definition, since $X$ is a discrete random variable,
$$E(X) = \sum_{x=1}^{\infty} xf_{X}(x)$$
$$= 1f(1) + 2f(2) + 3f(3) + \cdots$$
$$= f(1) + f(2) + f(3) + f(4)$$
$$+ f(2) + f(3) + f(4)$$
$$+ f(3) + f(4)$$
$$+ f(4) + \cdots \quad \text{adding row wise,}$$
$$= p(X \geq 1) + p(X \geq 2) + p(X \geq 3) + p(X \geq 4) + \cdots$$
$$= \sum_{n=1}^{\infty} p(X \geq n).$$
These properties are useful in simplifying the calculation effort one has to do while calculating expectations. It should be noted that all such calculations (Refer Example 2) need not to have the application of one or more these properties.
The following is an extension of property 3 that is we generalise it to two variables $X$ and $Y$. If $C_1, C_2, \cdots, C_n$ are real constants and $g_1(X, Y); g_2(X, Y) \cdots g_n(X, Y)$ are any functions in $X$ and $Y$ then,
$$E\left[ \sum_{i=1}^{n} C_i g_i(X, Y) \right] = \sum_{i=1}^{n} C_i E[g_i(X, Y)]$$
Property 7:
$V(C) = 0$ ($C$ is any constant)
Now $$V(C) = E(C^2) – (E(C))^2$$
$$= C^2 – C^2 = 0.$$
Property 8:
$$V(aX+b) = a^2 V(X)$$
$\textit{Proof:}$
$$V(aX+b) = E((aX+b)^2) – (E(aX+b))^2$$
$$= E(a^2X^2 + b^2 + 2abX) – (aE(X) + E(b))^2$$
$$= [a^2E(X^2) + E(b^2) + 2abE(X)] – [a^2E(X)^2 +E(b)^2 + 2abE(X)]$$
$$= a^2[E(X^2) – (E(X))^2] = a^2V(X)$$
Property 9:
$\text{Cov}(X, Y) = E(XY) – E(X) \cdot E(Y)$
$\textit{Proof:}$
By the definition,
$$\text{Cov}(X, Y) = E[(X – E(X))(Y – E(Y))]$$
$$= E[XY – XE(X) – YE(X) + E(X)E(Y)]$$
$$= E(XY) – E(XE(Y)) – E(YE(X)) + E(E(X)E(Y))$$
$$= E(XY) – E(X) \cdot E(Y) – E(Y) \cdot E(X) + E(X)E(Y)$$
$$= E(XY) – E(X)E(Y)$$
Observe that in the above derivation we use Property 1 and extension of Property 4.
We can deduce an interesting aspect of relationship between two variables combining Property 5 and Property 9. That is if $X$ and $Y$ are independent then, $E(XY) = E(X) \cdot E(Y)$ so that
$$E(XY) – E(X) \cdot E(Y) = 0 \quad \text{or} \quad \text{Cov}(X, Y) = 0$$
$\fbox{Independence of two random variables implies zero covariance.}$
But a caution! zero covariance does not necessarily imply independence. Let us find this with the help of the following example.
Example 1
Let the joint pdf of a random variable be
$$\begin{array}{c|ccc} X \backslash Y & -1 & 0 & 1 \\ \hline 0 & 0 & \frac{1}{6} & \frac{1}{12} \\ 1 & \frac{1}{4} & 0 & \frac{1}{2} \end{array} $$
Find $\text{Cov}(X, Y)$.
The marginal pdfs of $X$ and $Y$ are
$$ \begin{array}{c|cc} X = x & 0 & 1 \\ \hline f_X(x) & \frac{1}{4} & \frac{3}{4} \end{array} \qquad \begin{array}{c|ccc} Y = y & -1 & 0 & 1 \\ \hline
f_Y(y) & \frac{1}{4} & \frac{1}{6} & \frac{7}{12} \end{array}$$
$$ E(XY) = \sum \sum xy f_{XY}(x, y) = (0)(-1)(0) + (0)(0)\frac{1}{6} + (0)(1)\frac{1}{12} +$$
$$(1)(-1)\frac{1}{4} + (1)(0)(0) + (1)(1)\frac{1}{2} = \frac{1}{4}$$
$$E(X) = \sum_{x} x f_X(x) = \frac{3}{4}; \quad E(Y) = \sum y f_Y(y) = \frac{1}{3}$$
$$\therefore \quad \text{Cov}(X, Y) = E(XY) – E(X)E(Y) = \frac{1}{4} – \left(\frac{3}{4}\right)\left(\frac{1}{3}\right) = 0 $$
But $X$ and $Y$ are not independent since $f_{XY}(0, -1) = 0$ where as $f_X(0) = \frac{1}{4}$ and $f_Y(-1) = \frac{1}{4}$ hence $f_{XY}(0,-1) \neq f_X(0) \cdot f_Y(-1)$.
However if $\text{Cov}(X, Y) = 0$ it leads to a special cases of distributions that is uncorrelated variables. Before go further, let us define the correlation coefficient as
$$\rho_{XY} = \frac{\text{Cov}(X, Y)}{\sigma_X \cdot \sigma_Y} $$
where $\sigma_X$ and $\sigma_Y$ are the standard deviation of $X$ and $Y$ respectively.
Hence, now we can say that if the variables are correlated in some way, the covariance will be non-zero. Infact, if $\text{Cov}(X, Y) > 0$, then $Y$ tends to increase (decrease) as $X$ increases (decreases). If $\text{Cov}(X, Y) < 0$, then $Y$ tends to increase (decreases) as $X$ decreases (increases). observe a fact that $|\rho_{XY}| \leq 1$ . Also the sign of $\rho_{XY}$ is that of $\text{Cov}(X, Y)$ under the assumption that $\sigma_X$, are $\sigma_Y$ are positive.
Moments of Linear Combination of Random Variables
Property 10:
If $X_1$ and $X_2$ are two random variables and $a_1$ and $a_2$ are constants, then $E(a_1X_1 + a_2X_2) = a_1E(X_1) + a_2E(X_2)$.
$\textit{Proof:}$
Let us prove for continuous random variable (proof is similar to discrete case). Let $f_{X_1X_2}(x_1, x_2)$ be the joint pdf of $X_1$ and $X_2$ and $f_{X_1}(x_1)$ and $f_{X_2}(x_2)$ be their respective marginal pdfs.
Now$$ E(a_1X_1 + a_2X_2) = \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} (a_1x_1 + a_2x_2) f_{X_1X_2}(x_1, x_2) \, dx_1 dx_2 $$
$$ = \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} a_1x_1 f_{X_1X_2}(x_1, x_2) \, dx_1 dx_2 + \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} a_2x_2 f_{X_1X_2}(x_1x_2) \, dx_1 dx_2 $$
$$ = a_1 \int_{-\infty}^{\infty} x_1 \left[ \int_{-\infty}^{\infty} f_{X_1X_2}(x_1x_2) \, dx_2 \right] dx_1 + a_2 \int_{-\infty}^{\infty} x_2 \left[ \int_{-\infty}^{\infty} f_{X_1X_2}(x_1x_2) \, dx_1 \right] dx_2$$
$$ = a_1 \int_{-\infty}^{\infty} x_1 f_{X_1}(x_1) \, dx_1 + a_2 \int_{-\infty}^{\infty} x_2 f_{X_2}(x_2) \, dx_2 $$
$$ = a_1 E(X_1) + a_2 E(X_2) $$
This property can be generalised to $n$ random variables as
$$ E\left[ \sum_{i=1}^{n} a_i X_i \right] = \sum_{i=1}^{n} a_i E(X_i) $$
Also taking $a_1 = a_2 = 1$, this reduces to Property 4.
Property 11:
Let $X_1, X_2$ be random variables and $a_1, a_2$ be constants. Then,
$$V(a_1X_1 + a_2X_2) = a_1^2 V(X_1) + a_2^2 V(X_2) + 2ab \, \text{Cov}(X_1, X_2)$$
$\textit{Proof:}$
Let $Y = a_1X_1 + a_2X_2$
$$V(Y) = E(Y^2) – E(Y)^2$$
$$= E[(a_1X_1 + a_2X_2)^2] – [E(a_1X_1 + a_2X_2)]^2$$
$$= E[a_1^2X_1^2 + a_2^2X_2^2 + 2a_1a_2X_1X_2] – [a_1E(X_1) + a_2E(X_2)]^2$$
$$= [a_1^2E(X_1^2) + a_2^2E(X_2^2) + 2a_1a_2E(X_1X_2)]$$
$$- [a_1^2E(X_1)^2 + a_2^2E(X_2)^2 + 2a_1a_2E(X_1)E(X_2)]$$
$$= a_1^2[E(X_1^2) – E(X_1)^2] + a_2^2[E(X_2^2) – E(X_2)^2]$$
$$+ 2a_1a_2[E(X_1X_2) – E(X_1)E(X_2)]$$
$$= a_1^2 V(X_1) + a_2^2 V(X_2) + 2ab \, \text{Cov}(X_1, X_2)$$
$\textbf{Corollary:}$
If the random variables $X_1, X_2$ are independent then Property 11 becomes
$$V(a_1X_1 + a_2X_2) = a_1^2V(X_1) + a_2^2V(X_2)$$
Since $X_1, X_2$ are independent.
$\text{Cov}(X_1X_2) = 0$ and hence Property 11 becomes,
$$V(a_1X_1 + a_2X_2) = a_1^2V(X_1) + a_2^2V(X_2)$$
In particular if we take $a_2 = 0$, then the property is exactly what we derived in Property 8. The generalisation of Property 11 is as follows. If $X_1, X_2, \cdots, X_n$ are random variables, $a_1, a_2 \cdots a_n$ are constants then
$$V\left( \sum_{i=1}^{n} a_i X_i \right) = a_i^2 V(X_i) + 2 \sum_{1 \leq i < j \leq n} a_i a_j \, \text{Cov}(X_i, X_j)$$
Particularly, if $X_1, X_2, \cdots, X_n$ are independent then
$$V\left( \sum_{i=1}^{n} a_i X_i \right) = \sum_{i=1}^{n} a_i^2 V(X_i)$$
Property 12:
If $X_1, X_2$ are random variables $a_1, a_2, b_1, b_2$ are constants then
$$\text{Cov}(a_1X_1 + a_2X_2, b_1X_1 + b_2X_2)$$
$$= a_1b_1V(X_1) + a_2b_2V(X_2) + (a_1b_2 + a_2b_1)\text{Cov}(X_1, X_2)$$
$\textit{Proof:}$
Let $u = a_1X_1 + a_2X_2$ and $v = b_1X_1 + b_2X_2$
$\therefore \text{Cov}(u, v) = E[uv] – E(u)E(v) \tag{1}$
$$E[uv] = E[(a_1X_1 + a_2X_2)(b_1X_1 + b_2X_2)]$$
$$= E[a_1b_1X_1^2 + a_1b_2X_1X_2 + a_2b_1X_2X_1 + a_2b_2X_2^2]$$
$$= a_1b_1E[X_1^2] + (a_1b_2 + a_2b_1)E(X_1X_2) + a_2b_2E(X_2^2) \tag{2}$$
$\,$
$$E[u]E[v] = E[(a_1X_1 + a_2X_2)] \cdot E[(b_1X_1 + b_2X_2)]$$
$$= (a_1E(X_1) + a_2E(X_2))(b_1E(X_1) + b_2E(X_2))$$
$$= a_1b_1E(X_1)^2 + a_1b_2E(X_1)E(X_2) + a_2b_1E(X_2)E(X_1)$$
$$+ a_2b_2(E(X_2))^2$$
$$= a_1b_1E(X_1)^2 + (a_1b_2 + a_2b_1)E(X_1)E(X_2) + a_2b_2E(X_2)^2 \tag{3}$$
Now if we substitute (2) and (3) in (1) and collect the coefficients respectively in each term of (2) and (3) we get
$$\text{Cov}(u, v) = a_1b_1[E(X_1^2) – E(X_1)^2] + (a_1a_2 + a_2b_1)$$
$$[E(X_1X_2) – E(X_1)E(X_2)] + a_2b_2[E(X_2^2) – E(X_2)^2]$$
$$= a_1b_1V(X_1) + (a_1b_2 + a_2b_1)\text{Cov}(X_1, X_2) + a_2b_2V(X_2)$$
$\textbf{Corollary:}$
If the random variables $X_1, X_2$ are independent, then
$$\text{Cov}(a_1X_1 + a_2X_2, b_1X_1 + b_2X_2) = a_1b_1V(X_1) + a_2b_2V(X_2)$$
Since $\text{Cov}(X_1 + X_2) = 0$, proof is immediate from Property 12.
We can generalise Property 12 and its corollary for $n$ random variables as follows. If $X_1, X_2, \cdots, X_n$ are random variables; $a_1, a_2, \cdots, a_n$ and $b_1, b_2, \cdots b_n$ are constants, we define
$$u = \sum_{i=1}^{n} a_i X_i, \quad v = \sum_{i=1}^{n} b_i X_i \quad \text{then}$$
$$\text{Cov}(u, v) = \sum_{i=1}^{n} a_i b_i V(X_i) + \sum_{1 \leq i < j \leq n} (a_i b_j + a_j b_i)\text{Cov}(X_i, X_j)$$
In particular, if $X_1, X_2, \cdots, X_n$ are independent random variables, then
$$\text{Cov}(u, v) = \sum_{i=1}^{n} a_i b_i v(X_i)$$
Eventhough our discussion is restricted to two variables, we may try to apply the of these results generalised versions also so that we can extend these ideas more naturally.