Introduction
prerequisite:
Elementary Function, Jacobian, Matrix
Addition:
Addition of two numbers in a real line is Translation
$$ x \longmapsto x + a $$
Moves a number on the real line (left or right).
$$ 5 \longmapsto 5 + 3 = 8 \quad \text{or} \quad 5 – 3 = 2 $$
Right or Left but “volume” = length is $ |a| = 3 $ units.
Multiplication:
Extending this to multiplication of two numbers,
$ x \to ax \quad $ “scales” a number with $ a > 0 $ same direction, $ a < 0 $ flips and $ a = 0 $, the magnitude and direction is zero
So, the length (or box) in $ \mathbb{R}^1 $ is scaled by $ |a| $
(i.e.) $L_{New} = |a|\, L_{Old}$. Sign of $ \det a$ = orientation flip.
$4 \times 5$ or $-4 \times 5$.
$L_{old} = 5$, $L_{new} = 4$ units
$4 \times 5 = 20$ where as $-4 \times 5 = -20$ is flipped by 4 units.
‘Addition is strictly shift no stretching’
So, in $n$-D, addition just moves the entire space. $V \longmapsto V + u$
Whereas, multiplication in $n$-D is matrix multiplication
$$x \longrightarrow Ax$$
Volume scaling is
$$V_{new} = |A|\, V_{old} \qquad$$
Sign of $\det A $: orientation flip.
$\therefore$ Multiplication of matrices “scales” volume in any dimension, Length, Area, Volume, hypervolume etc. In general, it is “Volume”
Invariance of Volume
Consider $y = g(x)$, $x \in \mathbb{R}$.
$$\therefore \quad \dfrac{dy}{dx} = g'(x).$$
or $dy \simeq g'(x)\,dx$
Area under infinitesimal ‘area’ is $\int f(x)\,dx$
(i.e) $f(x)$ is height, $dx$ = small length
$\therefore$ Area = Length $\times$ height
$= f(x)\,dx$.
Now Box-Counting geometry implies that
$$f(x)\,dx = p(y)\,dy$$
(density in $x$) $\times$ (width in $x$) = (den $\times$ wid)$y$
$$\therefore \quad p(y) = f(x)\left|\dfrac{dx}{dy}\right|$$
Further, if $g'(x) < 0$, the interval flips orientation.
Hence $$|f(x)\,dx| = |p(y)\,dy|$$
Box Concept
A tiny box in $\mathbb{R}^1$ is $(x,\ x + \Delta x)$
$\therefore$ Length (or “volume”) is $dV_x = \Delta x$
Height $f(x)$ so that area of the “box” is
$$dA_x = f(x)\,dx$$
After transformation, $(y, y + \Delta y)$, $y = g(x)$
$$(x, x+\Delta x) \longrightarrow (y, y+\Delta y)$$
$dV_y = \Delta y$,
Since, in tiny box, the change is linear
$\therefore$ ratio of length is $\dfrac{dy}{dx} \simeq g'(x)$ (slope)
$\Rightarrow dy \simeq g'(x)\,dx$$
$$dA_y = f(g^{-1}(y))\,dy$$
Area Preservation $\Rightarrow$
$$|f(x)\,dx| = |f(g^{-1}(y))\,dy|.$$
In Higher Dimensions,
Area in $x_1 x_2 \cdots x_n$ = Area in $u_1 u_2 \cdots u_n$ $\left|\det\left(\dfrac{\partial x_1}{\partial u_1} \cdots \dfrac{\partial x_n}{\partial u_n}\right)\right|$
Here the determinant is called Jacobian of the transformation $u_i= f_i(x_i)$
$= n \times n$ matrix of $1^{st}$ order partial derivatives
$$J = \begin{vmatrix} \dfrac{\partial x_1}{\partial u_1} & \dfrac{\partial x_1}{\partial u_2} & \cdots & \dfrac{\partial x_1}{\partial u_n} \\ \vdots & \vdots & \ddots & \vdots \\ \dfrac{\partial x_n}{\partial u_1} & \dfrac{\partial x_n}{\partial u_2} & \cdots & \dfrac{\partial x_n}{\partial u_n} \end{vmatrix}$$
Example: Box Condition
Suggested Reading: [DL] Deep Learning, Ian Goodfellow, Yoshua Bengio, Aaron Courville — Chapter 3: Probability and Information Theory (pp. ~70)
Consider a function:
$$f_X(x) = \begin{cases} 1 & 0 < x < 1 \\ 0 & \text{elsewhere} \end{cases}$$
Let $y = \dfrac{x}{2} = g(x)$ $\quad \therefore \dfrac{dy}{dx} = \dfrac{1}{2}$
or $x = g^{-1}(y) = 2y$
Without checking the box condition (“Volume” Invariance).
$\therefore \quad f_Y(g^{-1}(y)) = \begin{cases} 1 & 0 < 2y < 1 \\ 0 & \text{elsewhere} \end{cases}$
$$= \begin{cases} 1 & 0 < y < \dfrac{1}{2} \\ 0 & \text{elsewhere} \end{cases}$$
$$|f_Y(y)\,dy| = |f_X(x)\,dx|.$$
$$f_Y(y) = f_X(x)\left|\dfrac{dx}{dy}\right|$$
$$= f_X(g^{-1}(y))\left|\dfrac{dx}{dy}\right| = f_X(g^{-1}(y))\,|J|$$
$$= \begin{cases} 2 & 0 < y < \dfrac{1}{2} \\ 0 & \text{elsewhere} \end{cases}$$
In the above example, if $f_X(x)$ is the PDF of a uniform random variable in (0,1) then without the Jacobian, $f_Y(y)$ is not a valid PDF. On the otherhand, if the ‘box’ is properly measured using Jacobian, Then the resultant $f_Y(y)$ is a valid PDF.
That is, without Jacobian
$$\int_0^{1/2} f_Y(y)\,dy \ne 1$$
with Jacobian
$$\int_0^{1/2} f_Y(y)\,dy = 1$$
SUMMARY:
Multiplication
$\qquad \downarrow$
scaled Transformation
$\qquad \downarrow$
Linear map
$\qquad \downarrow$
Derivative
$\qquad \downarrow$
Jacobian
$\qquad \downarrow$
Determinant.
$f_1: \mathbb{R} \to \mathbb{R}$ s.t. $f(x) = ax$
– scales the real line by a factor $a$
– Interval length scales by $|a|$
– $a < 0$, flips orientation.
$f_n: \mathbb{R}^n \to \mathbb{R}^n$ s.t. $f(X) = AX$,
$X = (x_1 \cdots x_n) \in \mathbb{R}^n$
$A$ can be a $n \times n$ matrix (or $m \times n$)
– Generalizes scaling to higher dimension
– Volume (Box) scaling $V_{New} = |A|\, V_{Old}$
– Orientation flip if $\det(A) < 0$
– $f_n$ generalizes $f_1$