Derivation Using Taylor series:-
1-D
Consider $f: R \to R$ and let us expand around a point $x_0$ (Quadratic)
$$f(x) \approx f(x_0) + f'(x_0)(x – x_0) + f”(x_0)\dfrac{(x-x_0)^2}{2}$$
$$\Rightarrow \quad f'(x) = f'(x_0) + f”(x_0)(x – x_0) \qquad \text{upto} ~ 1^{st} \text{order}$$
$$\therefore \quad f'(x) = 0$$
$$\Rightarrow \quad f'(x_0) = -f”(x_0)(x – x_0)$$
$$\Rightarrow \quad f”(x_0)\,x = f”(x_0)\,x_0 – f'(x_0)$$
$$\Rightarrow \quad x = x_0 – \dfrac{f'(x_0)}{f”(x_0)}$$
GD: $x = x_0 – \alpha f'(x_0)$ where, $\alpha \simeq \dfrac{1}{f”(x_0)}$ a rough scale for curvature
2-D
Let $$X = \begin{bmatrix} x_1 \\ x_2 \end{bmatrix} \qquad f: R^2 \to R $$
$$X_0 = \begin{bmatrix} x_1^0 \\ x_2^0 \end{bmatrix} \qquad \Delta X = X – X_0 = \begin{bmatrix} x_1 – x_1^0 \\ x_2 – x_2^0 \end{bmatrix}$$
$$\therefore \quad \nabla f = \begin{bmatrix} \dfrac{\partial f}{\partial x_1} \\ \dfrac{\partial f}{\partial x_2} \end{bmatrix} \qquad H = \begin{bmatrix} \dfrac{\partial^2 f}{\partial x_1^2} & \dfrac{\partial^2 f}{\partial x_1 \partial x_2} \\[2mm] \dfrac{\partial^2 f}{\partial x_1 \partial x_2} & \dfrac{\partial^2 f}{\partial x_2^2} \end{bmatrix}$$
$$f(X) \simeq f(X_0) + (\nabla f)_{X_0}(X – X_0) + \dfrac{1}{2}(\Delta X)^T H^0(\Delta X)$$
where $H^0: (H)_{x_0}$
Results on Matrix Differentiation:
1.constant $a = \begin{bmatrix} a_1 \\ \vdots \\ a_n \end{bmatrix}$, vector of variables $x = \begin{bmatrix} x_1 \\ x_2 \\ \vdots \\ x_n \end{bmatrix}$
$$a^TX = [a_1 \cdots a_n]\begin{bmatrix} x_1 \\ x_2 \\ \vdots \\ x_n \end{bmatrix} = a_1x_1 + a_2x_2 + \cdots + a_nx_n$$
$$\therefore \quad \dfrac{\partial}{\partial x_i}(a^TX) = a_i$$
(or) $$\dfrac{\partial}{\partial X}(a^TX) = \begin{bmatrix} a_1 \\ a_2 \\ \vdots \\ a_n \end{bmatrix} = a$$
$$\dfrac{\partial(a^TX)}{\partial X} = a$$
2.Consider $A_{m \times n} = [a_{ij}]_{m \times n}$
Then $$AX = [f_i(X)]_{m \times n} \qquad f_i(X) = \sum_{j=1}^{n} a_{ij}x_j \qquad i = 1, 2, \cdots m$$
$$\therefore \quad \dfrac{\partial f_i(X)}{\partial x_j} = a_{ij} \quad \text{where } f_i(X) = f_i(x_1 \cdots x_n) \qquad f: R^n \to R^m$$
$\therefore$ Jacobian of derivative
$$\dfrac{d(AX)}{dX} = \begin{bmatrix} \dfrac{\partial f_1}{\partial x_1} & \dfrac{\partial f_1}{\partial x_2} & \cdots & \dfrac{\partial f_1}{\partial x_n} \\ \vdots & \vdots & \ddots & \vdots \\ \dfrac{\partial f_m}{\partial x_1} & \dfrac{\partial f_m}{\partial x_2} & \cdots & \dfrac{\partial f_m}{\partial x_n} \end{bmatrix}_{m \times n} = A$$
$$\therefore \dfrac{d(AX)}{dX} = A$$
3.Consider quadratic form $X^TAX$
$$X^TAX = \begin{bmatrix} x_1 \cdots x_n \end{bmatrix} \begin{bmatrix} a_{11} & a_{12} & \cdots & a_{1n} \\ a_{21} & a_{22} & \cdots & a_{2n} \\ \vdots & \vdots & \ddots & \vdots \\ a_{n1} & a_{n2} & \cdots & a_{nn} \end{bmatrix}\begin{bmatrix} x_1 \\ x_2 \\ \vdots \\ x_n \end{bmatrix}$$
$$= \begin{bmatrix} x_1 \cdots x_n \end{bmatrix}_{1\times n}\begin{bmatrix} \sum_{j=1}^{n} a_{1j}x_j \\ \sum_{j=1}^{n} a_{2j}x_j \\ \vdots \\ \sum_{j=1}^{n} a_{nj}x_j \end{bmatrix}_{n \times 1}$$
$$= x_1\sum_{j=1}^{n} a_{1j}x_j + x_2\sum_{j=1}^{n} a_{2j}x_j + \cdots + x_n\sum_{j=1}^{n} a_{nj}x_j$$
$$X^TAX = \sum_{i,j} x_i\,a_{ij}\,x_j$$
Now $$\dfrac{\partial}{\partial x_1}[X^TAX] = 2a_{11}x_1 + a_{12}x_2 + \cdots + a_{1n}x_n$$
$$+ a_{21}x_2 + a_{31}x_3 + \cdots + a_{n1}x_n$$
$$= 2a_{11}x_1 + (a_{12}+a_{21})x_2 + (a_{13}+a_{31})x_3 + \cdots + (a_{1n}+a_{n1})x_n$$
$$\dfrac{\partial}{\partial x_i}[X^TAX] = 2a_{ii}x_1 +(a_{i2}+a_{2i})x_2 + \cdots + (a_{in}+a_{ni})x_n$$
$$\dfrac{\partial}{\partial X}[X^TAX] = (A + A^T)X$$
If $A$ is symmetric then
$$\dfrac{\partial}{\partial X}(X^TAX) = 2AX$$
or $$\dfrac{\partial}{\partial X}\left(\dfrac{1}{2}X^TAX\right) = AX.$$
NOTE: Shifted quadratic form.
Instead of $X$, consider $X – X_0$
$$\dfrac{\partial}{\partial X}\left[\dfrac{1}{2}(X-X_0)^TA(X-X_0)\right] = A(X-X_0)$$