f_bowl <- function(x, y) x^2 + y^2
f_bowl(3, 4)[1] 2512 Multivariable Calculus
Every model has more than one parameter. A loss function takes a whole vector and returns a single number, and minimizing it means knowing how that number responds to each component — and to all of them at once.
This chapter extends Chapter 11 from one input to many. The definitions barely change. What changes is that “the derivative” is no longer a number but a vector, and its direction carries as much information as its size.
12.1 Functions of several variables
A function of two variables takes a point in the plane and returns a number:
\[ f: \mathbb{R}^2 \to \mathbb{R}, \qquad f(x, y) = x^2 + y^2 \]
More generally \(f: \mathbb{R}^n \to \mathbb{R}\) takes a vector and returns a scalar. Such a function is sometimes called a scalar field, and it is the shape of every loss function you will meet: many parameters in, one number out.
Drawing one is the difficulty. \(f(x,y)\) needs three dimensions to plot directly, and a perspective drawing of a surface is hard to read values off. This book uses contours instead, which are precise, and which generalize to dimensions you cannot draw at all.
12.2 Level sets and contours
A level set is the set of inputs sharing one output value:
\[ \{\,(x,y) : f(x,y) = c\,\} \]
On a map these are contour lines of constant elevation; here they are exactly the same idea. Reading a contour plot is a skill worth acquiring deliberately: close contours mean steep, wide contours mean flat.
The circles are round, not oval — which is only true because the figure is drawn with equal axis scaling. A stretched version of this plot would suggest the bowl is steeper in one direction than another, and it is not.
Two more surfaces do most of the work in this chapter and the optimization chapters.
NoteIn machine learning
Figure 12.2 is the single most useful picture in optimization. That long narrow valley is what a badly scaled or collinear problem looks like, and it is why gradient descent zig-zags: the gradient points across the valley rather than along it. The stretch factor is the ratio of the Hessian’s eigenvalues — the condition number from Section 6.10, seen as a shape.
12.3 Partial derivatives
To differentiate a function of several variables, vary one input and hold the rest fixed. That is a partial derivative, written with a curly \(\partial\):
\[ \frac{\partial f}{\partial x} = \lim_{h \to 0}\frac{f(x+h, y) - f(x, y)}{h} \tag{12.1}\]
Identical to Equation 11.1 except that \(y\) sits still. Every rule from Chapter 11 applies, treating the other variables as constants.
Worked example. For \(f(x,y) = x^2y + 3y\):
\[ \frac{\partial f}{\partial x} = 2xy \qquad \frac{\partial f}{\partial y} = x^2 + 3 \]
For \(\partial f/\partial x\), the \(3y\) term is constant and vanishes. For \(\partial f/\partial y\), the \(x^2\) in \(x^2y\) is a constant multiplier.
At the point \((2, 1)\): \(\partial f/\partial x = 4\) and \(\partial f/\partial y = 7\).
f2 <- function(x, y) x^2 * y + 3 * y
h <- 1e-6
c(
df_dx = (f2(2 + h, 1) - f2(2 - h, 1)) / (2 * h),
df_dy = (f2(2, 1 + h) - f2(2, 1 - h)) / (2 * h)
)df_dx df_dy
4 7 The notation \(\partial f/\partial x\) and \(f_x\) mean the same thing; the second is compact when there are many.
12.4 The gradient
Collect the partials into a vector and you get the gradient:
\[ \nabla f = \begin{bmatrix} \partial f/\partial x_1 \\ \vdots \\ \partial f/\partial x_n \end{bmatrix} \tag{12.2}\]
Read “\(\nabla\)” as “del” or “grad”. By the convention in Section 13.2 it is a column vector, the same shape as the input.
The gradient has two properties that make it the central object of optimization, and both are geometric.
1. It points in the direction of steepest increase. Of all directions you could move from a point, \(\nabla f\) is the one along which \(f\) grows fastest, and \(\|\nabla f\|\) is that rate.
2. It is perpendicular to the level set. Moving along a contour keeps \(f\) constant, so the direction of fastest change must be at right angles to it.
The arrows are scaled to a third of their true length so they fit the frame; their directions are exact, and the right angles are real because the figure is equally scaled.
Worked example. For \(f = x^2 + y^2\) at \((1,2)\):
\[ \nabla f = \begin{bmatrix}2x\\2y\end{bmatrix} = \begin{bmatrix}2\\4\end{bmatrix}, \qquad \|\nabla f\| = \sqrt{20} \approx 4.47 \]
norm
2.000000 4.000000 4.472136
NoteIn machine learning
Gradient descent is one line: \(\boldsymbol{\theta} \leftarrow \boldsymbol{\theta} - \eta\nabla L\). Since \(\nabla L\) points uphill, subtracting it goes downhill, and \(\eta\) sets how far. Everything in Chapter 16 is a refinement of that step.
12.5 Directional derivatives
Partial derivatives measure change along the axes. The directional derivative measures change along any unit vector \(\mathbf{u}\):
\[ D_{\mathbf{u}}f = \nabla f \cdot \mathbf{u} \tag{12.3}\]
A dot product — so everything from Section 4.12 applies. Writing it as \(\|\nabla f\|\|\mathbf{u}\|\cos\theta = \|\nabla f\|\cos\theta\) makes both gradient properties fall out immediately:
- Largest when \(\theta = 0\), i.e. \(\mathbf{u}\) along \(\nabla f\) — steepest ascent.
- Zero when \(\theta = 90°\), i.e. \(\mathbf{u}\) along the contour — perpendicular to the level set.
- Most negative when \(\theta = 180°\) — steepest descent, the direction gradient descent takes.
Worked example. With \(\nabla f = (2,4)\) at \((1,2)\):
along_x diagonal along_gradient
2.000000 4.242641 4.472136 The third equals \(\|\nabla f\| = 4.47\), the largest any direction can give.
WarningWatch out
Equation 12.3 requires \(\mathbf{u}\) to be a unit vector. Feed it an unnormalized direction and you scale the answer by that vector’s length, which silently inflates every rate you compute.
12.6 The multivariable chain rule
If \(z = f(x,y)\) and both \(x\) and \(y\) depend on \(t\), then
\[ \frac{dz}{dt} = \frac{\partial f}{\partial x}\frac{dx}{dt} + \frac{\partial f}{\partial y}\frac{dy}{dt} \tag{12.4}\]
Every path by which \(t\) influences \(z\) contributes a term, and the terms add. That is the whole rule, and it is what backpropagation implements when a quantity feeds several downstream layers.
Compactly, it is a dot product of the gradient with the velocity:
\[ \frac{dz}{dt} = \nabla f \cdot \frac{d\mathbf{x}}{dt} \]
Worked example. \(f = x^2 + y^2\) with \(x = t\), \(y = t^2\). Directly: \(z = t^2 + t^4\), so \(dz/dt = 2t + 4t^3\). By the chain rule: \(2x(1) + 2y(2t) = 2t + 4t^3\). Same.
12.7 The Jacobian
When the output is a vector too — \(\mathbf{f}: \mathbb{R}^n \to \mathbb{R}^m\) — all the partials form the Jacobian, an \(m \times n\) matrix:
\[ \mathbf{J}_{ij} = \frac{\partial f_i}{\partial x_j} \]
Row \(i\) is the gradient of output \(i\). A gradient is the case \(m = 1\); the Jacobian generalizes it, and Section 13.4 uses it to state the chain rule as a matrix product.
Worked example. For \(\mathbf{f}(x,y) = (x^2y,\; x + y^2)\):
\[ \mathbf{J} = \begin{bmatrix} 2xy & x^2 \\ 1 & 2y \end{bmatrix} \]
[,1] [,2]
[1,] 4 4
[2,] 1 2 [,1] [,2]
[1,] 4 4
[2,] 1 2The Jacobian’s determinant measures how the map scales area locally (Section 5.13), which is what makes it appear in the change of variables formula for integrals (Section 14.9).
12.8 The Hessian
Second derivatives. For \(f: \mathbb{R}^n \to \mathbb{R}\) the Hessian collects every second partial:
\[ \mathbf{H}_{ij} = \frac{\partial^2 f}{\partial x_i \partial x_j} \tag{12.5}\]
an \(n \times n\) matrix — the Jacobian of the gradient. It is the multivariable version of \(f''\), and it describes curvature.
The Hessian is symmetric whenever the second partials are continuous, because \(\partial^2 f/\partial x\partial y = \partial^2 f/\partial y\partial x\) — the order of differentiation does not matter. That is not a small detail: it means the spectral theorem applies (Section 8.7), so a Hessian always has real eigenvalues and orthogonal eigenvectors.
Worked example. For \(f = 10x^2 + y^2\):
\[ \mathbf{H} = \begin{bmatrix}20 & 0\\ 0 & 2\end{bmatrix} \]
Constant, because the function is quadratic. Its eigenvalues are 20 and 2 — the curvatures along the two axes, and their ratio of 10 is exactly the elongation visible in Figure 12.2.
hess <- function(f, p, h = 1e-4) {
n <- length(p)
H <- matrix(0, n, n)
for (i in seq_len(n)) {
for (j in seq_len(n)) {
ei <- ej <- rep(0, n)
ei[i] <- h
ej[j] <- h
H[i, j] <- (
f(p + ei + ej) - f(p + ei - ej) -
f(p - ei + ej) + f(p - ei - ej)
) / (4 * h^2)
}
}
H
}
fv <- function(p) f_valley(p[1], p[2])
round(hess(fv, c(1, 1)), 4) [,1] [,2]
[1,] 20 0
[2,] 0 2
NoteIn machine learning
The Hessian’s eigenvalues set how hard a problem is to optimize. Their ratio is the condition number, and gradient descent’s convergence rate degrades directly with it (Section 16.4). Newton’s method multiplies by \(\mathbf{H}^{-1}\) to undo the stretch — which is why it converges so much faster and why nobody can afford it for large models: the Hessian has \(p^2\) entries.
12.9 Second-order conditions for extrema
In one dimension, \(f'=0\) located a critical point and the sign of \(f''\) classified it (Section 11.8). The same two steps work here, with vectors and matrices in place of numbers.
First-order condition. At any local extremum, every partial vanishes:
\[ \nabla f = \mathbf{0} \]
Second-order condition. Classify by the Hessian’s definiteness (Section 8.9):
| Hessian at a critical point | Classification |
|---|---|
| positive definite (all \(\lambda_i > 0\)) | local minimum |
| negative definite (all \(\lambda_i < 0\)) | local maximum |
| indefinite (mixed signs) | saddle point |
| singular (some \(\lambda_i = 0\)) | inconclusive |
The saddle case has no one-dimensional analogue, and it is the interesting one: the point is a minimum along some directions and a maximum along others, which is exactly what Figure 12.3 shows.
Worked example. \(f(x,y) = x^3 - 3xy + y^3\).
\[ \nabla f = \begin{bmatrix}3x^2 - 3y \\ -3x + 3y^2\end{bmatrix} = \mathbf{0} \]
gives \(y = x^2\) and \(x = y^2\), so \(x = x^4\), whose real solutions are \(x = 0\) and \(x = 1\). The critical points are \((0,0)\) and \((1,1)\).
\[ \mathbf{H} = \begin{bmatrix}6x & -3 \\ -3 & 6y\end{bmatrix} \]
f_cubic <- function(x, y) x^3 - 3 * x * y + y^3
H_at <- function(p) {
matrix(
c(6 * p[1], -3, -3, 6 * p[2]),
nrow = 2, byrow = TRUE
)
}
for (p in list(c(0, 0), c(1, 1))) {
cat(
"point (", p[1], ",", p[2], ") f =", f_cubic(p[1], p[2]),
" eigenvalues:",
round(eigen(H_at(p), symmetric = TRUE)$values, 4), "\n"
)
}point ( 0 , 0 ) f = 0 eigenvalues: 3 -3
point ( 1 , 1 ) f = -1 eigenvalues: 9 3 At \((0,0)\) the eigenvalues are \(3\) and \(-3\): indefinite, a saddle. At \((1,1)\) they are \(9\) and \(3\): both positive, so positive definite — a local minimum, with value \(-1\). (eigen() sorts descending, which is why the larger comes first.)
Closed loops around a point mean an extremum; contours crossing themselves mean a saddle. Both are visible without computing anything, which is why contour plots are worth reading fluently.
WarningWatch out
In high dimensions, saddle points vastly outnumber local minima. For a random symmetric Hessian, a critical point is a minimum only if all \(n\) eigenvalues happen to be positive, which becomes vanishingly unlikely as \(n\) grows.
This reframes a common worry about deep learning. The obstacle is rarely a bad local minimum; it is the long flat stretches near saddles, where the gradient is small and progress stalls. Momentum (Section 16.6) exists largely to push through them.
12.10 Summary
| Object | Definition | Shape | Tells you |
|---|---|---|---|
| Partial derivative | vary one input | scalar | rate along an axis |
| Gradient \(\nabla f\) | all partials | \(n \times 1\) | steepest ascent; \(\perp\) level set |
| Directional derivative | \(\nabla f \cdot \mathbf{u}\) | scalar | rate along \(\mathbf{u}\) |
| Jacobian \(\mathbf{J}\) | partials of a vector output | \(m \times n\) | local linear map |
| Hessian \(\mathbf{H}\) | second partials | \(n \times n\) | curvature |
| \(\nabla f = \mathbf{0}\) | — | — | critical point |
| \(\mathbf{H}\) definiteness | — | — | min, max, or saddle |
12.11 Exercises
1. Find both partial derivatives of \(f(x,y) = 3x^2y - y^3 + 5\) and evaluate at \((1,2)\).
TipSolution
2. For \(f = x^2 + y^2\) at \((3,4)\), find the direction of steepest ascent and the rate along it.
TipSolution
\(\nabla f = (6, 8)\), with norm 10. The unit direction is \((0.6, 0.8)\) and the rate is \(\|\nabla f\| = 10\).
norm unit1 unit2
6.0 8.0 10.0 0.6 0.8 The gradient points straight away from the origin, as every gradient of this bowl does — which is what makes its level sets circles.
3. Compute the Hessian of \(f = x^2 + xy + y^2\) and classify the critical point.
TipSolution
\(\nabla f = (2x + y,\; x + 2y) = \mathbf{0}\) only at the origin. \(\mathbf{H} = \begin{bmatrix}2 & 1\\1 & 2\end{bmatrix}\), constant.
[,1] [,2]
[1,] 2 1
[2,] 1 2[1] 3 1Eigenvalues 3 and 1, both positive: positive definite, so the origin is a minimum. This is the matrix from Chapter 8 — its level sets are the tilted ellipses of fig-quadratic-form, and the cross term \(xy\) is what tilts them off the axes.
4. Verify Equation 12.4 for \(f = xy\) with \(x = \cos t\), \(y = \sin t\) at \(t = \pi/4\).
TipSolution
\(dz/dt = y(-\sin t) + x(\cos t) = \cos^2 t - \sin^2 t = \cos 2t\). At \(t = \pi/4\) that is \(\cos(\pi/2) = 0\).
chain numeric
6.123234e-17 0.000000e+00 Zero: at \(t = \pi/4\) the product \(\cos t\sin t\) is at its maximum, so moving along the circle changes it not at all to first order.
5. Show that the gradient of \(f = x^2 + y^2\) at any point is perpendicular to the level set through that point.
TipSolution
The level set through \((a,b)\) is the circle of radius \(\sqrt{a^2+b^2}\). Its tangent at \((a,b)\) is perpendicular to the radius vector \((a,b)\). The gradient is \((2a, 2b) = 2(a,b)\) — parallel to the radius, hence perpendicular to the tangent.
[1] 0Dot product zero. This is general, not special to circles: moving along a level set holds \(f\) constant, so by Equation 12.3 the directional derivative along the contour is zero, which forces \(\nabla f \perp\) contour.
6. The function \(f(x,y) = x^2 - y^2\) has a critical point at the origin. Confirm it is a saddle, and find a direction along which \(f\) decreases.
TipSolution
[1] 0 0[1] 2 -2The gradient vanishes and the Hessian’s eigenvalues are \(2\) and \(-2\): indefinite, so a saddle.
\(f\) decreases along the eigenvector for the negative eigenvalue, which here is the \(y\)-axis:
Moving along \(y\) lowers the function; moving along \(x\) raises it. That is what a saddle is, and no one-dimensional function can do it.