4  Vectors

A vector is the smallest idea in linear algebra and the one everything else is built on. A row of a spreadsheet is a vector. A word embedding is a vector. The weights of a linear model are a vector. The gradient you follow downhill during training is a vector.

This chapter builds the three things you need: what a vector is, what you can do with one, and how to see what you are doing.

4.1 What a vector is

A vector is an ordered list of numbers.

\[ \mathbf{a} = \begin{bmatrix} 3 \\ 1 \end{bmatrix} \]

The numbers are the elements or components. Their order matters: \(\begin{bmatrix} 3 \\ 1\end{bmatrix}\) and \(\begin{bmatrix} 1 \\ 3\end{bmatrix}\) are different vectors. The number of elements is the dimension. A vector with \(n\) real-valued elements lives in the set \(\mathbb{R}^n\), written \(\mathbf{a} \in \mathbb{R}^n\) and read “a is in R-n”.

We write vectors as columns and refer to the \(i\)-th element as \(a_i\), not bold, because it is a single number. So \(a_1 = 3\) and \(a_2 = 1\).

There are three ways to picture the same object, and fluency means moving between them without thinking:

View \(\begin{bmatrix} 3 \\ 1\end{bmatrix}\) is… Useful when
List two numbers in a fixed order storing data, writing code
Point the location \((3, 1)\) plotting a dataset
Arrow a displacement: 3 right, 1 up reasoning about geometry

The arrow view is the one that makes the rest of linear algebra feel inevitable, so we lean on it throughout.

NoteIn machine learning

One observation in a dataset is a vector. If you record a patient’s age, systolic blood pressure, and BMI, that patient is a point in \(\mathbb{R}^3\). A dataset of 1,000 patients is 1,000 points in \(\mathbb{R}^3\). “Learning” mostly means finding structure in where those points sit.

4.2 Vectors in R

R has no separate column/row distinction for plain vectors — a numeric vector is just a sequence of numbers. The column convention is a mathematical one that we honor when we get to matrices.

a <- c(3, 1)
b <- c(1, 2)
a
[1] 3 1
[1] 2
a[1]
[1] 3

Indexing starts at 1, matching the math. length() gives the dimension.

Higher dimensions work the same way:

x <- c(2.1, -0.4, 7.0, 3.3, 0.0)
length(x)
[1] 5

4.3 The arrow view

Draw \(\mathbf{a}\) as an arrow from the origin to the point \((3, 1)\).

draw_vectors(list(a = a))
Figure 4.1: The vector \(\mathbf{a} = (3, 1)\) drawn as an arrow from the origin.

The arrow encodes two things at once: a direction (up and to the right, at a shallow angle) and a magnitude (how far). Everything we do to vectors changes one, the other, or both.

Strictly, a vector is a displacement, not a position, so the same vector can be drawn starting anywhere — \((3, 1)\) drawn from the origin and \((3, 1)\) drawn starting at \((5, 5)\) are the same vector. Anchoring at the origin is a convention that lets us identify vectors with points. We break it deliberately in Figure 4.2.

4.4 Addition

Add vectors element by element. They must have the same dimension.

\[ \mathbf{a} + \mathbf{b} = \begin{bmatrix} a_1 \\ a_2 \end{bmatrix} + \begin{bmatrix} b_1 \\ b_2 \end{bmatrix} = \begin{bmatrix} a_1 + b_1 \\ a_2 + b_2 \end{bmatrix} \]

Worked example. With \(\mathbf{a} = \begin{bmatrix}3\\1\end{bmatrix}\) and \(\mathbf{b} = \begin{bmatrix}1\\2\end{bmatrix}\):

\[ \mathbf{a} + \mathbf{b} = \begin{bmatrix} 3 + 1 \\ 1 + 2 \end{bmatrix} = \begin{bmatrix} 4 \\ 3 \end{bmatrix} \]

a + b
[1] 4 3

Geometrically, addition is tip-to-tail: start at the origin, walk along \(\mathbf{a}\), then from where you land walk along \(\mathbf{b}\). Where you end up is \(\mathbf{a} + \mathbf{b}\).

draw_vectors(
  list(a = a, b = b, `a+b` = a + b),
  origin = list(c(0, 0), a, c(0, 0)),
  line_type = c("solid", "dashed", "solid"),
  # b and a+b share a tip, so label b at its midpoint instead
  label_at = c("tip", "mid", "tip")
)
Figure 4.2: Tip-to-tail addition. Walking \(\mathbf{a}\) then \(\mathbf{b}\) (dashed) lands at the same place as the single arrow \(\mathbf{a} + \mathbf{b}\).

Because addition is elementwise and numbers commute, \(\mathbf{a} + \mathbf{b} = \mathbf{b} + \mathbf{a}\). Geometrically that is the statement that both routes around the parallelogram end at the same corner.

Subtraction works the same way: \(\mathbf{a} - \mathbf{b}\) is elementwise subtraction, and as an arrow it points from the tip of \(\mathbf{b}\) to the tip of \(\mathbf{a}\).

a - b
[1]  2 -1
WarningWatch out

R recycles shorter vectors instead of erring:

c(1, 2, 3, 4) + c(10, 20)
[1] 11 22 13 24

That is c(11, 22, 13, 24) — R reused 10, 20. Mathematically this is nonsense; in R it is silent. Check length() when a result surprises you.

4.5 Scalar multiplication

Multiplying a vector by a single number — a scalar — multiplies every element:

\[ c\mathbf{a} = \begin{bmatrix} c\,a_1 \\ c\,a_2 \end{bmatrix} \]

2 * a
[1] 6 2
-1 * a
[1] -3 -1
0.5 * a
[1] 1.5 0.5

The arrow keeps its direction and changes length by a factor of \(|c|\). A negative scalar flips it around.

draw_vectors(
  list(`2a` = 2 * a, a = a, `0.5a` = 0.5 * a, `-a` = -a),
  # Collinear arrows: push two labels off the shared line
  label_position = c("auto", "top", "bottom", "auto")
)
Figure 4.3: Scaling \(\mathbf{a}\). All four arrows lie on one line through the origin.

Notice that every multiple of \(\mathbf{a}\) lands on the same line through the origin. That line is everything you can reach using only \(\mathbf{a}\) and scaling — the first hint of what a span is (Section 7.3).

4.6 Linear combinations

Put addition and scaling together and you get the single most important operation in the subject. A linear combination of vectors \(\mathbf{v}_1, \dots, \mathbf{v}_k\) is

\[ c_1\mathbf{v}_1 + c_2\mathbf{v}_2 + \cdots + c_k\mathbf{v}_k \]

for scalars \(c_1, \dots, c_k\) called coefficients or weights.

Worked example. \(2\mathbf{a} - \mathbf{b}\) with our vectors:

\[ 2\begin{bmatrix}3\\1\end{bmatrix} - \begin{bmatrix}1\\2\end{bmatrix} = \begin{bmatrix}6\\2\end{bmatrix} - \begin{bmatrix}1\\2\end{bmatrix} = \begin{bmatrix}5\\0\end{bmatrix} \]

2 * a - b
[1] 5 0

Sweep the coefficients over a grid and you can see what two vectors can reach between them.

grid <- expand.grid(c1 = -2:2, c2 = -2:2)
pts <- t(apply(grid, 1, function(cc) cc[1] * a + cc[2] * b))
draw_plane(
  points = pts,
  vectors = list(a = a, b = b),
  point_color = amds_gray
)
Figure 4.4: Every point is \(c_1\mathbf{a} + c_2\mathbf{b}\) for some \(c_1, c_2 \in \{-2, \dots, 2\}\). With unrestricted coefficients these two vectors reach every point in the plane.
NoteIn machine learning

A linear model is a linear combination. Prediction \(\hat{y} = w_1x_1 + \cdots + w_px_p\) combines the feature vector’s components with the learned weights. Training a linear model means searching for the coefficients. Almost every “linear” method in ML — linear/logistic regression, PCA, the read-out layer of a neural network — is a question about which linear combination to use.

4.7 Length

How long is an arrow? In two dimensions, Pythagoras:

\[ \|\mathbf{a}\| = \sqrt{a_1^2 + a_2^2} \]

In \(n\) dimensions the same formula keeps working:

\[ \|\mathbf{a}\|_2 = \sqrt{\sum_{i=1}^{n} a_i^2} \tag{4.1}\]

This is the Euclidean norm or \(L_2\) norm. The subscript 2 distinguishes it from the alternatives in Section 4.8; when you see \(\|\cdot\|\) with no subscript, assume \(L_2\).

Worked example. \(\|\mathbf{a}\| = \sqrt{3^2 + 1^2} = \sqrt{10} \approx 3.162\).

sqrt(sum(a^2))
[1] 3.162278

Two idiomatic alternatives:

norm(a, type = "2") # only type "2" accepts a plain vector
[1] 3.162278
         [,1]
[1,] 3.162278

crossprod(a) computes \(\mathbf{a}^\top\mathbf{a}\) and is the fastest of the three on large vectors, though it returns a \(1\times1\) matrix rather than a plain number.

A norm has three properties worth naming, because they are exactly what makes it a sensible notion of size:

  1. \(\|\mathbf{a}\| \geq 0\), with equality only for the zero vector.
  2. \(\|c\mathbf{a}\| = |c|\,\|\mathbf{a}\|\) — scaling scales the length.
  3. \(\|\mathbf{a} + \mathbf{b}\| \leq \|\mathbf{a}\| + \|\mathbf{b}\|\) — the triangle inequality: a detour is never shorter than going direct.

4.8 Other norms

\(L_2\) is not the only way to measure size. The general \(p\)-norm is

\[ \|\mathbf{a}\|_p = \left(\sum_{i=1}^{n} |a_i|^p\right)^{1/p} \]

Three cases matter in practice:

Norm Formula For \(\mathbf{a} = (3, 1)\) Nickname
\(L_1\) \(\sum_i \lvert a_i\rvert\) \(4\) Manhattan, taxicab
\(L_2\) \(\sqrt{\sum_i a_i^2}\) \(\sqrt{10} \approx 3.162\) Euclidean
\(L_\infty\) \(\max_i \lvert a_i\rvert\) \(3\) max, Chebyshev 
sum(abs(a))
[1] 4
sqrt(sum(a^2))
[1] 3.162278
max(abs(a))
[1] 3

The clearest way to see the difference is to draw the unit ball of each norm: the set of all vectors of length exactly 1.

theta <- seq(0, 2 * pi, length.out = 400)
l2 <- cbind(cos(theta), sin(theta))
l1 <- cbind(c(1, 0, -1, 0, 1), c(0, 1, 0, -1, 0))
linf <- cbind(c(1, -1, -1, 1, 1), c(1, 1, -1, -1, 1))
draw_curves(list(L1 = l1, L2 = l2, Linf = linf))
Figure 4.5: Unit balls. Every point on a curve is at distance 1 from the origin, according to that norm. \(L_1\) is a diamond, \(L_2\) a circle, \(L_\infty\) a square.

The \(L_1\) diamond has corners on the axes. That geometric fact is the whole reason Lasso regression produces exactly-zero coefficients while ridge regression does not.

NoteIn machine learning

Norms appear as regularizers, terms added to a loss function to penalize large weights: ridge adds \(\lambda\|\mathbf{w}\|_2^2\), Lasso adds \(\lambda\|\mathbf{w}\|_1\). They also appear as losses: mean squared error is a squared \(L_2\) distance between predictions and targets, mean absolute error is an \(L_1\) distance.

4.9 Distance

The distance between two vectors is the length of their difference:

\[ d(\mathbf{a}, \mathbf{b}) = \|\mathbf{a} - \mathbf{b}\| \]

Worked example. \(\mathbf{a} - \mathbf{b} = \begin{bmatrix}2\\-1\end{bmatrix}\), so \(d = \sqrt{4 + 1} = \sqrt{5} \approx 2.236\).

sqrt(sum((a - b)^2))
[1] 2.236068
dist(rbind(a, b))
         a
b 2.236068
NoteIn machine learning

\(k\)-nearest neighbors, \(k\)-means, hierarchical clustering, and UMAP are all built on a distance between vectors. Changing the norm changes the neighbors you get, which is why the choice of distance is a modeling decision and not a detail.

4.10 Unit vectors

A unit vector has length 1. Any non-zero vector can be turned into one by dividing by its length — normalizing it:

\[ \hat{\mathbf{a}} = \frac{\mathbf{a}}{\|\mathbf{a}\|} \]

Worked example. \(\hat{\mathbf{a}} = \frac{1}{\sqrt{10}}\begin{bmatrix}3\\1\end{bmatrix} \approx \begin{bmatrix}0.949\\0.316\end{bmatrix}\).

a_hat <- a / sqrt(sum(a^2))
a_hat
[1] 0.9486833 0.3162278
sqrt(sum(a_hat^2))
[1] 1

Normalizing throws away magnitude and keeps direction. That is often exactly what you want — see cosine similarity in Section 4.15.

4.11 The dot product

The dot product (or inner product) of two vectors of the same dimension multiplies them elementwise and sums the result:

\[ \mathbf{a} \cdot \mathbf{b} = \mathbf{a}^\top\mathbf{b} = \sum_{i=1}^{n} a_ib_i \tag{4.2}\]

The output is a single number, not a vector. That is the point: the dot product collapses two vectors into one scalar summary of how much they agree.

Worked example. \(\mathbf{a}\cdot\mathbf{b} = (3)(1) + (1)(2) = 3 + 2 = 5\).

sum(a * b)
[1] 5

In R the matrix operator %*% also does this, returning a \(1\times1\) matrix:

a %*% b
     [,1]
[1,]    5
drop(a %*% b)
[1] 5

drop() strips the dimensions to give a plain number. For large vectors, crossprod() is the fast route:

[1] 5

Two identities to internalize now, because they get used constantly:

\[ \mathbf{a}\cdot\mathbf{a} = \|\mathbf{a}\|_2^2 \qquad\text{and}\qquad \mathbf{a}\cdot\mathbf{b} = \mathbf{b}\cdot\mathbf{a} \]

sum(a * a)
[1] 10
sum(a^2)
[1] 10

4.12 The dot product and angle

Here is where the dot product earns its keep. For non-zero vectors,

\[ \mathbf{a}\cdot\mathbf{b} = \|\mathbf{a}\|\,\|\mathbf{b}\|\cos\theta \tag{4.3}\]

where \(\theta\) is the angle between the two arrows. Rearranged:

\[ \cos\theta = \frac{\mathbf{a}\cdot\mathbf{b}}{\|\mathbf{a}\|\,\|\mathbf{b}\|} \]

Worked example.

\[ \cos\theta = \frac{5}{\sqrt{10}\cdot\sqrt{5}} = \frac{5}{\sqrt{50}} = \frac{5}{5\sqrt{2}} = \frac{1}{\sqrt{2}} \approx 0.707 \]

so \(\theta = 45°\).

cos_theta <- sum(a * b) / (sqrt(sum(a^2)) * sqrt(sum(b^2)))
cos_theta
[1] 0.7071068
acos(cos_theta) * 180 / pi
[1] 45
ends <- c(atan2(a[2], a[1]), atan2(b[2], b[1]))
sweep <- seq(ends[1], ends[2], length.out = 60)
draw_vectors(
  list(a = a, b = b),
  curves = list(cbind(0.6 * cos(sweep), 0.6 * sin(sweep))),
  curve_color = amds_gray,
  notes = list(`45°` = c(0.64, 0.56))
)
Figure 4.6: \(\mathbf{a}\) and \(\mathbf{b}\) meet at 45°. Their dot product, 5, is positive because they broadly point the same way.

The sign of the dot product is the fast reading:

\(\mathbf{a}\cdot\mathbf{b}\) \(\theta\) Interpretation
\(> 0\) \(< 90°\) pointing broadly the same way
\(= 0\) \(= 90°\) perpendicular; unrelated
\(< 0\) \(> 90°\) pointing broadly opposite

4.13 Orthogonality

Two vectors are orthogonal — the general word for perpendicular — when their dot product is zero:

\[ \mathbf{u} \perp \mathbf{v} \iff \mathbf{u}\cdot\mathbf{v} = 0 \]

Worked example. \(\mathbf{u} = \begin{bmatrix}2\\1\end{bmatrix}\) and \(\mathbf{v} = \begin{bmatrix}-1\\2\end{bmatrix}\): \(\mathbf{u}\cdot\mathbf{v} = -2 + 2 = 0\).

u <- c(2, 1)
v <- c(-1, 2)
sum(u * v)
[1] 0
draw_vectors(list(u = u, v = v))
Figure 4.7: Orthogonal vectors: the dot product is exactly zero.

Note that this definition works in \(\mathbb{R}^{300}\) where you cannot draw anything. That transfer — a geometric idea surviving as an algebraic test in dimensions you cannot picture — is the central move of linear algebra.

Pick two random vectors in \(\mathbb{R}^n\). As \(n\) grows, the angle between them concentrates sharply around 90°.

set.seed(2024)
random_cos <- function(n) {
  p <- rnorm(n)
  q <- rnorm(n)
  sum(p * q) / (sqrt(sum(p^2)) * sqrt(sum(q^2)))
}
sapply(c(2, 10, 100, 1000, 10000), function(n) {
  mean(abs(replicate(200, random_cos(n))))
})
[1] 0.645915797 0.254920985 0.078806980 0.024095830 0.008903705

The mean absolute cosine shrinks toward zero. In high dimensions, almost everything is almost orthogonal to almost everything else — one face of the “curse of dimensionality”, and the reason random projections preserve structure surprisingly well.

4.14 Projection

The projection of \(\mathbf{b}\) onto \(\mathbf{a}\) is the part of \(\mathbf{b}\) that lies along \(\mathbf{a}\) — the shadow \(\mathbf{b}\) casts on \(\mathbf{a}\)’s line:

\[ \operatorname{proj}_{\mathbf{a}}(\mathbf{b}) = \frac{\mathbf{a}\cdot\mathbf{b}}{\mathbf{a}\cdot\mathbf{a}}\,\mathbf{a} \tag{4.4}\]

The fraction is a scalar saying how many copies of \(\mathbf{a}\); multiplying by \(\mathbf{a}\) turns that back into a vector.

Worked example. \(\mathbf{a}\cdot\mathbf{b} = 5\) and \(\mathbf{a}\cdot\mathbf{a} = 10\), so

\[ \operatorname{proj}_{\mathbf{a}}(\mathbf{b}) = \tfrac{5}{10} \begin{bmatrix}3\\1\end{bmatrix} = \begin{bmatrix}1.5\\0.5\end{bmatrix} \]

The leftover, \(\mathbf{b} - \operatorname{proj}_{\mathbf{a}}(\mathbf{b}) = \begin{bmatrix}-0.5\\1.5\end{bmatrix}\), is the residual. It is orthogonal to \(\mathbf{a}\) by construction — check: \((3)(-0.5) + (1)(1.5) = 0\).

proj <- (sum(a * b) / sum(a * a)) * a
resid <- b - proj
proj
[1] 1.5 0.5
resid
[1] -0.5  1.5
sum(a * resid)
[1] 0
draw_vectors(
  list(a = a, b = b, proj = proj, resid = resid),
  origin = list(c(0, 0), c(0, 0), c(0, 0), proj),
  line_type = c("solid", "solid", "solid", "dashed"),
  # proj sits on a, resid shares b's tip: move both labels
  label_at = c("tip", "tip", "tip", "mid"),
  label_position = c("auto", "auto", "bottom", "auto")
)
Figure 4.8: Projecting \(\mathbf{b}\) onto \(\mathbf{a}\) splits \(\mathbf{b}\) into a part along \(\mathbf{a}\) and a residual perpendicular to it. The two parts add back to \(\mathbf{b}\).

That decomposition — the part I can explain, plus the part I cannot — is the geometric content of least squares regression, and we return to it in Section 6.8.

NoteIn machine learning

Fitting a linear model by least squares projects the target vector \(\mathbf{y}\) onto the space spanned by your features. The fitted values are the projection; the residuals are what is left over, orthogonal to every feature. “The residuals are uncorrelated with the predictors” is a geometric statement, not a statistical coincidence.

4.15 Cosine similarity

Rearranging Equation 4.3 gives a similarity measure between 1 (identical direction) and \(-1\) (opposite direction):

\[ \text{cos-sim}(\mathbf{a}, \mathbf{b}) = \frac{\mathbf{a}\cdot\mathbf{b}}{\|\mathbf{a}\|\,\|\mathbf{b}\|} = \hat{\mathbf{a}} \cdot \hat{\mathbf{b}} \]

It compares direction only — magnitude is divided out. That is exactly what you want when comparing documents of different lengths, or users with different activity levels.

Worked example. Three documents, counted over the vocabulary (model, patient, data, gradient):

d1 <- c(model = 3, patient = 0, data = 2, gradient = 1)
d2 <- c(model = 1, patient = 4, data = 0, gradient = 0)
d3 <- c(model = 6, patient = 0, data = 4, gradient = 2)

cos_sim <- function(x, y) {
  sum(x * y) / (sqrt(sum(x^2)) * sqrt(sum(y^2)))
}

c(
  `d1-d2` = cos_sim(d1, d2),
  `d1-d3` = cos_sim(d1, d3),
  `d2-d3` = cos_sim(d2, d3)
)
    d1-d2     d1-d3     d2-d3 
0.1944611 1.0000000 0.1944611

d3 is exactly 2 * d1 — twice as long, same subject matter — and cosine similarity returns exactly 1. Euclidean distance would call them far apart:

sqrt(sum((d1 - d3)^2))
[1] 3.741657

Same data, two defensible answers. Which one is right depends on whether length is signal or noise for your problem.

draw_bar(
  x = c("d1-d2", "d1-d3", "d2-d3"),
  y = c(cos_sim(d1, d2), cos_sim(d1, d3), cos_sim(d2, d3)),
  ylab = "cosine similarity"
)
Figure 4.9: Cosine similarity between the three documents. d1 and d3 are proportional, so their similarity is exactly 1.
NoteIn machine learning

Cosine similarity is the default retrieval metric for embeddings — text, images, audio. A vector database answering “find me documents like this one” is, at bottom, computing dot products of normalized vectors and returning the largest. The attention mechanism in a transformer is built from the same operation.

4.16 Vectorized thinking

Every operation in this chapter is defined elementwise, and R implements them in compiled code. Writing loops instead is both slower and harder to read.

set.seed(1)
p <- rnorm(1e6)
q <- rnorm(1e6)

loop_dot <- function(x, y) {
  total <- 0
  for (i in seq_along(x)) total <- total + x[i] * y[i]
  total
}

system.time(loop_dot(p, q))["elapsed"]
elapsed 
  0.043 
system.time(sum(p * q))["elapsed"]
elapsed 
  0.006 
system.time(drop(crossprod(p, q)))["elapsed"]
elapsed 
  0.002

The three give the same answer. Prefer the vectorized form for the same reason you prefer \(\sum_i a_ib_i\) to writing out a million terms: it says what you mean.

4.17 Summary

Operation Notation R Result
Addition \(\mathbf{a} + \mathbf{b}\) a + b vector
Scaling \(c\mathbf{a}\) c * a vector
\(L_2\) norm \(\|\mathbf{a}\|_2\) sqrt(sum(a^2)) scalar
\(L_1\) norm \(\|\mathbf{a}\|_1\) sum(abs(a)) scalar
Distance \(\|\mathbf{a} - \mathbf{b}\|\) sqrt(sum((a - b)^2)) scalar
Normalize \(\mathbf{a}/\|\mathbf{a}\|\) a / sqrt(sum(a^2)) unit vector
Dot product \(\mathbf{a}^\top\mathbf{b}\) sum(a * b) scalar
Angle \(\arccos\frac{\mathbf{a}\cdot\mathbf{b}}{\|\mathbf{a}\|\|\mathbf{b}\|}\) acos(...) scalar
Projection \(\frac{\mathbf{a}\cdot\mathbf{b}}{\mathbf{a}\cdot\mathbf{a}}\mathbf{a}\) (sum(a*b)/sum(a*a)) * a vector

4.18 Exercises

1. Let \(\mathbf{p} = \begin{bmatrix}4\\-3\end{bmatrix}\) and \(\mathbf{q} = \begin{bmatrix}-1\\2\end{bmatrix}\). Compute \(\mathbf{p} + \mathbf{q}\), \(3\mathbf{p}\), and \(\mathbf{p} - 2\mathbf{q}\) by hand, then check in R.

\(\mathbf{p} + \mathbf{q} = \begin{bmatrix}3\\-1\end{bmatrix}\), \(3\mathbf{p} = \begin{bmatrix}12\\-9\end{bmatrix}\), \(\mathbf{p} - 2\mathbf{q} = \begin{bmatrix}4 + 2\\-3 - 4\end{bmatrix} = \begin{bmatrix}6\\-7\end{bmatrix}\).

p <- c(4, -3)
q <- c(-1, 2)
p + q
[1]  3 -1
3 * p
[1] 12 -9
p - 2 * q
[1]  6 -7

2. Compute \(\|\mathbf{p}\|_1\), \(\|\mathbf{p}\|_2\), and \(\|\mathbf{p}\|_\infty\) for \(\mathbf{p} = \begin{bmatrix}4\\-3\end{bmatrix}\). Which is largest? Is that always the case?

\(\|\mathbf{p}\|_1 = 4 + 3 = 7\), \(\|\mathbf{p}\|_2 = \sqrt{16 + 9} = 5\), \(\|\mathbf{p}\|_\infty = 4\).

c(L1 = sum(abs(p)), L2 = sqrt(sum(p^2)), Linf = max(abs(p)))
  L1   L2 Linf 
   7    5    4

\(L_1\) is largest. This is general: \(\|\mathbf{x}\|_\infty \leq \|\mathbf{x}\|_2 \leq \|\mathbf{x}\|_1\) for every vector, which is why the unit balls in Figure 4.5 nest inside one another (\(L_1\) innermost, \(L_\infty\) outermost).

3. Find a non-zero vector orthogonal to \(\begin{bmatrix}5\\2\end{bmatrix}\). How many such vectors are there?

Swap the components and negate one: \(\begin{bmatrix}-2\\5\end{bmatrix}\), since \(5(-2) + 2(5) = 0\).

w <- c(5, 2)
sum(w * c(-2, 5))
[1] 0

Infinitely many — every non-zero scalar multiple of \(\begin{bmatrix}-2\\5\end{bmatrix}\) works. In \(\mathbb{R}^2\) they all lie on a single line; in \(\mathbb{R}^n\) they fill an \((n-1)\)-dimensional space.

4. Project \(\mathbf{b} = \begin{bmatrix}4\\4\end{bmatrix}\) onto \(\mathbf{a} = \begin{bmatrix}1\\0\end{bmatrix}\). Explain the answer geometrically without computing anything.

\(\mathbf{a}\) is the horizontal unit vector, so projecting onto it keeps the horizontal component and discards the vertical one: \(\begin{bmatrix}4\\0\end{bmatrix}\).

bb <- c(4, 4)
aa <- c(1, 0)
(sum(aa * bb) / sum(aa * aa)) * aa
[1] 4 0

Projecting onto a coordinate axis is just reading off that coordinate.

5. Two users rate four films: u1 <- c(5, 3, 0, 1) and u2 <- c(1, 0, 5, 4). Compute their cosine similarity and their Euclidean distance. Are they similar?

u1 <- c(5, 3, 0, 1)
u2 <- c(1, 0, 5, 4)
cos_sim(u1, u2)
[1] 0.2347382
sqrt(sum((u1 - u2)^2))
[1] 7.681146

Cosine similarity is low, around 0.23, and the distance is large: the two users like essentially different films. Both measures agree here because the rating vectors have similar magnitudes; they would disagree if one user rated everything generously.

6. Show that \(\|\mathbf{a} + \mathbf{b}\|^2 = \|\mathbf{a}\|^2 + 2(\mathbf{a}\cdot\mathbf{b}) + \|\mathbf{b}\|^2\). What does it reduce to when \(\mathbf{a}\) and \(\mathbf{b}\) are orthogonal?

Expand using \(\|\mathbf{x}\|^2 = \mathbf{x}\cdot\mathbf{x}\) and the fact that the dot product distributes over addition:

\[ \begin{aligned} \|\mathbf{a} + \mathbf{b}\|^2 &= (\mathbf{a} + \mathbf{b})\cdot(\mathbf{a} + \mathbf{b}) \\ &= \mathbf{a}\cdot\mathbf{a} + \mathbf{a}\cdot\mathbf{b} + \mathbf{b}\cdot\mathbf{a} + \mathbf{b}\cdot\mathbf{b} \\ &= \|\mathbf{a}\|^2 + 2(\mathbf{a}\cdot\mathbf{b}) + \|\mathbf{b}\|^2 \end{aligned} \]

When \(\mathbf{a}\cdot\mathbf{b} = 0\) the middle term vanishes and you get \(\|\mathbf{a} + \mathbf{b}\|^2 = \|\mathbf{a}\|^2 + \|\mathbf{b}\|^2\) — the Pythagorean theorem, now valid in any number of dimensions.

sum((a + b)^2)
[1] 25
sum(a^2) + 2 * sum(a * b) + sum(b^2)
[1] 25