10 Limits and Continuity

Calculus is built on one idea: what happens as something gets arbitrarily close to something else? Not at it — close to it. That distinction is the whole trick, and it is what lets you divide by something that is heading toward zero and get a sensible answer.

You will not compute many limits by hand after this chapter. But derivatives, integrals, convergence rates, and every “as $n \to \infty$” statement in statistics are limits underneath, and the vocabulary is unavoidable.

10.1 Why limits

Here is the problem limits solve. Consider

\[ f(x) = \frac{x^2 - 4}{x - 2} \]

At $x = 2$ this is $0/0$ — undefined, meaningless. But everywhere else the numerator factors as $(x-2)(x+2)$, and the $(x-2)$ cancels, leaving $x + 2$.

So $f$ is not defined at 2, yet as $x$ gets close to 2 the value gets close to 4.

xs <- c(2.1, 2.01, 2.001, 2.00001)
(xs^2 - 4) / (xs - 2)

[1] 4.10000 4.01000 4.00100 4.00001

That is a limit: the value the function approaches, whether or not it ever arrives.

# n chosen so the grid steps over x = 2,
# rather than landing on it
draw_fun(
  function(x) (x^2 - 4) / (x - 2),
  from = 0, to = 4, n = 500,
  ylab = "f(x)"
)

Figure 10.1: $(x^2-4)/(x-2)$ is the line $x+2$ with a single point removed. The function has no value at $x=2$, but it has a limit there, and the limit is 4.

The graph is a straight line with an invisible puncture. Evaluate exactly at the hole and R tells you what the algebra told you:

(2^2 - 4) / (2 - 2)

[1] NaN

10.2 Sequences and their limits

The simplest limits are of sequences — infinite ordered lists $a_1, a_2, a_3, \dots$

A sequence converges to $L$, written $\lim_{n \to \infty} a_n = L$, if the terms get and stay arbitrarily close to $L$. Formally: for any tolerance $\varepsilon > 0$, there is an $N$ beyond which $|a_n - L| < \varepsilon$ for every $n > N$.

Read that as a game. You name a tolerance; I must find a point past which the sequence never leaves it. If I can always win, the sequence converges.

n <- c(1, 10, 100, 1000, 10000)
1 / n

[1] 1e+00 1e-01 1e-02 1e-03 1e-04

$1/n \to 0$. A more interesting one:

\[ \lim_{n \to \infty}\left(1 + \frac{1}{n}\right)^n = e \tag{10.1}\]

(1 + 1 / n)^n

[1] 2.000000 2.593742 2.704814 2.716924 2.718146

exp(1)

[1] 2.718282

nn <- 1:120
draw_line(
  x = nn,
  y = list(
    `(1 + 1/n)^n` = (1 + 1 / nn)^nn,
    e = rep(exp(1), length(nn))
  ),
  points = FALSE,
  xlab = "n",
  ylab = "value"
)

Figure 10.2: $(1 + 1/n)^n$ creeping up on $e$. Convergence is slow — at $n = 100$ the third decimal place is still wrong.

Not everything converges. $a_n = (-1)^n$ oscillates between $-1$ and $1$ forever and has no limit; $a_n = n$ diverges to infinity. Note that “$\lim = \infty$” is a statement about growing without bound, not about reaching a number.

10.3 The limit of a function

\[ \lim_{x \to a} f(x) = L \]

means $f(x)$ can be made as close to $L$ as you like by taking $x$ close enough to $a$ — but not equal to $a$. The value $f(a)$ is irrelevant, and may not exist.

That exclusion is the entire reason limits are useful. It is what let $(x^2-4)/(x-2)$ have a limit at a point where it has no value, and it is what will make the derivative work in Section 11.2, where the quantity of interest is $0/0$ at exactly the point we care about.

When a function is well behaved, the limit is just the value:

\[ \lim_{x \to 3} x^2 = 9 \]

That is not a triviality — it is the definition of continuity (Section 10.6), and it is why most limits are boring. The interesting ones are where it fails.

10.4 One-sided limits

You can approach $a$ from below or from above, and the answers can differ:

\[ \lim_{x \to a^-} f(x) \quad\text{(from the left)} \qquad \lim_{x \to a^+} f(x) \quad\text{(from the right)} \]

The two-sided limit exists only when both one-sided limits exist and agree.

The sign function is the standard counterexample:

\[ \operatorname{sign}(x) = \begin{cases} -1 & x < 0 \\ 0 & x = 0 \\ 1 & x > 0 \end{cases} \]

draw_fun(sign, from = -2, to = 2, n = 501, ylab = "sign(x)")

Figure 10.3: The sign function. Approaching zero from the left gives $-1$; from the right, $+1$. The two disagree, so no two-sided limit exists at 0.

sign(c(-0.001, 0, 0.001))

[1] -1  0  1

Left limit $-1$, right limit $+1$, so $\lim_{x\to 0}\operatorname{sign}(x)$ does not exist — even though $\operatorname{sign}(0)$ is perfectly well defined. Existence of the value and existence of the limit are independent.

10.5 Limits at infinity

$\lim_{x \to \infty} f(x) = L$ asks what happens for arbitrarily large $x$. For rational functions, the highest powers decide:

\[ \lim_{x \to \infty}\frac{3x^2 + 2x}{x^2 + 1} = 3 \]

Divide numerator and denominator by $x^2$ and everything but the leading terms goes to zero.

g <- function(x) (3 * x^2 + 2 * x) / (x^2 + 1)
g(c(10, 100, 10000))

[1] 3.168317 3.019698 3.000200

Some useful ones to know by sight:

Limit	Value
$\lim_{x\to\infty} 1/x^k$, $k>0$	$0$
$\lim_{x\to\infty} e^{-x}$	$0$
$\lim_{x\to\infty} \log(x)/x$	$0$
$\lim_{x\to\infty} (1 + a/x)^x$	$e^a$
$\lim_{x\to 0} \sin(x)/x$	$1$

The third says logarithms lose to linear growth; the general principle is Section 10.7.

10.6 Continuity

A function is continuous at $a$ when three things hold:

$f(a)$ exists,
$\lim_{x\to a} f(x)$ exists,
they are equal.

\[ \lim_{x \to a} f(x) = f(a) \tag{10.2}\]

Informally: you can draw the graph without lifting your pen. Each condition rules out a different failure — $(x^2-4)/(x-2)$ fails the first at $x=2$, $\operatorname{sign}$ fails the second at 0, and a function redefined at a single point fails the third.

Continuity matters because it is what most theorems assume. A continuous function on a closed interval attains a maximum and a minimum somewhere on it — which is the guarantee that makes optimization a sensible thing to attempt.

Watch out

Continuity does not imply differentiability. $|x|$ is continuous everywhere — no gap, no jump — but has a corner at 0 with no well-defined slope (Section 3.9). Smooth is a stronger condition than unbroken.

10.7 Growth rates and big-O notation

Often you do not care about the exact limit, only about how fast something grows. Big-O notation captures that:

\[ f(n) = O(g(n)) \quad\text{means}\quad \limsup_{n\to\infty}\frac{|f(n)|}{g(n)} < \infty \]

In words: $f$ grows no faster than $g$, up to a constant factor. Constants and lower-order terms are discarded, so $3n^2 + 100n + 7$ is $O(n^2)$.

The standard hierarchy, each entry beaten by the next:

\[ O(1) \ll O(\log n) \ll O(n) \ll O(n\log n) \ll O(n^2) \ll O(n^3) \ll O(2^n) \]

k <- 1:30
draw_line(
  x = k,
  y = list(
    `log n` = log(k),
    `n` = k,
    `n log n` = k * log(k),
    `n^2` = k^2
  ),
  points = FALSE,
  xlab = "n",
  ylab = "operations"
)

Figure 10.4: Growth orders. On this scale $n^2$ has already left the frame while $\log n$ is barely moving. The gaps only widen.

In machine learning

Big-O is how you know what is feasible before writing any code. Matrix multiplication and matrix inversion are $O(n^3)$ (Section 5.14), so $n = 1{,}000$ is routine and $n = 100{,}000$ is not. Exact Gaussian process regression is $O(n^3)$ in the number of observations, which is precisely why approximations exist. A gradient descent step is $O(np)$, which is why it scales where direct solves do not.

10.8 Limits numerically in R

You can approximate a limit by evaluating closer and closer. This works — until it spectacularly does not.

h <- 10^-(1:8)
(( 1 + h)^(1 / h))

[1] 2.593742 2.704814 2.716924 2.718146 2.718268 2.718280 2.718282 2.718282

Converging to $e$, as Equation 10.1 promises. Now push further:

h2 <- 10^-(14:17)
((1 + h2)^(1 / h2))

[1] 2.716110 3.035035 1.000000 1.000000

Watch how it fails. At $h = 10^{-14}$ the answer is still roughly right. At $10^{-15}$ it overshoots to about $3.04$ — worse than the estimate at $h = 10^{-1}$. Then it collapses to exactly 1 and stays there.

The reason is arithmetic, not mathematics. Storing $1 + h$ keeps only about 16 significant digits, so most of $h$ is thrown away — and then raising that slightly wrong number to the enormous power $1/h$ amplifies the damage. Once $h$ falls below machine epsilon, $1 + h$ rounds to exactly 1, and 1 to any power is 1.

Note that the failure is not gradual. It is erratic before it is catastrophic, which means you cannot detect it by checking whether successive estimates are still moving.

.Machine$double.eps

[1] 2.220446e-16

1 + 1e-17 == 1

[1] TRUE

Watch out

Numerically, closer is not always better. A limit says “as $h \to 0$”; floating point says “below about $10^{-16}$ there is nothing left to subtract”. Every naive numerical limit has a sweet spot: too large and you have not converged, too small and rounding error dominates.

This is not an edge case. It sets the accuracy of every finite-difference derivative, and Section 11.11 shows the resulting U-shaped error curve in detail.

10.9 Summary

Concept	Notation	Key point
Sequence limit	$\lim_{n\to\infty}a_n = L$	terms get and stay close to $L$
Function limit	$\lim_{x\to a}f(x) = L$	$f(a)$ is irrelevant
One-sided	$x \to a^-$, $x \to a^+$	two-sided exists iff both agree
Continuity	$\lim_{x\to a}f(x) = f(a)$	no gaps; weaker than smooth
Big-O	$f = O(g)$	grows no faster than $g$

10.10 Exercises

1. Evaluate $\lim_{x\to 3}\dfrac{x^2-9}{x-3}$ algebraically, then confirm numerically.

Solution

Factor: $\frac{(x-3)(x+3)}{x-3} = x+3$ for $x \neq 3$, so the limit is $6$.

xa <- 3 + 10^-(1:5)
(xa^2 - 9) / (xa - 3)

[1] 6.10000 6.01000 6.00100 6.00010 6.00001

The same removable discontinuity as Figure 10.1: cancelling a factor is legitimate precisely because the limit never evaluates at the bad point.

2. Does $\lim_{x\to 0} 1/x$ exist? What about $\lim_{x\to 0} 1/x^2$?

Solution

Neither exists as a finite number, but they fail differently.

$1/x$ goes to $-\infty$ from the left and $+\infty$ from the right — the one-sided limits disagree, so there is no limit in any sense.

$1/x^2$ goes to $+\infty$ from both sides. It still has no finite limit, but the behavior is consistent, and we write $\lim_{x\to 0}1/x^2 = \infty$ as shorthand for “grows without bound”.

hh <- c(-0.01, -0.001, 0.001, 0.01)
rbind(one_over_x = 1 / hh, one_over_x2 = 1 / hh^2)

             [,1]   [,2]  [,3]  [,4]
one_over_x   -100 -1e+03 1e+03   100
one_over_x2 10000  1e+06 1e+06 10000

3. Is $f(x) = \begin{cases} x^2 & x \leq 1 \\ 2x - 1 & x > 1\end{cases}$ continuous at $x = 1$?

Solution

Check the three conditions. $f(1) = 1^2 = 1$. The left limit is $1^2 = 1$; the right limit is $2(1)-1 = 1$. All three agree, so yes, continuous.

fp <- function(x) ifelse(x <= 1, x^2, 2 * x - 1)
fp(c(0.999, 1, 1.001))

[1] 0.998001 1.000000 1.002000

The two pieces were chosen to meet. Change $2x-1$ to $2x$ and the right limit becomes 2, producing a jump.

4. Order these by growth rate: $n^2$, $n\log n$, $2^n$, $\log n$, $n$.

Solution

\[ \log n \;\ll\; n \;\ll\; n\log n \;\ll\; n^2 \;\ll\; 2^n \]

nq <- 20
c(
  log_n = log(nq), n = nq, n_log_n = nq * log(nq),
  n_sq = nq^2, two_n = 2^nq
)

       log_n            n      n_log_n         n_sq        two_n 
2.995732e+00 2.000000e+01 5.991465e+01 4.000000e+02 1.048576e+06

At $n = 20$ the exponential is already five orders of magnitude past $n^2$. Exponential growth is not “fast” — it is a different category.

5. Find the smallest h for which 1 + h != 1 in double precision, and relate it to .Machine$double.eps.

Solution

c(
  eps = .Machine$double.eps,
  at_eps = 1 + .Machine$double.eps != 1,
  at_half_eps = 1 + .Machine$double.eps / 2 != 1
)

         eps       at_eps  at_half_eps 
2.220446e-16 1.000000e+00 0.000000e+00

Machine epsilon is the gap between 1 and the next representable number, about $2.2\times10^{-16}$. Adding anything smaller than half of it to 1 rounds straight back to 1.

This single number sets the floor for every numerical limit, derivative, and convergence test you will write.

Limit	Value
\(\lim_{x\to\infty} 1/x^k\), \(k>0\)	\(0\)
\(\lim_{x\to\infty} e^{-x}\)	\(0\)
\(\lim_{x\to\infty} \log(x)/x\)	\(0\)
\(\lim_{x\to\infty} (1 + a/x)^x\)	\(e^a\)
\(\lim_{x\to 0} \sin(x)/x\)	\(1\)