3 Derivatives

Throughout this chapter, we will work exclusively with continuous functions that map intervals of real numbers into the real numbers. So unless I specify otherwise, whenever I refer to a function f in this chapter, assume that:

The domain of f is the real numbers \mathbb{R}, or an interval of reals.
The codomain of f is the real numbers \mathbb{R}.
f is continuous.

3.1 How steep is the function?

The derivative of a function, loosely speaking, is a measure of how steeply the function is increasing or decreasing at a particular point along its domain. This idea of steepness is easiest to see with linear functions.

Figure 3.1: Two linear functions—one steeper, one flatter.

Between the two linear functions depicted in Figure 3.1, g(x) is steeper and h(x) is flatter. Specifically, the slope of g(x) is greater in magnitude than that of h(x). You can calculate the slope of a linear function using the “rise over run” formula. Given any two distinct points along the domain, x_1 and x_2, we calculate the slope by dividing the difference in function values (the “rise”) by the difference in domain values (the “run”): \text{slope} = \frac{f(x_1) - f(x_2)}{x_1 - x_2}. \tag{3.1}

For example, for the functions we’ve plotted in Figure 3.1, let’s compare the function values at x_1 = 4 to those at x_2 = 0. For the steeper function g, we have g(x_1) = g(4) = 4 and g(x_2) = g(0) = 0. Therefore, the slope of g is 1: \text{slope of $g$} = \frac{g(x_1) - g(x_2)}{x_1 - x_2} = \frac{g(4) - g(0)}{4} = \frac{4 - 0}{4 - 0} = 1. By contrast, the slope of the flatter function h is just 1/4: \text{slope of $h$} = \frac{h(x_1) - h(x_2)}{x_1 - x_2} = \frac{h(4) - h(0)}{4 - 0} = \frac{3 - 2}{4} = \frac{1}{4}.

Exercise 3.1 Calculate the slope of each function depicted in the figure below. What are the similarities and differences with the slopes of the functions depicted in Figure 3.1?

Answer

The slope of g is -1: \begin{aligned} \frac{g(5) - g(4)}{5 - 4} = \frac{0 - 1}{1} = -1. \end{aligned} The slope of h is -1/4: \begin{aligned} \frac{h(5) - h(1)}{5 - 1} = \frac{2 - 3}{4} = \frac{-1}{4}. \end{aligned} The major contrast with the functions from the earlier figure is that the slopes are now negative, reflecting the fact that these functions are decreasing instead of increasing. For decreasing functions, the “steeper” decrease is still the function whose slope is greater in absolute value.

Linear functions are convenient because they are equally “steep” at every point along their domain. Nonlinear functions, by definition, are not so convenient—their steepness varies across their domain. As an example, take a look at the nonlinear functions depicted in Figure 3.2: \begin{aligned} g(x) &= \frac{x^2}{5}; \\ h(x) &= 4 - \frac{(x - 5)^2}{5}. \end{aligned}

Figure 3.2: Two increasing nonlinear functions. At lower values of x, h(x) is increasing more steeply than g(x). The opposite is true at greater values of x.

At the leftmost part of the figure, close to x = 0, the red function g(x) is nearly flat while the blue function h(x) is clearly increasing. By contrast, at the rightmost part of the figure, close to x = 5, we see that g(x) is clearly increasing while h(x) is nearly flat. We can glean that much through the eyeball test. But how can we quantify the “steepness” of each function at each point? Is there a precise way to say that a function is “nearly flat” at a particular point? And how can we identify the crossover point, where g(x) starts to increase more steeply than h(x) does?

3.2 Defining the derivative

One way to calculate the steepness of a nonlinear function at a particular point would be to use the “rise over run” formula (Equation 3.1). In other words, to gauge the steepness at some point c in the domain of the function, we pick a nearby point x and calculate \text{steepness at $c$} \approx \frac{f(x) - f(c)}{x - c}. The closer our chosen point x is to c, the better this approximation will be. For example, let’s calculate this approximation at c = 0 for the red function g(x) plotted in Figure 3.2. We can tell from the graph that the function is basically flat at c = 0, so we should get a “steepness” calculation close to 0.

You can do a few of these calculations yourself to see that the approximation gets closer to 0 as we pick approximation points closer to c = 0 along the x-axis… \begin{aligned} \frac{g(3) - g(0)}{3 - 0} = \frac{(3^2/5) - (0^2/5)}{3 - 0} = \frac{9/5}{3} = \frac{3}{5}; \\ \frac{g(2) - g(0)}{2 - 0} = \frac{(2^2/5) - (0^2/5)}{2 - 0} = \frac{4/5}{2} = \frac{2}{5}; \\ \frac{g(1) - g(0)}{1 - 0} = \frac{(1^2/5) - (0^2/5)}{1 - 0} = \frac{1/5}{1} = \frac{1}{5}; \end{aligned} …or you can look at Figure 3.3 for an animated illustration of the process.

Figure 3.3: Closer and closer approximations of the how steep the function is at c = 0.

No matter how close of an approximation we take, there’s always a closer one to take. You should never be satisfied with the approximation \text{steepness at $c$} \approx \frac{f(x) - f(c)}{x - c}, because you could have calculated an even better approximation at the midpoint between c and your previously chosen approximation point x, \text{steepness at $c$} \approx \frac{f(\frac{x + c}{2}) - f(c)}{\frac{x + c}{2} - c}.

The only way to escape the we-could-have-gotten-even-closer complaint is to take the limit. We will formally define the derivative of a function as the limit of the rise-over-run calculation (Equation 3.1) as we choose approximation points x ever closer and closer to c, the point at which we want to calculate the steepness of the function.

Definition 3.1 (Derivative of a function) Consider the function f : X \to \mathbb{R}, where X \subseteq \mathbb{R}, and the point c \in X. The derivative of f at c, denoted f'(c), is defined by the limit f'(c) = \lim_{x \to c} \frac{f(x) - f(c)}{x - c}, provided that this limit exists. When this limit exists, we say that f is differentiable at the point c.

If f is differentiable at every point in its domain, we simply say that it is differentiable.

Let’s use the formal definition to calculate the derivative of g(x) at x = 0, as we attempted to approximate in Figure 3.3. Remember that g(x) = x^2 / 5. Therefore, we have \begin{aligned} g'(0) &= \lim_{x \to 0} \frac{g(x) - g(0)}{x - 0} \\ &= \lim_{x \to 0} \frac{\frac{x^2}{5} - \frac{0^2}{5}}{x} \\ &= \lim_{x \to 0} \frac{x}{5} \\ &= 0. \end{aligned} This calculation confirms what we could see from the graph: the function g(x) is essentially flat at x = 0.

It would be rather tedious to go through this calculation for any individual point whose derivative we want to calculate. Luckily, we don’t have to do that. We can plug an arbitrary point c into the formula for a derivative to calculate g'(c): \begin{aligned} g'(c) &= \lim_{x \to c} \frac{g(x) - g(c)}{x - c} \\ &= \lim_{x \to c} \frac{\frac{x^2}{5} - \frac{c^2}{5}}{x - c} \\ &= \lim_{x \to c} \frac{x^2 - c^2}{5 (x - c)} \\ &= \lim_{x \to c} \frac{(x + c) (x - c)}{5 (x - c)} \\ &= \lim_{x \to c} \frac{x + c}{5} \\ &= \frac{2 c}{5}. \end{aligned} This formula confirms something we can see in Figure 3.2: g(x) gets steeper as we go further to the right along the x-axis. For example, the effective “slope” of the function at x = 1 is g'(x) = 2 / 5 = 0.4, whereas the effective slope at x = 5 is g'(x) = 2.

Exercise 3.2 Take the other function plotted in Figure 3.2, h(x) = 4 - \frac{(x - 5)^2}{5}. Show that h'(x) = - \frac{2}{5} (x - 5) = 2 - \frac{2 x}{5}. Confirm that h'(x) decreases as x increases, then find the point in the domain at which g becomes steeper than h.

Answer

For any real number c, we have \begin{aligned} h'(c) &= \lim_{x \to c} \frac{h(x) - h(c)}{x - c} \\ &= \lim_{x \to c} \frac{[4 - \frac{(x - 5)^2}{5}] - [4 - \frac{(c - 5)^2}{5}]}{x - c} \\ &= \lim_{x \to c} \frac{(c - 5)^2 - (x - 5)^2}{5 (x - c)} \\ &= \lim_{x \to c} \frac{(c^2 - 10 c + 25) - (x^2 - 10 x + 25)}{5 (x - c)} \\ &= \lim_{x \to c} \frac{c^2 - 10 c - x^2 + 10 x}{5 (x - c)} \\ &= \lim_{x \to c} \left[\frac{10 (x - c)}{5 (x - c)} - \frac{x^2 - c^2}{5 (x - c)}\right] \\ &= \lim_{x \to c} \left[2 - \frac{(x + c) (x - c)}{5 (x - c)}\right] \\ &= \lim_{x \to c} \left[2 - \frac{x + c}{5}\right] \\ &= 2 - \frac{2 c}{5}. \end{aligned}

To confirm that h'(x) decreases as x increases, suppose x < y. We have h'(y) - h'(x) = \left[2 - \frac{2y}{5}\right] - \left[2 - \frac{2x}{5}\right] = \frac{2 (x - y)}{5} < 0 and thus h'(y) < h'(x).

Finally, let’s find the crossover point at which g becomes steeper than h, i.e., at which g'(x) > h'(x). We know that g'(x) = \frac{2x}{5} and that h'(x) = 2 - \frac{2x}{5}. The statement g'(x) > h'(x) is therefore equivalent to \frac{2x}{5} > 2 - \frac{2x}{5}, which in turn is equivalent to \frac{4x}{5} > 2. This statement in turn is equivalent to x > 2 \cdot \frac{5}{4} = \frac{10}{4} = 2.5, so the crossover point is x = 2.5.

Exercise 3.3 Let f : \mathbb{R} \to \mathbb{R} be a quadratic function, meaning there exist real numbers a, b, and c such that f(x) = a x^2 + b x + c. Show that f'(x) = 2 a x + b.

Hint: The calculations will be very similar to the ones from Exercise 3.2, as the function h(x) there is itself a quadratic function.

Answer

For any real number d, we have \begin{aligned} f'(d) &= \lim_{x \to d} \frac{f(x) - f(d)}{x - d} \\ &= \lim_{x \to d} \frac{[a x^2 + b x + c] - [a d^2 + b d + c]}{x - d} \\ &= \lim_{x \to d} \frac{a (x^2 - d^2) + b (x - d)}{x - d} \\ &= \lim_{x \to d} \frac{a (x + d) (x - d) + b (x - d)}{x - d} \\ &= \lim_{x \to d} [a (x + d) + b] \\ &= 2 a d + b. \end{aligned}

The many ways to denote derivatives

You will sometimes see the derivative written in the fractional form, \frac{d f(x)}{d x}. I don’t love this notation because it makes it hard to denote “the derivative of f at the specific point x = c.” You’re stuck with either the ambiguous-seeming \frac{d f(c)}{d x} or with the clunky-seeming \left. \frac{d f(x)}{d x} \right|_{x = c}.

Economists are sometimes fond of using subscripts to denote derivatives, writing f_x(c) to denote the derivative of f at the specific point x = c. They do this more commonly for functions with multiple arguments, as we’ll see when we get to multivariable calculus. In my scholarship I only use this convention in the most dire of notational circumstances, and I will never use it in this course.

3.2.1 Alternative definition

There’s another way to define the derivative that is sometimes more convenient to work with. In this alternative definition, we define the “rise” in terms of f(c + h) - f(c) and the “run” in terms of the increment h, which may be positive or negative. We then consider the limit as this increment becomes smaller and smaller in magnitude: f'(c) = \lim_{h \to 0} \frac{f(c + h) - f(c)}{h}. To be clear, both definitions will always lead you to the same answer—which one you use is ultimately a matter of which way you find easiest to solve the problem at hand.

Proposition 3.1 (Alternative definition of derivative) Consider the function f : X \to \mathbb{R}, where X \subseteq \mathbb{R}, and the point c \in X. The derivative f'(c) exists and is equal to \lim_{h \to 0} \frac{f(c + h) - f(c)}{h} if and only if this limit exists.

As an example, we can use the alternative definition to calculate the derivative of the function g(x) plotted in Figure 3.2: \begin{aligned} g'(c) &= \lim_{h \to 0} \frac{g(c + h) - g(c)}{h} \\ &= \lim_{h \to 0} \frac{\frac{(c + h)^2}{5} - \frac{c^2}{5}}{h} \\ &= \lim_{h \to 0} \frac{(c + h)^2 - c^2}{5 h} \\ &= \lim_{h \to 0} \frac{(c^2 + 2 c h + h^2) - c^2}{5h} \\ &= \lim_{h \to 0} \frac{2 c h + h^2}{5 h} \\ &= \lim_{h \to 0} \frac{2 c + h}{5} \\ &= \frac{2 c}{5}. \end{aligned} I personally find this definition easier for this particular problem. You might find the other definition easier to work with. Either way, we end up in the same place, with g'(c) = 2c / 5.

Exercise 3.4 Redo Exercise 3.3 using the alternative formula for a derivative.

Answer

Let f(x) = a x^2 + b x + c, and consider any real number d. We have \begin{aligned} f'(d) &= \lim_{h \to 0} \frac{f(d + h) - f(d)}{h} \\ &= \lim_{h \to 0} \frac{[a (d + h)^2 + b (d + h) + c] - [a d^2 + b d + c]}{h} \\ &= \lim_{h \to 0} \frac{[a d^2 + 2 a d h + a h^2 + b d + b h + c] - [a d^2 + b d + c]}{h} \\ &= \lim_{h \to 0} \frac{2 a d h + a h^2 + b h}{h} \\ &= \lim_{h \to 0} [2 a d + a h + b] \\ &= 2 a d + b. \end{aligned}

3.2.2 When the derivative doesn’t exist

The derivative is the limit of the rise-over-run formula. Because the denominator of that formula approaches zero as we take the limit, its limit is not always guaranteed to exist—and in fact sometimes it does not.

To begin with, you never need to bother trying to calculate the derivative of a function at a point where the function is not continuous—it won’t exist. In logic-speak: if f is not continuous at c, then f'(c) does not exist. Or, equivalently (never forget the contrapositive!), if f'(c) exists, then f is continuous at c.

Theorem 3.1 (Differentiable implies continuous) Consider a function f : X \to \mathbb{R}, where X \subseteq \mathbb{R}, and the point c \in X. If f is differentiable at c, then f is continuous at c.

Proof. Suppose f is differentiable at c, meaning there is a real number y such that \lim_{x \to c} \frac{f(x) - f(c)}{x - c} = y. Using the properties of function limits (Proposition 2.3), we then have \begin{aligned} \lim_{x \to c} f(x) &= \lim_{x \to c} [f(x) - f(c) + f(c)] \\ &= f(c) + \lim_{x \to c} [f(x) - f(c)] \\ &= f(c) + \lim_{x \to c} \left[\frac{f(x) - f(c)}{x - c} \cdot (x - c)\right] \\ &= f(c) + \underbrace{\left(\lim_{x \to c} \frac{f(x) - f(c)}{x - c}\right)}_{ = y} \cdot \underbrace{\left(\lim_{x \to c} [x - c]\right)}_{= 0} \\ &= f(c) + y \cdot 0 \\ &= f(c), \end{aligned} so f is continuous at c.

Theorem 3.1 tells us that every differentiable function is continuous, but it says nothing about whether every continuous function is differentiable. (In terms of formal logic, the theorem is an “if” statement rather than an “if and only if.”) In fact, there are continuous functions that are not differentiable.

One example of a function that is continuous but not differentiable is the absolute value function, f(x) = |x| at the point c = 0. The derivative does not exist because the relevant left- and right-hand limits do not agree: \begin{alignedat}{2} \lim_{x \to 0^-} \frac{f(x) - f(0)}{x - 0} &= \lim_{x \to 0^-} \frac{-x}{x} &&= -1; \\ \lim_{x \to 0^+} \frac{f(x) - f(0)}{x - 0} &= \lim_{x \to 0^+} \frac{x}{x} &&= 1. \end{alignedat}

The most common type of non-differentiability in a continuous function that you’ll run into is a “kink” in the graph of the function—a point where the direction of the function appears to change sharply instead of smoothly. The absolute value function illustrates this type of non-differentiability.

Figure 3.4: Non-differentiability due to a kink.

Another source of non-differentiability is when the function is so steep that its graph approaches a vertical line. As an example, take the square root function, f(x) = \sqrt{x}, evaluated at the point c = 0. The “rise over run” limit here becomes infinitely large as x \to 0: \begin{aligned} \lim_{x \to 0} \frac{f(x) - f(0)}{x - 0} &= \lim_{x \to 0} \frac{\sqrt{x}}{x} \\ &= \lim_{x \to 0} \frac{1}{\sqrt{x}} \\ &= \infty. \end{aligned}

Figure 3.5: Non-differentiability due to vertical slope.

3.3 Taking derivatives in practice

I take derivatives all the time in my day job as a formal theorist, yet only rarely do I find myself explicitly taking the limit from Definition 3.1. Instead, I rely on some common derivatives and some useful rules to break down complex-seeming problems.

3.3.1 Derivatives of common functions

Many of the functions that commonly arise in statistics and game theory have derivatives of known forms. Here are the most important ones you need to know.

Linear functions: Any function of the form f(x) = \alpha + \beta x, where \alpha and \beta are real numbers, has a derivative of \beta at all points in its domain: f'(x) = \beta.
Power functions: Any function of the form f(x) = x^a, where a is a real number, has a derivative of f'(x) = a x^{a - 1}.
Natural exponent: The function f(x) = e^x, where e is Euler’s number (approximately 2.718), has a derivative of f'(x) = e^x. That’s not a typo—the natural exponent is its own derivative. That is one of the many reasons why Euler’s number is special.
Natural logarithm: The function f(x) = \log x has a derivative of f'(x) = 1/x.

Exercise 3.5 (Derivative of a linear function) Using one of the formal definitions of a derivative, confirm that the derivative of a linear function is \beta at all points in its domain. Why is this sensible?

Answer

To confirm the claim, for any point c in the domain we have \begin{aligned} \lim_{x \to c} \frac{f(x) - f(c)}{x - c} &= \lim_{x \to c} \frac{(\alpha + \beta x) - (\alpha + \beta c)}{x - c} \\ &= \lim_{x \to c} \frac{\beta x - \beta c}{x - c} \\ &= \lim_{x \to c} \frac{\beta (x - c)}{x - c} \\ &= \lim_{x \to c} \beta \\ &= \beta. \end{aligned} This is sensible because we calculate the derivative by using the rise-over-run formula, and the defining characteristic of a linear function is that the rise-over-run calculation is the same no matter what two points on the domain we’re using to calculate it.

Exercise 3.6 (Derivative of a constant function) A constant function is a function of the form f(x) = y, where y is a constant real number. Without taking an explicit limit, how do you know from the above rules that f'(x) = 0 for all x?

Answer

A constant function is a linear function with slope 0: f(x) = y = y + 0x. Because the derivative of any linear function is its slope, we have f'(x) = 0.

3.3.2 Properties of derivatives

Using a few rules in combination with the derivatives of common functions from above, we can calculate the derivatives of most functions we run into in practice without explicitly taking limits of the rise-over-run ratio.

If we take a function and multiply it by a constant, the derivative of the resulting function is the same multiple of the original derivative. This is the constant multiple rule.

Proposition 3.2 (Constant multiple rule) Let f : X \to \mathbb{R} be differentiable, where X \subseteq \mathbb{R}. Define the function g : X \to \mathbb{R} by g(x) = c f(x), where c is a constant real number. For all x \in X, g'(x) = c f'(x).

Proof. For any x \in X, we have \begin{aligned} \lim_{h \to 0} \frac{g(x + h) - g(x)}{h} &= \lim_{h \to 0} \frac{c f(x + h) - c f(x)}{h} \\ &= \lim_{h \to 0} \left[c \cdot \frac{f(x + h) - f(x)}{h}\right] \\ &= c \lim_{h \to 0} \frac{f (x + h) - f(x)}{h} \\ &= c f'(x), \end{aligned} and therefore g'(x) = c f'(x).

For example, we can use the constant multiple rule to find the derivative of any logarithmic function, not just the natural logarithm. Let f(x) = \log_b(x), where b > 0 and b \neq 1. Using the base change formula from Proposition 2.9, we know that f(x) = \log_b(x) = \frac{\log(x)}{\log(b)} = \frac{1}{\log(b)} \cdot \log(x). In other words, the logarithm with base b is just a constant multiple of the natural logarithm. Using this fact, we can find the derivative: f'(x) = \frac{1}{\log(b)} \cdot \frac{d \log(x)}{d x} = \frac{1}{\log(b)} \cdot \frac{1}{x} = \frac{1}{x \log(b)}.

Another convenient property is that the derivative of the sum is equal to the sum of the derivatives. You’ll end up using the sum rule so often that you stop thinking about it.

Proposition 3.3 (Sum rule) Let f : X \to \mathbb{R} and g : X \to \mathbb{R} be differentiable, where X \subseteq \mathbb{R}. Define the function h : X \to \mathbb{R} by h(x) = f(x) + g(x). For all x \in X, h'(x) = f'(x) + g'(x).

Proof. For any x \in X, we have \begin{aligned} \lim_{t \to 0} \frac{h(x + t) - h(x)}{t} &= \lim_{t \to 0} \frac{[f(x + t) + g(x + t)] - [f(x) + g(x)]}{t} \\ &= \lim_{t \to 0} \left[\frac{f(x + t) - f(x)}{t} + \frac{g(x + t) - g(x)}{t}\right] \\ &= \lim_{t \to 0} \frac{f(x + t) - f(x)}{t} + \lim_{t \to 0} \frac{g(x + t) - g(x)}{t} \\ &= f'(x) + g'(x), \end{aligned} and therefore h'(x) = f'(x) + g'(x).

I’ve stated the sum rule just for a sum of two functions. In fact, it applies with equal force to any sum of a finite number of functions. For any collection of differentiable functions f_1, f_2, \ldots, f_n, their sum h(x) = \sum_{i=1}^n f_i(x) has the derivative h'(x) = \sum_{i=1}^n f_i'(x). \tag{3.2}

Exercise 3.7 (Derivative of a linear combination of functions) Let f : X \to \mathbb{R} and g : X \to \mathbb{R} be differentiable, where X \subseteq \mathbb{R}. Let a \in \mathbb{R} and b \in \mathbb{R} be constants. Define the function h : X \to \mathbb{R} by h(x) = a f(x) + b g(x). Use the constant rule and the sum rule to show that h'(x) = a f'(x) + b g'(x).

Answer

This is one of those cases where it’s convenient to use the fractional notation for a derivative. We have \begin{aligned} \frac{d h(x)}{d x} &= \frac{d}{dx} \left[a f(x) + b g(x)\right] \\ &= \frac{d}{dx} \left[a f(x)\right] + \frac{d}{dx} \left[b g(x)\right] \\ &= a \frac{d f(x)}{d x} + b \frac{d g(x)}{d x}, \end{aligned} where the first equality uses our definition of h(x), the second uses the sum rule, and the third uses the constant multiple rule twice.

Exercise 3.8 (Derivative of a polynomial) Let f : X \to \mathbb{R} be a polynomial, so that \begin{aligned} f(x) = c_n x^n + c_{n-1} x^{n-1} + \cdots + c_1 x + c_0 = \sum_{k=0}^n c_k x^k \end{aligned} for some natural number n and real-valued coefficients c_0, \ldots, c_n. Use the constant multiple rule, the derivative of a power function, and Equation 3.2 to prove that \begin{aligned} f'(x) = n c_n x^{n - 1} + (n - 1) c_{n-1} x^{n-2} + \cdots + c_1 = \sum_{k=1}^n k c_k x^{k-1}. \end{aligned}

Answer

Using Equation 3.2, we have \begin{aligned} f'(x) = \sum_{k=0}^n \frac{d}{dx} \left[c_k x^{k}\right]. \end{aligned} Using the constant multiple rule, we then have \begin{aligned} f'(x) = \sum_{k=0}^n c_k \frac{d}{dx} \left[x^{k}\right]. \end{aligned} Finally, using the derivative of a power function, we have \begin{aligned} f'(x) = \sum_{k=0}^n c_k \left[k x^{k-1}\right]. \end{aligned} We can take the k = 0 term out of the sum, \begin{aligned} f'(x) &= \sum_{k=0}^n c_k \left[k x^{k-1}\right] \\ &= \underbrace{0 \cdot c_0 \cdot x^{-1}}_{= 0} + \sum_{k=1}^n k c_k x^{k-1} \\ &= \sum_{k=1}^n k c_k x^{k-1}, \end{aligned} proving the claim.

Unfortunately for us, derivatives of products don’t work quite as intuitively as derivatives of sums. In general, it is not true that the derivative of f(x) g(x) equals f'(x) g'(x).

As an illustration, think about the quadratic function h(x) = x^2 = x \cdot x. If it were true that the derivative of the product equals the product of the derivatives, then we would have h(x) = 1 \cdot 1 = 1, falsely implying that the quadratic function is actually linear.

All that said, we still have a convenient formula to calculate the derivative of a product—it’s just not as convenient as the derivative of a sum. The product rule to take the derivative of f(x) g(x) essentially breaks the problem into two parts. First we treat f(x) as if it were a constant and take the derivative of g(x), then we do the reverse, and finally we add the two results together.

Proposition 3.4 (Product rule) Let f : X \to \mathbb{R} and g : X \to \mathbb{R} be differentiable, where X \subseteq \mathbb{R}. Define the function h : X \to \mathbb{R} by h(x) = f(x) \cdot g(x). For all x \in X, \begin{aligned} h'(x) = f(x) g'(x) + f'(x) g(x). \end{aligned}

Proof. Take any x \in X. Differentiability of f implies continuity of f (per Theorem 3.1) and thus \lim_{t \to 0} f(x + t) = f(x). Consequently, \begin{aligned} & \lim_{t \to 0} \frac{h(x + t) - h(x)}{t} \\ &= \lim_{t \to 0} \frac{f(x + t) g(x + t) - f(x) g(x)}{t} \\ &= \lim_{t \to 0} \frac{f(x + t) g(x + t) - f(x + t) g(x) + f(x + t) g(x) - f(x) g(x)}{t} \\ &= \lim_{t \to 0} \frac{f(x + t) [g(x + t) - g(x)]}{t} + \lim_{t \to 0} \frac{[f(x + t) - f(x)] g(x)}{t} \\ &= \left[\lim_{t \to 0} f(x + t)\right] \cdot \left[\lim_{t \to 0} \frac{g(x + t) - g(x)}{t}\right] + \left[\lim_{t \to 0} \frac{f(x + t) - f(x)}{t}\right] \cdot g(x) \\ &= f(x) g'(x) + f'(x) g(x), \end{aligned} and therefore h'(x) = f(x) g'(x) + f'(x) g(x).

Going back to our example of the quadratic function h(x) = x^2, proper use of the product rule confirms that h'(x) = 2x. Let f(x) = g(x) = x, so that f'(x) = g'(x) = 1 and h(x) = f(x) \cdot g(x). The product rule then gives us \begin{aligned} h'(x) &= f(x) g'(x) + f'(x) g(x) \\ &= x \cdot 1 + 1 \cdot x \\ &= 2x. \end{aligned}

Now let’s get some more practice with the product rule through some exercises.

Exercise 3.9 (Product rule applications) Find the derivatives of the following functions.

f(x) = (2x + 1) (x - 3).
f(x) = x e^x.
f(x) = e^{2x}.
f(x) = e^x \cdot \log(x) \cdot x^3.

Answers

You could solve this one by observing that f(x) = 2x^2 - 5x - 3, then applying the formula for the derivative of a polynomial (see Exercise 3.8) to yield f'(x) = 4 x - 5. We get to the same place if we instead use the product rule: \begin{aligned} f'(x) &= (2x + 1) \cdot 1 + 2 \cdot (x - 3) \\ &= 2x + 1 + 2x - 6 \\ &= 4x - 5. \end{aligned}
Let g(x) = x so that g'(x) = 1, and let h(x) = e^x so that h'(x) = e^x. We have \begin{aligned} f'(x) &= g(x) h'(x) + g'(x) h(x) \\ &= x \cdot e^x + 1 \cdot e^x \\ &= (x + 1) e^x. \end{aligned}
Let g(x) = h(x) = e^x so that g'(x) = h'(x) = e^x. We have \begin{aligned} f'(x) &= g(x) h'(x) + g'(x) h(x) \\ &= e^x \cdot e^x + e^x \cdot e^x \\ &= e^{2x} + e^{2x} \\ &= 2 e^{2x}. \end{aligned}
This one makes us use the product rule twice. We have \begin{aligned} f'(x) &= \frac{d}{dx} \left[e^x \cdot \log(x) \cdot x^3\right] \\ &= e^x \cdot \left[\frac{d}{dx} \left(\log(x) \cdot x^3\right)\right] + \left[\frac{d}{dx} e^x\right] \cdot \left[\log(x) \cdot x^3\right] \\ &= e^x \cdot \left[\log(x) \cdot \left(\frac{d}{dx} x^3\right) + \left(\frac{d}{dx} \log(x)\right) \cdot x^3\right] \\ &\quad + \left[\frac{d}{dx} e^x\right] \cdot \left[\log(x) \cdot x^3\right] \\ &= e^x \cdot \left[\log(x) \cdot (3 x^2) + \left(\frac{1}{x}\right) \cdot x^3\right] + \left[e^x \cdot \log(x) \cdot x^3\right] \\ &= 3 e^x \cdot \log(x) \cdot x^2 + e^x \cdot x^2 + e^x \cdot \log(x) \cdot x^3. \end{aligned} This example is an illustration of a generalization of the product rule: \begin{aligned} &\frac{d}{dx} [f(x) \cdot g(x) \cdot h(x)] \\&\qquad = f(x) g(x) h'(x) + f(x) g'(x) h(x) + f'(x) g(x) h(x). \end{aligned} I’ll leave it to you to prove that general claim.

Exercise 3.10 (Derivative of the square of a function) Let f : X \to \mathbb{R} be differentiable, where X \subseteq \mathbb{R}. Define the function g : X \to \mathbb{R} by g(x) = f(x)^2. Using the product rule, prove that g'(x) = 2 f'(x) f(x).

Answer

We have g(x) = f(x) \cdot f(x), so the product rule implies g'(x) = f(x) f'(x) + f'(x) f(x) = 2 f'(x) f(x).

The quotient rule for functions in the form \frac{f(x)}{g(x)} is unfortunately a bit trickier to remember. If you have to Google this one every time you use it, don’t worry — you’re not alone.

Proposition 3.5 (Quotient rule) Let f : X \to \mathbb{R} and g : X \to \mathbb{R} \setminus \{0\} be differentiable, where X \subseteq \mathbb{R}. Define the function h : X \to \mathbb{R} by h(x) = \frac{f(x)}{g(x)}. For all x \in X, h'(x) = \frac{f'(x) g(x) - f(x) g'(x)}{g(x)^2}.

I’m not going to prove the quotient rule here—you’ll do that yourself once you’ve got the chain rule in hand. Before then, let’s work through some practice with the quotient rule. If you use it enough, maybe you’ll actually start to remember it?

Exercise 3.11 (Contest success function) A helpful tool in formal models of conflict is the contest success function, whose most common form is f(x) = \frac{\alpha x}{\alpha x + y}. You can think of x here as the amount of resources that I’m devoting toward the conflict, y > 0 as the amount of resources that my enemy has devoted, and \alpha > 0 as the “force multiplier” on my effort (how much bang for the buck I’m getting). The output of the function represents my probability of winning the conflict, as a function of the amount of resources I devote.

Use the quotient rule to derive f'(x).
The derivative f'(x) represents approximately how much my chance of victory increases with each unit of resources I expend. Imagine that I don’t want to put in a unit of resources that gets me less than a 1% increase in the chance of victory. Then what’s the largest value of x that I’d be willing to choose? (The answer will depend on the values of \alpha and y.)

Answer

Think of f(x) = g(x) / h(x) where g(x) = \alpha x and h(x) = \alpha x + y. Then, using the quotient rule, we have \begin{aligned} f'(x) &= \frac{g'(x) h(x) - g(x) h'(x)}{h(x)^2} \\ &= \frac{\alpha (\alpha x + y) - (\alpha x) \alpha}{(\alpha x + y)^2} \\ &= \frac{\alpha y}{(\alpha x + y)^2}. \end{aligned}

The second question essentially asks us to find the value of x that solves f'(x) = 0.01: \begin{aligned} f'(x) = 0.01 &\quad\Leftrightarrow\quad \frac{\alpha y}{(\alpha x + y)^2} = 0.01 \\ &\quad\Leftrightarrow\quad (\alpha x + y)^2 = 100 \alpha y \\ &\quad\Leftrightarrow\quad \alpha x + y = 10 \sqrt{\alpha y} \\ &\quad\Leftrightarrow\quad x = \frac{10 \sqrt{\alpha y} - y}{\alpha}. \end{aligned}

Exercise 3.12 (Derivative of a conditional probability) Let X be a random variable that has a binomial distribution with n = 2 and probability parameter p, so that we have \begin{aligned} \Pr(X = 0) &= (1 - p)^2, \\ \Pr(X = 1) &= 2 p (1 - p), \\ \Pr(X = 2) &= p^2. \end{aligned} The conditional probability \Pr(X = 2 \mid X \geq 1) is equal to \begin{aligned} \Pr(X = 2 \mid X \geq 1) = \frac{\Pr(X = 2)}{\Pr(X = 1) + \Pr(X = 2)} = \frac{p^2}{2 p (1 - p) + p^2}. \end{aligned} Taking this conditional probability to be a function of the parameter p, find its derivative and show that the derivative is positive for all p \in (0, 1].

Answer

Defining the function f(p) = \frac{p^2}{2p (1 - p) + p^2} = \frac{p^2}{2p - p^2}, we have \begin{aligned} f'(p) &= \frac{2p (2p - p^2) - p^2 (2 - 2p)}{(2p - p^2)^2} \\ &= \frac{4 p^2 - 2 p^3 - 2 p^2 + 2 p^3}{(2p - p^2)^2} \\ &= \frac{p^2}{(2p - p^2)^2}. \end{aligned} We have p^2 > 0 and (2p - p^2)^2 > 0 for all p \neq 0, and therefore f'(p) > 0 for all p \in (0, 1].

3.4 The chain rule

What do we do when we run into a function that isn’t obviously a sum or product or quotient of some other function whose derivative we know how to take? For example, think about the function h(x) = e^{-(x - \mu)^2}, a close cousin of the probability density function of the normal distribution with mean \mu and variance 1. We know that the derivative of e^x is e^x. And by observing that -(x - \mu)^2 = -[x^2 - 2 x \mu + \mu^2] = 2 x \mu - x^2 - \mu^2, we know that its derivative is 2 (\mu - x). Is there a way we can put these together to figure out the derivative of h(x)?

The chain rule helps us deal with situations like this one. Specifically, suppose we have a function of the form h(x) = f(g(x)), aka the composition (Definition 1.11) of f and g, where f and g are both differentiable. In our example, we can decompose h into functions f and g whose derivatives we know. We’ll decompose it using \begin{aligned} f(x) &= e^x; \\ g(x) &= -(x - \mu)^2, \end{aligned} which we can confirm is a valid decomposition via f(g(x)) = e^{g(x)} = e^{-(x - \mu)^2} = h(x).

Proposition 3.6 (Chain rule) Let f : Y \to \mathbb{R} and g : X \to Y be differentiable, where X \subseteq \mathbb{R} and Y \subseteq \mathbb{R}. Define the composition h by h(x) = f(g(x)). For all x \in X, h'(x) = f'(g(x)) \cdot g'(x).

Using the chain rule with our example of h(x) = e^{-(x - \mu)^2}, we can calculate the derivative: \begin{aligned} h'(x) &= f'(g(x)) \cdot g'(x) \\ &= f'(-(x - \mu)^2) \cdot g'(x) \\ &= e^{-(x - \mu)^2} \cdot 2 (\mu - x) \\ &= 2 (\mu - x) e^{-(x - \mu)^2}. \end{aligned} If we look at the graph of this function, we see that it increases on (-\infty, \mu) and decreases on (\mu, \infty).

Figure 3.6: A function shaped like a bell curve

Our expression for the derivative provides further confirmation that the function increases up to x = \mu and decreases thereafter. The key thing to observe is that e^z > 0 for all real numbers z. Therefore, if x < \mu, then we have h'(x) = 2 \underbrace{(\mu - x)}_{> 0} \underbrace{e^{-(x - \mu)^2}}_{> 0} > 0. If instead x > \mu, then we have h'(x) = 2 \underbrace{(\mu - x)}_{< 0} \underbrace{e^{-(x - \mu)^2}}_{> 0} < 0.

Exercise 3.13 (Derivative of an exponential function) Let b be a positive constant, and define the function f(x) = b^x. Using the chain rule, show that f'(x) = b^x \cdot \log(b).

Hint: Start out by using the fact that b = e^{\log(b)}.

Answer

We have f(x) = b^x = (e^{\log(b)})^x = e^{\log(b) \cdot x}. Define the “inner” function h(x) = \log(b) \cdot x (with derivative h'(x) = \log(b)) and the “outer” function g(x) = e^x (with derivative g'(x) = e^x), so that we have f(x) = e^{\log(b) \cdot x} = e^{h(x)} = g(h(x)). Then, using the chain rule, we have \begin{aligned} f'(x) &= g'(h(x)) \cdot h'(x) \\ &= g'(\log(b) \cdot x) \cdot h'(x) \\ &= e^{\log(b) \cdot x} \cdot \log(b) \\ &= b^x \cdot \log(b). \end{aligned}

Exercise 3.14 (Derivative of the exponential CDF) For all x \geq 0, the cumulative distribution function of the exponential distribution with rate parameter \lambda > 0 is F(x) = 1 - e^{-\lambda x}. Characterize the probability density function of this distribution by finding the derivative of the cumulative distribution function.

Answer

Let’s first just look at e^{-\lambda x}. We can think of this as a composition of the “outer” function g(x) = e^x (with derivative g'(x) = e^x) and the “inner” function h(x) = -\lambda x (with derivative h'(x) = -\lambda). Therefore, its derivative is g'(h(x)) \cdot h'(x) = g'(-\lambda x) \cdot h'(x) = e^{-\lambda x} \cdot -\lambda.

Returning to the full expression for the CDF and applying the sum rule, we have \begin{aligned} F'(x) &= \frac{d}{dx} \left[1 - e^{-\lambda x}\right] \\ &= \left[\frac{d}{dx} (1) - \frac{d}{dx} \left(e^{-\lambda x}\right) \right] \\ &= \left[0 - e^{-\lambda x} \cdot -\lambda\right] \\ &= \lambda e^{-\lambda x}. \end{aligned}

Exercise 3.15 (Derivative of the logistic CDF) For all x \in \mathbb{R}, the cumulative distribution function of the logistic distribution is F(x) = \frac{1}{1 + e^{-x}}. Characterize the probability density function of this distribution by finding the derivative of the cumulative distribution function.

Answer

We have F(x) = (1 + e^{-x})^{-1}. By repeatedly applying the chain rule, we have \begin{aligned} F'(x) &= (-1) (1 + e^{-x})^{-2} \cdot \frac{d}{dx} [1 + e^{-x}] \\ &= -(1 + e^{-x})^{-2} \cdot \frac{d}{dx} e^{-x} \\ &= -(1 + e^{-x})^{-2} \cdot e^{-x} \cdot \frac{d}{dx} [-x] \\ &= -(1 + e^{-x})^{-2} \cdot e^{-x} \cdot -1 \\ &= \frac{e^{-x}}{(1 + e^{-x})^2}. \end{aligned}

Exercise 3.16 (Derivative of the logarithm of a function) Let f be a differentiable function that maps from X \subseteq \mathbb{R} into the strictly positive reals, (0, \infty). Define the function g(x) = \log [f(x)]. Prove that g'(x) = \frac{f'(x)}{f(x)}.

Answer

g is the composition of the natural logarithm and f. Therefore, by the chain rule, \begin{aligned} g'(x) &= \log' [f(x)] \cdot f'(x) \\ &= \frac{1}{f(x)} \cdot f'(x) \\ &= \frac{f'(x)}{f(x)}. \end{aligned}

Exercise 3.17 (Proving the quotient rule) Use the product rule and the chain rule to prove the quotient rule.

Hint: Start with the fact that \frac{f(x)}{g(x)} = f(x) \cdot g(x)^{-1}.

Answer

Define h(x) = \frac{f(x)}{g(x)} = f(x) \cdot g(x)^{-1}. By the product rule, \begin{aligned} h'(x) &= f(x) \cdot \frac{d}{dx} \left[g(x)^{-1}\right] + f'(x) \cdot g(x)^{-1}. \end{aligned} Using the chain rule, we have \begin{aligned} \frac{d}{dx} \left[g(x)^{-1}\right] &= (-1) g(x)^{-2} \cdot g'(x) \\ &= \frac{-g'(x)}{g(x)^2}. \end{aligned} Substituting this into the product rule calculation from before, we have \begin{aligned} h'(x) &= f(x) \cdot \frac{d}{dx} \left[g(x)^{-1}\right] + f'(x) \cdot g(x)^{-1} \\ &= f(x) \cdot \frac{-g'(x)}{g(x)^2} + f'(x) \cdot \frac{1}{g(x)} \\ &= \frac{- f(x) g'(x)}{g(x)^2} + \frac{f'(x)}{g(x)} \\ &= \frac{- f(x) g'(x)}{g(x)^2} + \frac{f'(x) g(x)}{g(x)^2} \\ &= \frac{f'(x) g(x) - f(x) g'(x)}{g(x)^2}. \end{aligned}

So why does the chain rule work? Here’s one oversimplified way to think about it. Consider a function h that’s a composition of differentiable functions f and g, so that h(x) = f(g(x)). Imagine starting at some value c in the domain, then moving some small amount \Delta to the right along the x-axis. What happens to the value of h(x) as we move from x = c to x = c + \Delta?

First let’s trace what happens to the value of the “inner” function, g(x). If g'(c) > 0, so g is increasing near x = c, then we’d expect g(c + \Delta) > g(c); we’d expect the opposite if g'(c) < 0. More to the point, since we’re taking \Delta to be small—close to 0—we can use the derivative at x = c to approximate g(c + \Delta): \begin{aligned} g(c + \Delta) &= g(c) + g(c + \Delta) - g(c) \\ &= g(c) + \Delta \cdot \frac{g(c + \Delta) - g(c)}{\Delta} \\ &\approx g(c) + \Delta \cdot \lim_{d \to 0} \frac{g(c + d) - g(c)}{d} \\ &= g(c) + \Delta g'(c). \end{aligned}

Now let’s move to the “outer” function, f(y). In general, for a shift of size \epsilon for the input to f, we have f(y + \epsilon) \approx f(y) + \epsilon f'(y), following the same approximation logic we just went through. In this case, the original input is y = g(c), which is shifting by approximately \epsilon = \Delta g'(c). Putting everything together, the value of the composition at x = c + \Delta is approximately \begin{aligned} h(c + \Delta) &= f(g(c + \Delta)) \\ &\approx f(g(c) + \Delta g'(c)) \\ &\approx f(g(c)) + (\Delta g'(c)) \cdot f'(g(c)) \\ &= h(c) + \Delta \cdot \underbrace{f'(g(c)) \cdot g'(c)}_{\text{look familiar?!}}. \end{aligned} If we plug this approximation into the definition of the derivative of the composition h at c, we just end up with the chain rule: \begin{aligned} h'(c) &= \lim_{\Delta \to 0} \frac{h(c + \Delta) - h(c)}{\Delta} \\ &\approx \lim_{\Delta \to 0} \frac{[h(c) + \Delta \cdot f'(g(c)) \cdot g'(c)] - h(c)}{\Delta} \\ &= \lim_{\Delta \to 0} \frac{\Delta \cdot f'(g(c)) \cdot g'(c)}{\Delta} \\ &= f'(g(c)) \cdot g'(c). \end{aligned}

Exercise 3.18 (Composition of two decreasing functions) Let f : \mathbb{R} \to \mathbb{R} and g : \mathbb{R} \to \mathbb{R} both be differentiable functions that are strictly decreasing. By strictly decreasing, I mean that if x < x', then g(x) > g(x'); and if y < y', then f(y) > f(y').

Consider the composition h(x) = (f \circ g)(x) = f(g(x)). First, without taking any derivatives, explain why h is a strictly increasing function of x. Then use the chain rule to confirm this.

Answer

Consider any real numbers a and b where a < b. Because g is strictly decreasing, we have g(a) > g(b). Then, because f is strictly decreasing, we have f(g(a)) < f(g(b)). This in turn means that h(a) < h(b). We’ve shown that a < b implies h(a) < h(b), which means h is strictly increasing.

To confirm this with the chain rule, notice that f strictly decreasing means f'(y) < 0 for all y, and similarly g'(x) < 0 for all x. Therefore, for all x, we have h'(x) = \underbrace{f'(g(x))}_{< 0} \cdot \underbrace{g'(x)}_{< 0} > 0, so h is strictly increasing.

3.5 Second derivatives and beyond

In economics, inflation is a measure of the rate of increase in the price level. When prices are going up, inflation is positive. The quicker the prices are going up, the higher the measure of inflation is. In the rare event that prices are going down, we’d say we have “deflation,” in which case the standard measures of inflation would be negative. Mathematically, you can think about the price level as a function of time, and then think of inflation as the derivative of this function.

Politicians will sometimes brag that inflation is decreasing. But they usually don’t mean that the price level is actually going down: that would mean we’re having deflation, which usually occurs only in the context of Great Depression-type economic cataclysms. Instead, a decrease in inflation typically means that prices are increasing, but not as quickly as they were earlier. In other words, the rate of increase is itself decreasing. But now we’re thinking about the rate of change in a rate of change … or the derivative of a derivative, also known as the second derivative.

Definition 3.2 (Second derivative) Let f : X \to \mathbb{R} be differentiable, where X \subseteq \mathbb{R}. For any domain point c \in X, the second derivative of f at c is \begin{aligned} f''(c) = \lim_{x \to c} \frac{f'(x) - f'(c)}{x - c}, \end{aligned} provided that this limit exists. When it does, we say that f is twice differentiable at the point c.

If f is twice differentiable at every point in its domain, we simply say that it is twice differentiable.

Sometimes in formal theory papers, you’ll see a function described just in terms of its first and second derivatives. For instance, I might write, “Let f : [0, 1] \to [0, 1] be twice differentiable, with f' > 0 and f'' < 0.” Here’s a little cheat sheet to interpret statements like that one.

Case	First derivative	Second derivative	Interpretation
1	Positive: f'(x) > 0	Positive: f''(x) > 0	The function is increasing ever more steeply as we move to the right on the x axis
2	Positive: f'(x) > 0	Negative: f''(x) < 0	The function is increasing, but the rate of increase is slowing down as we move to the right
3	Negative: f'(x) < 0	Positive: f''(x) > 0	The function is decreasing, but the rate of decrease is slowing down as we move to the right
4	Negative: f'(x) < 0	Negative: f''(x) < 0	The function is decreasing ever more steeply as we move to the right on the x axis

If a function has a non-negative second derivative, f''(x) \geq 0, we call it convex. Additionally, if the second derivative is strictly positive, f''(x) > 0, we call the function strictly convex. A classic example of a strictly convex function is exponential growth. For example, let f(t) be the number of mice you will have at time t if you start with one boy mouse and one girl mouse at t = 0. If the number of mice doubles with each unit of time, then you’ll have \begin{aligned} f(0) &= 2, \\ f(1) &= 4, \\ f(2) &= 8, \\ f(3) &= 16, \\ f(4) &= 32, \\ \vdots \end{aligned} So not only is the function increasing, but it’s increasing more quickly as time goes on. Because the rate of increase is itself increasing, the function is strictly convex.

On the other hand, we call a function concave if f''(x) \leq 0, and additionally strictly concave if f''(x) < 0. Concave functions often arise in economic contexts. In particular, we often think that concave functions are a good way to model the amount of “utility” (happiness, loosely speaking) that a person derives from consuming a quantity of a good.

For instance, let t be the number of tacos I’ve eaten and let u(t) be how happy I am as a function of the number of tacos I’ve eaten. There’s a big happiness boost going from 0 tacos (starving) to 1 taco (still hungry but not starving). There’s a smaller happiness boost going from 1 taco to 2 tacos (barely full). The boost keeps decreasing from there, and once I’m full I would actively prefer to not eat any additional tacos. Altogether, then, u(t) is strictly concave—each additional taco raises my happiness by a lower and lower amount.

A linear function is both convex and concave. However, it is neither strictly convex nor strictly concave. All that is a long way of saying that the second derivative of a linear function is always zero. That’s true because the derivative of a linear function is its slope, a constant, so the derivative of the derivative is zero.

Exercise 3.19 (Concave and convex functions) Using the second derivative, characterize the conditions under which each of the following functions is convex or concave:

A quadratic function of the form f(x) = a x^2 + b x + c.
An exponential function of the form f(x) = b^x, where b > 0.
A logarithmic function of the form f(x) = \log_b x, where b > 0 and b \neq 1.

Answer

We have f'(x) = 2 a x + b and thus f''(x) = 2a. So the function is concave if a \leq 0 and convex if a \geq 0.
We have f'(x) = b^x \cdot \log(b) and thus f''(x) = b^x \cdot (\log(b))^2. If b = 1, then we have \log(b) = 0 and the function is linear (both concave and convex). Otherwise, we have \log(b) \neq 0, hence (\log(b))^2 > 0, hence b^x \cdot (\log(b))^2 > 0, so the function is strictly convex.
We have f'(x) = \frac{1}{x \log(b)} = \frac{1}{\log(b)} \cdot x^{-1} and thus f''(x) = \frac{1}{\log(b)} \cdot \left[-1 \cdot x^{-2}\right] = \frac{-1}{x^2 \log(b)}. The sign of the second derivative is the same as that of \log(b), so the function is convex if b \in (0, 1) and is concave if b \in (1, \infty).

Exercise 3.20 (Concave portion of the normal PDF) A function can be concave on part of its domain and convex on other parts. For example, take the probability density function of the standard normal distribution, f(x) = \frac{1}{\sqrt{2 \pi}} e^{-x^2 / 2}.

You can see from the graph that the standard normal PDF is concave on [-1, 1], while it is convex on the rest of its domain. Use its second derivative to prove this.

Answer

Using the constant multiple rule and the chain rule, we have \begin{aligned} f'(x) &= \frac{d}{dx} \left[\frac{1}{\sqrt{2 \pi}} e^{-x^2 / 2}\right] \\ &= \frac{1}{\sqrt{2 \pi}} \frac{d}{dx} \left[e^{-x^2/2}\right] \\ &= \frac{1}{\sqrt{2 \pi}} \left[e^{-x^2 / 2} \cdot \frac{-2x}{2}\right] \\ &= \frac{-1}{\sqrt{2 \pi}} \cdot x e^{-x^2/2}. \end{aligned} We can then use the constant multiple rule and the product rule to find the second derivative: \begin{aligned} f''(x) &= \frac{d}{dx} \left[\frac{-1}{\sqrt{2 \pi}} \cdot x e^{-x^2/2}\right] \\ &= \frac{-1}{\sqrt{2 \pi}} \frac{d}{dx} \left[x e^{-x^2/2}\right] \\ &= \frac{-1}{\sqrt{2 \pi}} \left[x \cdot \frac{d}{dx} \left(e^{-x^2/2}\right) + 1 \cdot e^{-x^2/2}\right] \\ &= \frac{-1}{\sqrt{2 \pi}} \left[x \cdot \left(-x e^{-x^2/2}\right) + e^{-x^2/2}\right] \\ &= \frac{-1}{\sqrt{2 \pi}} \left[e^{-x^2 / 2} - x^2 e^{-x^2/2}\right] \\ &= \frac{1}{\sqrt{2 \pi}} (x^2 - 1) e^{-x^2/2}. \end{aligned} Because \frac{1}{\sqrt{2 \pi}} > 0 and e^{-x^2/2} > 0 for all x, the sign of the second derivative will always be the same as that of x^2 - 1. Consequently, the second derivative is weakly negative—and thus the function is concave—if and only if -1 \leq x \leq 1.

You can, in principle, keep taking derivatives beyond the second one. The third derivative f''' is the derivative of the second derivative: f'''(c) = \lim_{x \to c} \frac{f''(x) - f''(c)}{x - c}. And the fourth derivative is … well, you get the picture. In practice, it is very rare to have to work with a derivative beyond the first or the second. (And if I’m ever at a formal theory talk where the model entails an assumption on the third derivative, I will harass the authors about it, as our colleague Peter Bils can attest.)

3.6 Concept review

Slope: For a linear function, f(x) = \alpha + \beta x, the slope is the coefficient \beta. It can be calculated using the rise-over-run formula: for any distinct points x_1 and x_2 in the domain of f, \beta = \frac{f(x_1) - f(x_2)}{x_1 - x_2}.
Derivative: The derivative of f at the point c, denoted f'(c), is a measure of how steeply the function is increasing or decreasing at that point. The formal definition of the derivative is that f'(c) = \lim_{x \to c} \frac{f(x) - f(c)}{x - c} = \lim_{h \to 0} \frac{f(c + h) - f(c)}{h}, provided that this limit exists.
Differentiability: A function f : X \to \mathbb{R} is differentiable at a point in its domain, c \in X, if the derivative f'(c) exists. We call f a differentiable function if it is differentiable at every point in its domain.
Constant multiple rule: If g(x) = c f(x), where c is a constant real number, then g'(x) = c f'(x).
Sum rule: If h(x) = f(x) + g(x), then h'(x) = f'(x) + g'(x).
Product rule: If h(x) = f(x) g(x), then h'(x) = f(x) g'(x) + f'(x) g(x).
Quotient rule: If h(x) = f(x) / g(x), then h'(x) = \frac{f'(x) g(x) - f(x) g'(x)}{g(x)^2}.
Chain rule: If h(x) = f(g(x)), then h'(x) = f'(g(x)) \cdot g'(x).
Second derivative: The second derivative of a differentiable function f(x), denoted f''(x), is the derivative of the function’s derivative: f''(c) = \lim_{x \to c} \frac{f'(x) - f'(c)}{x - c}.
Twice differentiability: A function f : X \to \mathbb{R} is twice differentiable at a point in its domain, c \in X, if the second derivative f''(c) exists. We call f a twice differentiable function if it is twice differentiable at every point in its domain.
Convex function: A twice differentiable function is convex if f''(x) \geq 0, and additionally is strictly convex if f''(x) > 0.
Concave function: A twice differentiable function is concave if f''(x) \leq 0, and additionally is strictly concave if f''(x) < 0.

Chain rule: If h(x) = f(g(x)), then h'(x) = f'(g(x)) \cdot g'(x).
Concave function: A twice differentiable function is concave if f''(x) \leq 0, and additionally is strictly concave if f''(x) < 0.
Constant multiple rule: If g(x) = c f(x), where c is a constant real number, then g'(x) = c f'(x).
Convex function: A twice differentiable function is convex if f''(x) \geq 0, and additionally is strictly convex if f''(x) > 0.
Derivative: The derivative of f at the point c, denoted f'(c), is a measure of how steeply the function is increasing or decreasing at that point. The formal definition of the derivative is that f'(c) = \lim_{x \to c} \frac{f(x) - f(c)}{x - c} = \lim_{h \to 0} \frac{f(c + h) - f(c)}{h}, provided that this limit exists.
Differentiability: A function f : X \to \mathbb{R} is differentiable at a point in its domain, c \in X, if the derivative f'(c) exists. We call f a differentiable function if it is differentiable at every point in its domain.
Product rule: If h(x) = f(x) g(x), then h'(x) = f(x) g'(x) + f'(x) g(x).
Quotient rule: If h(x) = f(x) / g(x), then h'(x) = \frac{f'(x) g(x) - f(x) g'(x)}{g(x)^2}.
Second derivative: The second derivative of a differentiable function f(x), denoted f''(x), is the derivative of the function’s derivative: f''(c) = \lim_{x \to c} \frac{f'(x) - f'(c)}{x - c}.
Slope: For a linear function, f(x) = \alpha + \beta x, the slope is the coefficient \beta. It can be calculated using the rise-over-run formula: for any distinct points x_1 and x_2 in the domain of f, \beta = \frac{f(x_1) - f(x_2)}{x_1 - x_2}.
Sum rule: If h(x) = f(x) + g(x), then h'(x) = f'(x) + g'(x).
Twice differentiability: A function f : X \to \mathbb{R} is twice differentiable at a point in its domain, c \in X, if the second derivative f''(c) exists. We call f a twice differentiable function if it is twice differentiable at every point in its domain.