Skip to main content

Mathematical Scholars Calculus 3

Gabriel Kerr

Section 3.1 Geometry I

So far, we have avoided the usual undergraduate approach to calling a vector something with a ‘magnitude’ and ‘direction’. The reason for this is that the notion of magnitude is in fact additional data that we impose upon a vector space. By this I mean that, given a vector \(\mb{v}\) in a vector space \(V\text{,}\) there is no unique way to assign that vector a length. This is because a vector space should be thought of like a skeleton of a person. It has only enough structure to do vector arithmetic, but not enough to talk about distance and angles. However, we can put some flesh on this skeleton and start working with geometry by adding additional bells and whistles. To do so, we introduce inner product spaces which are vector spaces with an additional pairing.

Definition 3.1.1.

An inner product space \(V\) over \(K\) is a vector space along with a pairing
\begin{equation*} \left\langle , \right\rangle : V \times V \to K \end{equation*}
which satisfies
Linear
\begin{equation*} \left\langle \lambda \mb{v}_1 + \mb{v}_2 , \mb{v} \right\rangle = \lambda \left\langle \mb{v}_1 , \mb{v} \right\rangle + \left\langle \mb{v}_2 , \mb{v} \right\rangle. \end{equation*}
Conjugate Symmetric
\begin{equation*} \left\langle \mb{u} , \mb{v} \right\rangle = \overline{\left\langle \mb{v} , \mb{u} \right\rangle}. \end{equation*}
Positive Definite
\begin{equation*} \left\langle \mb{v} , \mb{v} \right\rangle \geq 0 \end{equation*}
with equality if and only if \(\mb{v} = \mb{0}\text{.}\)
Note that the positive definite condition makes sense even when \(K = \mathbb{C}\text{.}\) In that case, one might worry that \(\left\langle \mb{v} , \mb{v} \right\rangle\) is a complex number like \(1 + 2i\text{.}\) Then what does it mean for it to be greater than zero? However, we see from conjugate symmetry, that
\begin{equation*} \left\langle \mb{v} , \mb{v} \right\rangle = \overline{\left\langle \mb{v} , \mb{v} \right\rangle} \end{equation*}
which tells us that the value must be a real number (only real numbers equal their conjugates). But we know what it means to say a real number is non-negative.

Subsection 3.1.1 Transpose Matrices

There are several examples of inner product spaces, the most commonly used being the real vector space \(\mathbb{R}^n\) with the dot product. To discuss this, we need the simple definition of transpose matrix.

Definition 3.1.2.

Given an \(m \times n\) matrix \(A = (a_{ij})\text{,}\) its transpose is the \(n \times m\) matrix \(A^T = (\tilde{a}_{ij})\) where \(\tilde{a}_{ij} = a_{ji}\text{.}\) We say that a square matrix is symmetric if \(A^T = A\text{.}\)

Example 3.1.3. Transpose of a matrix.

Finding the transpose matrix is simple, just swap row entries with column entries. For example, if
\begin{equation*} A = \left[ \begin{matrix} 1 \amp 2 \amp 3 \\ 4 \amp 5 \amp 6 \end{matrix} \right] \end{equation*}
then
\begin{equation*} A^T = \left[ \begin{matrix} 1 \amp 4 \\ 2 \amp 5 \\ 3 \amp 6 \end{matrix} \right]. \end{equation*}
The transpose behaves nicely with respect to several of the operations we have introduced.

Proof.

The first two equations can be justified by writing out the formulas and checking both sides. For the third, note that \(A\) is invertible if and only if there is an inverse matrix \(B\) for which
\begin{equation*} I = I^T = (AB)^T = B^T A^T \hspace{.3in} I = I^T = (BA)^T = A^T B^T. \end{equation*}
Thus \(B\) is an inverse to \(A\) if and only if \(B^T\) is an inverse to \(A^T\text{.}\) For the statement on determinants, we note that if \(A\) is not invertible then by (3), so is \(A^T\) so both determinants are zero. On the other hand, if \(A\) is invertible, then by Proposition 2.7.6 \(A\) is a product of elementary matrices
\begin{equation*} A = E_1 \cdots E_r \end{equation*}
so that
\begin{equation*} A^T = E_r^T \cdots E_1^T. \end{equation*}
By Proposition 2.7.16 we have
\begin{equation*} \det (A) = \det (E_1 \cdots E_r) = \det (E_1) \cdots \det (E_r) \end{equation*}
and
\begin{equation*} \det (A^T) = \det (E_1^T ) \cdots \det (E_r^T) . \end{equation*}
But one observes that the Type I and II elementary matrices are symmetric, and the transpose of a Type III elementary matrix is also Type III which has determinant equal to \(1\) (by Lemma 2.7.14). This implies each of the factors in the two above equations are equal so that \(\det (A^T) = \det (A)\text{.}\) Finally, equation (5) is justified by Proposition 2.5.11.

Subsection 3.1.2 Dot Product

Long overdue is the following notion which gives the central example of a real inner product space.

Definition 3.1.5.

Given column vectors \(\mb{u}\) and \(\mb{v}\) in \(\mathbb{R}^n\text{,}\) the dot product of \(\mb{u}\) with \(\mb{v}\) is the real number
\begin{equation*} \mb{u} \cdot \mb{v} = \mb{u}^T \mb{v} . \end{equation*}
If we are given \(\mb{u}\) and \(\mb{v}\) as row vectors, we take \(\mb{u} \cdot \mb{v} = \mb{u} \mb{v}^T\text{.}\) We say that \(\mb{u}\) and \(\mb{v}\) are orthogonal if \(\mb{u} \cdot \mb{v} = 0\text{.}\)

Example 3.1.6. Dot products of vectors.

Let us make sure that we understand just how easy the dot product is. If
\begin{equation*} \mb{u} = \left[ \begin{matrix} a_1 \\ \vdots \\ a_n \end{matrix} \right] \hspace{.2in} \mb{v} = \left[ \begin{matrix} b_1 \\ \vdots \\ b_n \end{matrix} \right] \end{equation*}
then
\begin{equation*} \mb{u} \cdot \mb{v} = a_1 b_1 + a_2 b_2 + \cdots + a_n b_n. \end{equation*}
Notice we can obtain any real number as a dot product, but, as we will see, these numbers tell us about the relative direction of vectors.
The diligent student is encouraged to verify the following lemma.
The most important aspect of the dot product is that it allows one to bring geometric notions into the vector space \(\mathbb{R}^n\text{.}\) First, we consider a real vector \(\mb{v} \in \mathbb{R}^n\) as an arrow from the origin to the endpoint \(\mb{v}\text{.}\) Then we have the important ancient result:
For any inner product space (over \(\mathbb{R}\) or \(\mathbb{C}\)), the square root
\begin{equation*} \| \mb{v} \| = \sqrt{\left\langle \mb{v} , \mb{v} \right\rangle} \end{equation*}
makes sense and we often refer to this number as the norm of \(\mb{v}\text{.}\) One can check that the norm in any inner product space satisfies the equation
\begin{equation*} \| \lambda \mb{v} \| = | \lambda | \, \| \mb{v} \| . \end{equation*}
The fact that the proposition above is the Pythagorean Theorem can be seen as in two dimensions that this is just the distance formula (which is really the Pythagorean Theorem) and that in three dimensions this is the Pythagorean Theorem applied twice. In higher dimensions, one could either argue inductively or simply take this as a definition. While length is of great geometric importance, so is angle. Surprisingly, the dot product gives us both!

Proof.

First, if \(\mb{u}\) and \(\mb{v}\) are linearly dependent, then \(\mb{v} = \lambda \mb{u}\) with \(\lambda \ne 0\) (since both vectors are non-zero). If \(\lambda \gt 0\) then both vectors point in the same direction, so their angle should be \(\theta = 0\) or \(\cos \theta = 1\text{.}\) But in this case we see
\begin{equation*} \mb{u} \cdot \mb{v} = \mb{u} \cdot (\lambda \mb{u}) = \lambda \| \mb{u} \|^2 = \| \mb{u} \| \| \lambda \mb{u} \| = \| \mb{u} \| \| \mb{v} \| \end{equation*}
which verifies the equality. If \(\lambda \lt 0\) then we have that they point in opposite directions and thus \(\theta = \pi\) or \(\cos \theta = -1\text{.}\) Again we observe
\begin{equation*} \mb{u} \cdot \mb{v} = \mb{u} \cdot (\lambda \mb{u}) = \lambda \| \mb{u} \|^2 = - \| \mb{u} \| \| \lambda \mb{u} \| = \| \mb{u} \| \| \mb{v} \| (-1). \end{equation*}
So let us assume they are linearly independent. Then the arrow vectors \(\mb{u}, \mb{v}\) and \(\mb{u} - \mb{v}\) make up a triangle. If the angle at \(\mb{u}\) and \(\mb{v}\) is \(\theta\) then the Law of Cosines (a basic generalization of the Pythagorean Theorem) says that
\begin{align*} \| \mb{u} \|^2 + \| \mb{v} \|^2 - 2 \| \mb{u} \| \| \mb{v} \| \cos (\theta ) \amp = \| \mb{u} - \mb{v} \|^2 , \\ \amp = (\mb{u} - \mb{v} ) \cdot (\mb{u} - \mb{v} ), \\ \amp = \mb{u} \cdot \mb{u} - 2 \mb{u} \cdot \mb{v} + \mb{v} \cdot \mb{v}, \\ \amp = \| \mb{u} \|^2 + \| \mb{v} \|^2 - 2 \mb{u} \cdot \mb{v}. \end{align*}
Subtracting \(\| \mb{u}\|^2 + \| \mb{v} \|^2\) from both sides and dividing by \(-2\) gives the result.
The upshot of this amazing result is that we can now get quantitative and qualitative information about vectors very easily.

Example 3.1.10. Computing the angle between vectors.

If we were asked to find the angle between the row vectors \(\mb{u} = [ 1, 0, -1 ]\) and \(\mb{v} = [ 0, 1, -1 ]\) before having the dot product, it is not surprising that we would struggle. But now we just compute lengths and the dot product to see:
\begin{align*} \| \mb{u} \| \amp = \sqrt{ 1^2 + 0^2 + (-1)^2} = \sqrt{2}, \\ \| \mb{v} \| \amp = \sqrt{ 0^2 + 1^2 + (-1)^2} = \sqrt{2}, \\ \mb{u} \cdot \mb{v} \amp = 1 \cdot 0 + 0 \cdot 1 + (-1) \cdot (-1) = 1. \end{align*}
Putting these into our formula gives
\begin{equation*} \sqrt{2} \cdot \sqrt{2} = 1 \cdot \cos (\theta) \end{equation*}
or
\begin{equation*} \cos (\theta ) = \frac{1}{2} \end{equation*}
implying that \(\theta = \frac{\pi}{3}\text{.}\) In this case, we get a quick pleasant answer. However, we see even when we find that \(\mb{u} \cdot \mb{v} \gt 0\) that the angle made between \(\mb{u}\) and \(\mb{v}\) is acute. Were it \(0\text{,}\) they would be perpendicular and if it were negative, they would form an obtuse angle. Point being, even without evaluating \(\arccos\text{,}\) we obtain qualitative information about our vectors.
Moreover, one can easily prove some basic, but useful, inequalities such as.

Proof.

This can be shown for a general inner product space using the Cauchy-Schwarz inequality below. Here we check it for the dot product on \(\mathbb{R}^n\) by squaring both sides and seeing that \(\mb{u} \cdot \mb{v} \leq \| \mb{u} \| \| \mb{v} \|\) (because \(\cos (\theta ) \leq 1\)).
Note that this is the triangle inequality because it is saying that the sum of two side lengths of a triangle is greater than the side length of the third.

Subsection 3.1.3 Other Examples

With all of this discussion of the dot product, one might wonder why we bothered with the general definition of an inner product space. In fact, many of the most useful and advanced applications of linear algebra require working over \(\mathbb{C}\) (for example, quantum mechanics), and so it is worth mentioning a few more interesting examples.

Example 3.1.12. Hermitian inner product.

Let \(V\) be the vector space of column vectors in \(\mathbb{C}^n\text{.}\) We can define a straightforward generalization of the dot product here by taking
\begin{equation*} \left\langle \mb{u} , \mb{v} \right\rangle = \mb{u}^T \cdot \bar{\mb{v}}. \end{equation*}
Writing this out, we have if
\begin{equation*} \mb{u} = \left[ \begin{matrix} w_1 \\ \vdots \\ w_n \end{matrix} \right] \hspace{.2in} \mb{v} = \left[ \begin{matrix} z_1 \\ \vdots \\ z_n \end{matrix} \right] \end{equation*}
then
\begin{equation*} \mb{u} \cdot \mb{v} = w_1 \bar{z}_1 + w_2 \bar{z}_2 + \cdots + w_n \bar{z}_n. \end{equation*}
One can check this gives an inner product on \(\mathbb{C}^n\text{.}\) In fact, this inner product has a lot more structure than meets the eye and is the right answer to how to do geometry in complex space. For example, one still has that \(\| \mb{v} \|\) gives the length of the vector \(\mb{v}\text{.}\) But \(\left\langle \mb{u}, \mb{v} \right\rangle\) gives two quantities, relating to both an angle and an area swept out by the two vectors.
Another important example to consider is the following Hilbert Space.

Example 3.1.13. Hilbert space of functions.

Let \([a, b]\) be an interval in \(\mathbb{R}\) and consider all square integrable functions \(L^2 ([a,b] )\) from \([a,b]\) to \(\mathbb{C}\) (this means that both the real and imaginary parts of the square of the function are integrable). Then we can define the inner product
\begin{equation*} \left\langle f , g \right\rangle = \frac{1}{b - a} \int_{a}^b f(t) \overline{g(t)} \diff{t}. \end{equation*}
Here, when we integrate a complex valued function \(f(t) = x(t) + i y(t)\) we mean
\begin{equation*} \int_a^b f(t) \diff{t} = \int_a^b x(t) \diff{t} + i \int_a^b y(t) \diff{t}. \end{equation*}
This inner product space is widely used and generalized in applications. We will see in the next section a rather interesting set of vectors in this space that form the heart of Fourier series.
The result concerning angles between vectors and the dot product suggests a more general inequality for inner product spaces.
There is a rather fun proof of this fact which we will put as a guided exercise.

Exercises 3.1.4 Exercises

1.

Show that, given any matrix \(A\text{,}\) the matrix \(A^T A\) is symmetric.

2.

Find the angle between the vectors
(a)
\begin{equation*} \mb{u} = \twovec{1}{-1} , \mb{v} = \twovec{0}{3} \end{equation*}
(b)
\begin{equation*} \mb{u} = \threevec{-1}{1}{0} , \mb{v} = \threevec{0}{-1}{1} \end{equation*}

3.

Find the unit vector pointing in the direction of \(\mb{v}\) when
(a)
\begin{equation*} \mb{v} = \twovec{3}{5} \end{equation*}
(b)
\begin{equation*} \mb{v} = \threevec{-1}{2}{-2} \end{equation*}

4.

Suppose \(\mb{u}\) is a non-zero vector in \(\mathbb{R}^n\text{.}\) Projecting another vector \(\mb{v}\) to \(\mb{u}\) means finding the part of \(\mb{v}\) that points parallel to \(\mb{u}\text{.}\) More precisely, there are unique vectors \(\mb{v}_\perp\) and \(\mb{v}_{\|}\) for which
\begin{equation*} \mb{v} = \mb{v}_\perp + \mb{v}_{\|} \end{equation*}
where \(\mb{v}_\perp\) is orthogonal to \(\mb{u}\) and \(\mb{v}_{\|}\) is the projection. Explain why the formula
\begin{equation*} \frac{\mb{u} \cdot \mb{v}}{\|\mb{u} \|^2} \mb{u} \end{equation*}
gives the vector projection \(\mb{v}_{\|}\text{.}\)

5. Cauchy-Schwarz Proof.

Let \(V\) be an inner product space and suppose \(\mb{v}\) is a non-zero vector.
(a)
Show that \(\mb{v}\) and \(\mb{u}\) are linearly dependent then the Cauchy-Inequality is actually an equality.
(b)
Show that if \(\mb{v}\) and \(\mb{u}\) are linearly independent then there is no \(t\) and \(\theta\) in \(\mathbb{R}\) for which
\begin{equation*} \| \mb{u} - t e^{i\theta} \mb{v} \|^2 = 0 . \end{equation*}
(c)
Show that if \(\left\langle \mb{u} , \mb{v} \right\rangle\) has polar decomposition \(r e^{i \psi }\) then
\begin{equation*} \| \mb{u} - t e^{i\psi} \mb{v} \|^2 = \|\mb{v} \|^2 t^2 - 2 \, | \left\langle \mb{u} , \mb{v} \right\rangle | \, t + \|\mb{u}\|^2. \end{equation*}
(d)
Using parts (b) and (c), show that the Cauchy-Schwarz inequality for independent vectors follows from the fact that there are no real solutions to this equation
\begin{equation*} \|\mb{v} \|^2 t^2 - 2 | \left\langle \mb{u} , \mb{v} \right\rangle | t + \|\mb{u}\|^2 = 0 \end{equation*}
for \(t\text{.}\)
PrevTopNext