Chapter 9 Inner Product Spaces and Orthogonal Structure

In 1807 Joseph Fourier submitted a memoir on heat conduction to the French Academy of Sciences, in which he represented functions by trigonometric series; Lagrange and Laplace then raised objections to the practice of expanding functions in trigonometric series. The Academy made heat conduction the subject of its prize question for 1811, and Fourier won the prize with an expanded version of his memoir; the systematic account appeared in 1822 as Théorie analytique de la chaleur. The method forced mathematicians to rethink a basic question: in what sense can smooth sine and cosine waves approximate a function with corners or jumps? What it means to “have an expansion” cannot be separated from the class of functions and the mode of convergence.

In modern language: in $L^2[-\pi,\pi]$ the trigonometric system forms a complete orthogonal system, and the Fourier series converges to the original function in the mean-square sense; pointwise convergence requires a separate discussion.

Looking back on this history, later generations asked: what allowed Fourier to see past intuition? The key lay not in the summation tricks of calculus but in a deep “geometric perpendicularity” hidden among the trigonometric functions:

\int_{-\pi}^{\pi} \sin(mx)\sin(nx)\,dx = 0 \quad (m,n\in\mathbb{Z}_{>0},\ m\neq n)

(1)

This integral is exactly zero, which geometrically means that sine waves of different frequencies are independent of one another and do not interfere. And the Fourier coefficients that give the amplitude of each component,

b_n = \frac{1}{\pi}\int_{-\pi}^{\pi} f(x)\sin(nx)\,dx

(2)

have exactly the same algebraic form as the formula for projecting a geometric arrow onto orthogonal coordinate axes in a finite-dimensional space—the only difference is that the “dot product,” a sum of products of discrete coordinates there, is elevated here into an integral of continuous functions over an interval.

This observation turns “functions are vectors too” from a mere metaphor into a fact: once we choose a function space, an inner product, and the appropriate completeness conditions, we can study length, orthogonality, and projection. Hilbert spaces extend this geometry to infinite dimensions and have become an important language for quantum theory, partial differential equations, and signal processing.

Here, however, Fourier’s story serves only as a mirror for ideas. Properties that are taken for granted in finite dimensions often require additional conditions in infinite-dimensional spaces—convergence, completeness, the domain of an operator, and so on.

The scope of this book is the rigorous and well-controlled setting of finite-dimensional linear spaces. This takes nothing away from the value of the theory in this chapter: in a bare vector space we have only “addition” and “scalar multiplication”; it is a limp world with no length, no angle, and no way to define a perpendicular. It is precisely the introduction of the inner product that gives the space a rigorous measure of length (the norm), a notion of angle, and optimal projection—turning a purely algebraic structure into a solid, concrete geometric stage.

The orthogonality, Gram-Schmidt orthogonalization, orthogonal projection operators, and spectral decomposition of symmetric matrices that we forge in $\mathbb{R}^n$ and $\mathbb{C}^n$ lose none of their core ideas in infinite-dimensional function spaces. In other words, what this chapter develops is by no means an algebraic special case confined to finite dimensions; it is a geometric foundation that spans the discrete and the continuous and, once the analytic conditions are supplied, extends to infinite dimensions.

Chapter Structure and Learning Objectives¶

The introduction pointed out that “orthogonality of functions” and “orthogonality of vectors” share exactly the same structure. This analogy rests on a precise axiomatic definition—and that is where this chapter begins.

§9.1 lays the foundation: it defines the inner product axiomatically and explains why the complex field forces us to introduce conjugation. Positive definiteness guarantees that “length” is never negative, the central premise that lets the whole geometric framework stand. From the inner product follow naturally the norm, orthogonality, and the Cauchy-Schwarz inequality, the last of which also foreshadows, in an elegant form, the geometric root of the uncertainty principle in Chapter 10. The section closes with Gram-Schmidt orthogonalization—an algorithm that gives a general method for constructing an orthogonal basis in any inner product space and is the geometric essence of the QR decomposition.

With the inner product in hand, §9.2 naturally asks: which linear transformations fully preserve this geometric structure? Orthogonal matrices (over the reals) and unitary matrices (over the complex numbers) are the answer. Their inverse equals their transpose or conjugate transpose, their 2-norm condition number in numerical computation is always 1, and geometrically they are compositions of rotations and reflections—they only rotate the orientation of the coordinate system and do not change distances or angles between vectors.

§9.3 is the heart of the chapter. Self-adjointness ( $\mathbf{A} = \mathbf{A}^*$ ) means that a matrix “respects the inner product,” and from this it follows precisely that the eigenvalues must be real and that eigenvectors belonging to different eigenvalues must be orthogonal. Here the spectral decomposition theorem brings the diagonalization theory of Chapter 8 to its most perfect form: symmetric/Hermitian matrices are not only diagonalizable but can be diagonalized by an orthogonal (unitary) matrix. Positive definiteness further characterizes the “energy character” of such matrices and is an indispensable language in optimization and statistics.

§9.4 lands the theory in geometry through quadratic forms. The principal axis theorem shows that every quadratic form can be brought to a diagonal canonical form by rotating the coordinate system—the eigenvalues determine the shape of the ellipsoid or hyperboloid, and the eigenvectors give the directions of the principal axes. Hermitian forms then extend this picture to the complex field.

Once you have read the whole chapter, your understanding of “why symmetric matrices are special” will deepen from Chapter 8’s “they happen to be diagonalizable” to “self-adjointness gives them an orthogonal structure by nature.” This viewpoint will give you an entirely different depth of field when you meet the introduction to quantum mechanics in Chapter 10 and the singular value decomposition in Chapter 11.

The two symbols in the section titles of this chapter mark their reading priority: ◇ marks an optional extension (for example, §9.2.3 on unitary matrices and quantum computing), which can be skipped without affecting the main line; ◆ marks an advanced proof (for example, the equivalent conditions for positive definiteness), where on a first reading you may grasp the conclusion first and return to its derivation later.

9.1 Inner Product Spaces¶

Why Do We Need Inner Product Spaces?¶

In Chapter 2 we saw how the real inner product $\mathbf{u} \cdot \mathbf{v}$ connects algebraic computation with geometric measurement—it encodes both the length of a vector ( $\|\mathbf{v}\| = \sqrt{\mathbf{v} \cdot \mathbf{v}}$ ) and the angle between vectors ( $\cos\theta = \frac{\mathbf{u} \cdot \mathbf{v}}{\|\mathbf{u}\|\|\mathbf{v}\|}$ ).

We now face a broader question: how can similar geometric notions be defined in the complex vector space $\mathbb{C}^n$ ? Carrying the definition $\mathbf{u} \cdot \mathbf{v} = \sum_i u_i v_i$ of Chapter 2 over to the complex field unchanged runs into a fatal problem:

\mathbf{v} \cdot \mathbf{v} = \sum_i v_i^2

(3)

can be negative or even purely imaginary over the complex numbers! For example, take $\mathbf{v} = \begin{bmatrix} i \\ 0 \end{bmatrix}$ ; then $\mathbf{v} \cdot \mathbf{v} = i^2 = -1 < 0$ . This would mean that the “length” of a vector could be negative, which is geometrically meaningless.

The solution is to introduce conjugation:

⟨ \mathbf{v} | \mathbf{v} ⟩ = \sum_i \overline{v_i} v_i = \sum_i |v_i|^2 \geq 0

(4)

This guarantees that “length” is never negative, and so keeps geometric intuition valid. This key insight leads us into the systematic theory of inner product spaces.

An inner product space is a vector space equipped with an additional inner product operation, which brings in geometric notions. When defining an inner product on a complex vector space, we must pay particular attention to how conjugation is handled.

9.1.1 Definition and Properties of the Inner Product¶

Definition 1 (Inner Product)

Let $V$ be a vector space over a field $\mathbb{K}$ ( $\mathbb{R}$ or $\mathbb{C}$ ). An inner product is a function $⟨ \cdot|\cdot ⟩: V \times V \to \mathbb{K}$ such that for all $\mathbf{u}, \mathbf{v}, \mathbf{w} \in V$ and $\alpha, \beta \in \mathbb{K}$ the following properties hold:

Conjugate symmetry: $⟨ \mathbf{u}| \mathbf{v} ⟩ = \overline{⟨ \mathbf{v}| \mathbf{u} ⟩}$
- In particular, when $\mathbb{K} = \mathbb{R}$ , $⟨ \mathbf{u}| \mathbf{v} ⟩ = ⟨ \mathbf{v}| \mathbf{u} ⟩$
Linearity in the second argument: $⟨ \mathbf{w}|\alpha\mathbf{u} + \beta\mathbf{v} ⟩ = \alpha⟨ \mathbf{w}|\mathbf{u} ⟩ + \beta⟨ \mathbf{w}|\mathbf{v} ⟩$
Positive definiteness: $⟨ \mathbf{v}| \mathbf{v} ⟩ > 0$ for every nonzero vector $\mathbf{v} \in V$ , and $⟨ \mathbf{v}| \mathbf{v} ⟩ = 0$ if and only if $\mathbf{v} = \mathbf{0}$

The most common inner product is the standard inner product on Euclidean space:

import numpy as np

# Inner product on a real vector space
x = np.array([1, 2, 3])
y = np.array([4, 5, 6])

# Standard inner product (two equivalent ways)
inner_product1 = np.dot(x, y)
inner_product2 = x @ y  # using the @ operator

print(f"Inner product of real vectors: {inner_product1}")  # output: 32

# Inner product on a complex vector space
z1 = np.array([1+2j, 3-1j, 0+1j])
z2 = np.array([2-1j, 1+1j, 3-2j])

# ⚠️ A common mistake
wrong_inner = np.dot(z1, z2)  # (10+8j) - wrong!
print(f"Wrong method np.dot(): {wrong_inner}")

# ✅ The correct method
correct_inner = np.vdot(z1, z2)  # -4j - correct!
print(f"Correct method np.vdot(): {correct_inner}")

# Verify positive definiteness: ⟨z1|z1⟩ must be a nonnegative real number
self_inner = np.vdot(z1, z1)
print(f"⟨z1|z1⟩ = {self_inner.real} ≥ 0")  # 16.0

# Check by hand
manual_result = np.conj(z1) @ z2
print(f"Manual computation check: {manual_result}")  # output: -4j

Proof 1

(Readers comfortable with matrix multiplication can see at a glance that $⟨ \mathbf{y}|\mathbf{A}|\mathbf{x} ⟩ = \overline{⟨ \mathbf{x}|\mathbf{A}^*|\mathbf{y} ⟩}$ ; below we give a more detailed proof.) We prove the lemma using the conventions of bra-ket notation and the properties of the inner product.

Bra-ket notation and the complex inner product
In bra-ket notation, $|\mathbf{x}⟩$ denotes a column vector and $⟨ \mathbf{x}|$ its conjugate transpose (a row vector). Here we take the inner product of $\mathbf{y}$ and $\mathbf{A}\mathbf{x}$ , both $m$ -dimensional: $⟨\mathbf{y}|\mathbf{A}\mathbf{x}⟩=\sum_{i=0}^{m-1}\overline{y_i}(\mathbf{A}\mathbf{x})_i$ .
Computing $⟨ \mathbf{y}|\mathbf{A}|\mathbf{x} ⟩$
Here $\mathbf{x}$ is an $n$ -dimensional vector and $\mathbf{y}$ an $m$ -dimensional vector. The matrix $\mathbf{A}$ acting on $\mathbf{x}$ gives the vector $\mathbf{A}\mathbf{x}$ , whose $i$ -th component is
$(\mathbf{A}\mathbf{x})_i = \sum_{j=0}^{n-1} a_{ij} x_j$
(10)
Hence the inner product is
$⟨ \mathbf{y}|\mathbf{A}|\mathbf{x} ⟩ = \sum_{i=0}^{m-1} \overline{y_i} \cdot (\mathbf{A}\mathbf{x})_i = \sum_{i=0}^{m-1} \overline{y_i} (\sum_{j=0}^{n-1} a_{ij} x_j)$
(11)
Interchanging the order of summation:
$⟨ \mathbf{y}|\mathbf{A}|\mathbf{x} ⟩ = \sum_{i=0}^{m-1}\sum_{j=0}^{n-1} \overline{y_i} a_{ij} x_j$
(12)
Computing $⟨ \mathbf{x}|\mathbf{A}^*|\mathbf{y} ⟩$
The matrix $\mathbf{A}^*$ acting on $\mathbf{y}$ gives the vector $\mathbf{A}^*\mathbf{y}$ , whose $j$ -th component is
$(\mathbf{A}^*\mathbf{y})_j = \sum_{i=0}^{m-1} (\mathbf{A}^*)_{ji} y_i = \sum_{i=0}^{m-1} \overline{a_{ij}} y_i$
(13)
By the definition of the inner product:
$⟨ \mathbf{x}|\mathbf{A}^*|\mathbf{y} ⟩ = \sum_{j=0}^{n-1} \overline{x_j} \cdot (\mathbf{A}^*\mathbf{y})_j = \sum_{j=0}^{n-1} \overline{x_j} (\sum_{i=0}^{m-1} \overline{a_{ij}} y_i)$
(14)
$⟨ \mathbf{x}|\mathbf{A}^*|\mathbf{y} ⟩ = \sum_{j=0}^{n-1}\sum_{i=0}^{m-1} \overline{x_j} \overline{a_{ij}} y_i$
(15)
The final conclusion By 2 and 3,
$\overline{⟨ \mathbf{x}|\mathbf{A}^*|\mathbf{y} ⟩} = \overline{\sum_{j=0}^{n-1}\sum_{i=0}^{m-1} \overline{x_j} \overline{a_{ij}} y_i} = \sum_{j=0}^{n-1}\sum_{i=0}^{m-1} x_j a_{ij} \overline{y_i} = \sum_{i=0}^{m-1}\sum_{j=0}^{n-1} \overline{y_i} a_{ij} x_j = ⟨ \mathbf{y}|\mathbf{A}|\mathbf{x} ⟩$
(16)
The derivation above proves that
$\underline{⟨ \mathbf{y}|\mathbf{A}|\mathbf{x} ⟩ = \overline{⟨ \mathbf{x}|\mathbf{A}^*|\mathbf{y} ⟩}}$
(17)

The inner product brings in the notion of a norm (length):

The inner product also brings in the notions of angle and orthogonality:

Proof 2

If $\mathbf{v} = \mathbf{0}$ , the inequality clearly holds. Assume $\mathbf{v} \neq \mathbf{0}$ .

For any scalar $\lambda$ , positive definiteness of the inner product gives

\|\mathbf{u} - \lambda\mathbf{v}\|^2 = ⟨ \mathbf{u} - \lambda\mathbf{v}, \mathbf{u} - \lambda\mathbf{v} ⟩ \geq 0

(20)

Expanding:

⟨ \mathbf{u}, \mathbf{u} ⟩ - \lambda⟨ \mathbf{u}, \mathbf{v} ⟩ - \overline{\lambda}⟨ \mathbf{v}, \mathbf{u} ⟩ + |\lambda|^2⟨ \mathbf{v}, \mathbf{v} ⟩ \geq 0

(21)

Take $\lambda = \frac{⟨ \mathbf{v}, \mathbf{u} ⟩}{⟨ \mathbf{v}, \mathbf{v} ⟩}$ (the optimal projection); substituting and simplifying gives

\|\mathbf{u}\|^2 - \frac{|⟨ \mathbf{u}, \mathbf{v} ⟩|^2}{\|\mathbf{v}\|^2} \geq 0

(22)

Rearranging:

\underline{|⟨ \mathbf{u}, \mathbf{v} ⟩|^2 \leq \|\mathbf{u}\|^2 \cdot \|\mathbf{v}\|^2}

(23)

Taking square roots gives the Cauchy-Schwarz inequality.

Equality holds if and only if $\mathbf{u} - \lambda\mathbf{v} = \mathbf{0}$ , that is, $\mathbf{u} = \lambda\mathbf{v}$ .

Using the Cauchy-Schwarz inequality, we can define the angle between vectors:

9.1.2 Orthonormal Bases¶

A powerful feature of inner product spaces is that we can define orthogonal bases and orthonormal bases, which are extremely useful both in computation and in theoretical analysis.

In an orthonormal basis, representing and manipulating vectors becomes remarkably simple:

Property 1 (Properties of Orthonormal Bases)

Let $\mathbb{K} \in \{\mathbb{R}, \mathbb{C}\}$ and let $V$ be a finite-dimensional inner product space over the field $\mathbb{K}$ whose inner product $⟨ \cdot, \cdot ⟩$ is linear in the second argument and conjugate linear in the first (the physics convention). Let $\{\mathbf{e}_0, \mathbf{e}_1, \ldots, \mathbf{e}_{n-1}\}$ be an orthonormal basis of $V$ . Then for every $\mathbf{v} \in V$ :

The coordinates of $\mathbf{v}$ are $v_i = ⟨ \mathbf{e}_i, \mathbf{v} ⟩$ , that is, $\mathbf{v} = \sum_{i=0}^{n-1} ⟨ \mathbf{e}_i, \mathbf{v} ⟩\, \mathbf{e}_i$
The inner product of two vectors can be computed from their coordinates: $⟨ \mathbf{u}, \mathbf{v} ⟩ = \sum_{i=0}^{n-1} \overline{u_i}\, v_i$
The norm satisfies a generalization of the Pythagorean theorem: $\|\mathbf{v}\|^2 = \sum_{i=0}^{n-1} |v_i|^2$

Proof 3

Let $\mathbf{v} = \sum_{i=0}^{n-1} c_i \mathbf{e}_i$ .

Property 1. For fixed $j$ , use the linearity of the inner product in the second argument:

⟨ \mathbf{e}_j, \mathbf{v} ⟩ = ⟨ \mathbf{e}_j,\; \sum_{i=0}^{n-1} c_i \mathbf{e}_i ⟩ = \sum_{i=0}^{n-1} c_i ⟨ \mathbf{e}_j, \mathbf{e}_i ⟩ = \sum_{i=0}^{n-1} c_i\, \delta_{ji} = c_j

(26)

Hence $v_j = c_j = ⟨ \mathbf{e}_j, \mathbf{v} ⟩$ .

Property 2. Substitute $\mathbf{u} = \sum_i u_i \mathbf{e}_i$ and $\mathbf{v} = \sum_j v_j \mathbf{e}_j$ , and use conjugate linearity in the first argument and linearity in the second:

⟨ \mathbf{u}, \mathbf{v} ⟩ = ⟨ \sum_{i} u_i \mathbf{e}_i,\; \sum_{j} v_j \mathbf{e}_j ⟩ = \sum_{i,j} \overline{u_i}\, v_j ⟨ \mathbf{e}_i, \mathbf{e}_j ⟩ = \sum_{i,j} \overline{u_i}\, v_j\, \delta_{ij} = \sum_{i=0}^{n-1} \overline{u_i}\, v_i

(27)

Property 3. Set $\mathbf{u} = \mathbf{v}$ in Property 2:

\|\mathbf{v}\|^2 = ⟨ \mathbf{v}, \mathbf{v} ⟩ = \sum_{i=0}^{n-1} \overline{v_i}\, v_i = \sum_{i=0}^{n-1} |v_i|^2

(28)

\underline{ \mathbf{v} = \sum_{i=0}^{n-1} ⟨ \mathbf{e}_i, \mathbf{v} ⟩\, \mathbf{e}_i,\qquad ⟨ \mathbf{u}, \mathbf{v} ⟩ = \sum_{i=0}^{n-1} \overline{u_i}\, v_i,\qquad \|\mathbf{v}\|^2 = \sum_{i=0}^{n-1} |v_i|^2 }

(29)

9.1.3 The Gram-Schmidt Orthogonalization Method¶

Given a set of linearly independent (complex) vectors, how can we turn it into an orthogonal basis or an orthonormal basis? The answer is the Gram-Schmidt orthogonalization method (the Gram-Schmidt process).

Definition 7 (Gram-Schmidt Orthogonalization)

Gram-Schmidt orthogonalization is the process that turns a set of linearly independent vectors $\{\mathbf{v}_0, \mathbf{v}_1, \ldots, \mathbf{v}_{n-1}\}$ into a set of orthogonal vectors $\{\mathbf{u}_0, \mathbf{u}_1, \ldots, \mathbf{u}_{n-1}\}$ :

$\mathbf{u}_0 := \mathbf{v}_0$
For $k = 1, 2, \ldots, n-1$ :
$\mathbf{u}_k := \mathbf{v}_k - \sum_{j=0}^{k-1} \text{proj}_{\mathbf{u}_j}(\mathbf{v}_k)$
(30)
where the projection is defined by (the inner product is linear in the second argument)
$\text{proj}_{\mathbf{u}}(\mathbf{v}) := \frac{⟨ \mathbf{u}, \mathbf{v}⟩}{⟨ \mathbf{u}, \mathbf{u} ⟩} \mathbf{u}$
(31)

To obtain an orthonormal basis, simply normalize each orthogonal vector: $\mathbf{e}_k := \dfrac{\mathbf{u}_k}{\|\mathbf{u}_k\|}$

Example 2 (Gram-Schmidt Orthogonalization)

Consider three linearly independent vectors in $\mathbb{R}^3$ :

\mathbf{v}_0 = (1, 0, 0), \quad \mathbf{v}_1 = (1, 1, 0), \quad \mathbf{v}_2 = (1, 1, 1)

(32)

$\mathbf{u}_0 := \mathbf{v}_0 = (1, 0, 0)$
$\mathbf{u}_1 := \mathbf{v}_1 - \text{proj}_{\mathbf{u}_0}(\mathbf{v}_1)$
- $\text{proj}_{\mathbf{u}_0}(\mathbf{v}_1) = \dfrac{⟨ \mathbf{u}_0, \mathbf{v}_1 ⟩}{⟨ \mathbf{u}_0, \mathbf{u}_0 ⟩} \mathbf{u}_0 = \dfrac{1}{1} (1, 0, 0) = (1, 0, 0)$
- $\mathbf{u}_1 = (1, 1, 0) - (1, 0, 0) = (0, 1, 0)$
$\mathbf{u}_2 := \mathbf{v}_2 - \text{proj}_{\mathbf{u}_0}(\mathbf{v}_2) - \text{proj}_{\mathbf{u}_1}(\mathbf{v}_2)$
- $\text{proj}_{\mathbf{u}_0}(\mathbf{v}_2) = \dfrac{⟨ \mathbf{u}_0, \mathbf{v}_2 ⟩}{⟨ \mathbf{u}_0, \mathbf{u}_0 ⟩} \mathbf{u}_0 = (1, 0, 0)$
- $\text{proj}_{\mathbf{u}_1}(\mathbf{v}_2) = \dfrac{⟨ \mathbf{u}_1, \mathbf{v}_2 ⟩}{⟨ \mathbf{u}_1, \mathbf{u}_1 ⟩} \mathbf{u}_1 = \dfrac{1}{1} (0, 1, 0) = (0, 1, 0)$
- $\mathbf{u}_2 = (1, 1, 1) - (1, 0, 0) - (0, 1, 0) = (0, 0, 1)$

The result $\{\mathbf{u}_0, \mathbf{u}_1, \mathbf{u}_2\} = \{(1,0,0),(0,1,0),(0,0,1)\}$ is exactly the standard orthonormal basis of $\mathbb{R}^3$ .

Example 3 (Gram-Schmidt Orthogonalization in a Complex Vector Space)

Consider two linearly independent vectors in $\mathbb{C}^2$ :

\mathbf{v}_0 = (1, i), \quad \mathbf{v}_1 = (1+i, 2)

(33)

$\mathbf{u}_0 = \mathbf{v}_0 = (1, i)$ , $\;\|\mathbf{u}_0\|^2 = ⟨ \mathbf{u}_0, \mathbf{u}_0 ⟩ = \overline{1}\cdot 1 + \overline{i}\cdot i = 1 + 1 = 2$
$\mathbf{u}_1 = \mathbf{v}_1 - \text{proj}_{\mathbf{u}_0}(\mathbf{v}_1)$
- $⟨ \mathbf{u}_0, \mathbf{v}_1 ⟩ = \overline{1}\cdot(1+i) + \overline{i}\cdot 2 = (1+i) + (-i)(2) = 1 - i$
- $\text{proj}_{\mathbf{u}_0}(\mathbf{v}_1) = \dfrac{1-i}{2}(1, i) = (\dfrac{1-i}{2},\, \dfrac{i(1-i)}{2}) = (\dfrac{1-i}{2},\, \dfrac{1+i}{2})$
- $\mathbf{u}_1 = (1+i, 2) - (\dfrac{1-i}{2}, \dfrac{1+i}{2}) = (\dfrac{1+3i}{2}, \dfrac{3-i}{2})$

Normalizing gives an orthonormal basis:

$\mathbf{e}_0 = \dfrac{\mathbf{u}_0}{\|\mathbf{u}_0\|} = (\dfrac{1}{\sqrt{2}}, \dfrac{i}{\sqrt{2}})$
$\|\mathbf{u}_1\|^2 = ⟨ \mathbf{u}_1, \mathbf{u}_1 ⟩ = \dfrac{|1+3i|^2 + |3-i|^2}{4} = \dfrac{10 + 10}{4} = 5$
$\mathbf{e}_1 = \dfrac{\mathbf{u}_1}{\|\mathbf{u}_1\|} = (\dfrac{1+3i}{2\sqrt{5}}, \dfrac{3-i}{2\sqrt{5}})$

The Gram-Schmidt process can be expressed uniformly in matrix form.

§2.5 already demonstrated in code a first version of orthogonalizing two vectors; the definition in §9.1.3 extends the same idea to $k$ vectors and expresses it uniformly in the language of matrices: in $\mathbf{V} = \mathbf{QR}$ , the columns of $\mathbf{Q}$ are the normalized results of removing projections step by step, and $\mathbf{R}$ records the coefficients of each projection.

The QR decomposition applies column operations to $\mathbf{V}$ , so $\mathbf{Q}$ and $\mathbf{V}$ have the same image. This property is why the QR decomposition is widely used in numerical methods such as solving least-squares problems and computing eigenvalues.

The theory of inner product spaces provides a unified framework that systematically extends geometric notions—orthogonality, length, and angle—to abstract vector spaces. Within this framework, Gram-Schmidt orthogonalization gives a concrete algorithm for constructing orthogonal and orthonormal bases, which is of fundamental importance both for building orthogonal decompositions in theoretical analysis and for solving linear systems stably in numerical computation.

import numpy as np
import scipy.linalg as sclinalg

def gram_schmidt(vectors, rtol=None, atol=0.0):
    """MGS; input has shape (n, d), and each row is one vector.
    The tolerance is set from the largest vector norm in the input. Raises an explicit error on numerical degeneracy;
    this is a test of linear independence up to a tolerance, not the exact algebraic rank.
    """
    vectors = np.asarray(vectors)
    if vectors.ndim != 2:
        raise ValueError("vectors must be a two-dimensional array")
    n, d = vectors.shape
    dtype = np.result_type(vectors.dtype, np.float64)
    if n == 0:
        return np.empty((0, d), dtype=dtype)
    vectors = vectors.astype(dtype)
    if rtol is None:
        rtol = max(n, d) * np.finfo(float).eps
    if rtol < 0 or atol < 0:
        raise ValueError("tolerances must not be negative")
    threshold = atol + rtol * np.max(np.linalg.norm(vectors, axis=1))
    basis = []
    for v in vectors:
        w = v.copy()
        for q in basis:
            w -= np.vdot(q, w) * q
        norm = np.linalg.norm(w)
        if norm <= threshold:
            raise ValueError("input is numerically degenerate at the given tolerance; adjust the tolerance or use a rank-revealing decomposition")
        basis.append(w / norm)
    return np.asarray(basis).reshape(-1, d)

# Test: complex vectors
vectors = np.array([
    [1, 1j],
    [1+1j, 2]
])

orthonormal_basis = gram_schmidt(vectors)
print("Orthonormal basis:")
print(orthonormal_basis)

# Orthogonality check (using the conjugate transpose)
gram_matrix = orthonormal_basis @ orthonormal_basis.conj().T
print("\nOrthogonality check (Gram matrix):")
print(np.round(gram_matrix, 10))



# Compare with SciPy QR
Q, R = sclinalg.qr(vectors.T)
print("\nResult of the SciPy QR function:")
print(Q.T)

# Compare with NumPy's built-in QR
Q, R = np.linalg.qr(vectors.T)
print("\nResult of the NumPy QR function:")
print(Q.T)

# Compare with SciPy orth, which uses the singular value decomposition. The basis it returns is ordered by the distribution of "variance" or "energy" and usually does not line up with the directions of the original vectors at all.
scipy_orth = sclinalg.orth(vectors.T)
print("\nResult of the SciPy orth function, with an orthogonality check:")
print(scipy_orth.T)
gram_matrix = scipy_orth.conj().T @ scipy_orth
print(np.round(gram_matrix, 10))

§9.1 Summary

This section answered a fundamental question: when a vector space is extended from the real field to the complex field, how do we keep the geometric meaning of “length” and “angle”? The answer lies in introducing conjugation, rewriting the naive dot product as $⟨ \mathbf{u}|\mathbf{v}⟩ = \sum_i \overline{u_i} v_i$ , which guarantees that the inner product of a vector with itself, $⟨ \mathbf{v}|\mathbf{v}⟩$ , is always a nonnegative real number. This seemingly small correction is the foundation of the whole chapter.

On this basis, the section built the axiomatic system of inner product spaces (conjugate symmetry, linearity in the second argument, positive definiteness) and derived from it, naturally, unified definitions of norm, orthogonality, and angle. The Cauchy-Schwarz inequality $|⟨ \mathbf{u},\mathbf{v}⟩| \leq \|\mathbf{u}\|\|\mathbf{v}\|$ guarantees that the definition of angle is legitimate, and orthonormal bases reveal the elegant fact that “the coefficients of an orthogonal expansion are just inner-product projections.”

The Gram-Schmidt orthogonalization method (equivalent to the QR decomposition) provides a concrete algorithm for systematically constructing an orthonormal basis from any linearly independent set. It is not only a theoretical tool but also a core technique in numerical linear algebra for handling ill-conditioned matrices.

With the framework of inner product spaces in place, the next section naturally asks: which linear transformations preserve this geometric structure completely—that is, preserve both length and angle?

9.2 Orthogonal Matrices and Unitary Matrices¶

Orthogonal matrices and unitary matrices are special matrices that preserve the lengths and inner products of vectors, and they play an important role in many applications. Orthogonal matrices correspond to distance-preserving transformations of real vector spaces, and unitary matrices are their natural generalization to complex vector spaces. Both classes of matrices play central roles in numerical computation, quantum mechanics, signal processing, and other fields.

9.2.1 Definitions and Basic Properties of Orthogonal and Unitary Matrices¶

Definitions¶

An orthogonal matrix is the special case of a unitary matrix over the real numbers—when all the entries of a matrix are real, the conjugate transpose reduces to the transpose, and the definition of a unitary matrix reduces naturally to that of an orthogonal matrix.

Basic Properties¶

Property 3 (Properties of Orthogonal/Unitary Matrices)

For an orthogonal matrix $\mathbf{Q}$ or a unitary matrix $\mathbf{U}$ :

The inverse equals the transpose/conjugate transpose: $\mathbf{Q}^{-1} = \mathbf{Q}^\top$ or $\mathbf{U}^{-1} = \mathbf{U}^*$
Lengths of vectors are preserved: for every vector $\mathbf{x}$ , $\|\mathbf{Qx}\| = \|\mathbf{x}\|$ or $\|\mathbf{Ux}\| = \|\mathbf{x}\|$
Inner products of vectors are preserved: for all vectors $\mathbf{x}$ and $\mathbf{y}$ , $⟨ \mathbf{Qx}, \mathbf{Qy} ⟩ = ⟨ \mathbf{x}, \mathbf{y} ⟩$ or $⟨ \mathbf{Ux}, \mathbf{Uy} ⟩ = ⟨ \mathbf{x}, \mathbf{y} ⟩$
The determinant has absolute value 1: $|\det(\mathbf{Q})| = 1$ or $|\det(\mathbf{U})| = 1$
- For a real orthogonal matrix, $\det(\mathbf{Q}) = \pm 1$
- For a unitary matrix, $|\det(\mathbf{U})| = 1$ (that is, $\det(\mathbf{U})$ is a complex number of modulus 1)
◇Every eigenvalue has modulus 1:
- For a real orthogonal matrix, each eigenvalue is either $\pm 1$ or one of a conjugate pair of complex numbers $e^{\pm i\theta}$
- For a unitary matrix, every eigenvalue is a complex number $e^{i\theta}$ of modulus 1

The matrix 2-norm is the operator norm induced by the vector $\ell_2$ norm.

◇The 2-norm condition number is 1: $\kappa_2(\mathbf{Q}) = \|\mathbf{Q}\|_2\|\mathbf{Q}^{-1}\|_2 = 1$ (or $\kappa_2(\mathbf{U}) = 1$ ), which means that they are very stable in numerical computation—orthogonal/unitary transformations do not amplify numerical errors

Proof 4

We prove that orthogonal matrices preserve the lengths and inner products of vectors (the proof for unitary matrices is similar; simply replace the transpose by the conjugate transpose):

Preservation of length.

\|\mathbf{Qx}\|^2 = (\mathbf{Qx})^\top (\mathbf{Qx}) = \mathbf{x}^\top \mathbf{Q}^\top \mathbf{Q}\mathbf{x} = \mathbf{x}^\top \mathbf{I}\mathbf{x} = \mathbf{x}^\top \mathbf{x} = \|\mathbf{x}\|^2

(37)

Preservation of inner products.

⟨ \mathbf{Qx}, \mathbf{Qy} ⟩ = (\mathbf{Qx})^\top (\mathbf{Qy}) = \mathbf{x}^\top \mathbf{Q}^\top \mathbf{Q}\mathbf{y} = \mathbf{x}^\top \mathbf{y} = ⟨ \mathbf{x}, \mathbf{y} ⟩

(38)

The other properties follow directly from the definition and basic properties of determinants.

Note

Connection with Chapter 8. An orthogonal matrix is not necessarily orthogonally diagonalizable (that is, diagonalizable by a similarity transformation whose matrix is itself orthogonal). For example, the rotation matrix

\mathbf{Q}_\theta = \begin{pmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{pmatrix}

(39)

has the complex pair of eigenvalues $e^{\pm i\theta}$ (when $\theta \neq 0, \pi$ ). By the theory of Chapter 8 it can be diagonalized by a general similarity transformation (using complex eigenvectors), but not by an orthogonal transformation.

Real symmetric matrices (the subject of the next section), by contrast, are always orthogonally diagonalizable, which is why they are so important; note that being real symmetric and being orthogonal are different conditions, and neither includes the other.

Orthogonality of Columns and Rows¶

The columns and the rows of an orthogonal or unitary matrix each form an orthonormal basis:

Proof 5

Orthonormal columns. The condition $\mathbf{Q}^\top \mathbf{Q} = \mathbf{I}$ means that the column vectors of $\mathbf{Q}$ form an orthonormal basis. Specifically, let $\mathbf{q}_i$ be column $i$ of $\mathbf{Q}$ (so that row $i$ of $\mathbf{Q}^\top$ is $\mathbf{q}_i^\top$ ); then

(\mathbf{Q}^\top \mathbf{Q})_{ij} = \mathbf{q}_i^\top \mathbf{q}_j = \delta_{ij}

(40)

This is exactly the definition of an orthonormal basis.

Orthonormal rows. The condition $\mathbf{Q}\mathbf{Q}^\top = \mathbf{I}$ means that the row vectors of $\mathbf{Q}$ also form an orthonormal basis. This can be verified by a similar computation.

Key observation. The condition $\mathbf{Q}^\top \mathbf{Q} = \mathbf{I}$ implies $\mathbf{Q}\mathbf{Q}^\top = \mathbf{I}$ (and conversely). The proof is as follows:

From $\mathbf{Q}^\top \mathbf{Q} = \mathbf{I}$ we know that $\mathbf{Q}$ is invertible and $\mathbf{Q}^{-1} = \mathbf{Q}^\top$ . Therefore

\mathbf{Q}\mathbf{Q}^\top = \mathbf{Q}\mathbf{Q}^{-1} = \mathbf{I}

(41)

This shows that orthonormal columns automatically imply orthonormal rows, and conversely. This is a consequence of the symmetry in the definition of an orthogonal matrix.

Examples of Orthogonal Matrices¶

Example 4 (Basic Orthogonal Matrices)

Here are some basic types of orthogonal matrices:

The identity matrix $\mathbf{I}$ is the simplest orthogonal matrix
Permutation matrices (swapping rows or columns) are orthogonal matrices, for example
$\mathbf{P} = \begin{pmatrix} 0 & 1 \\ 1 & 0 \end{pmatrix}$
(42)
Reflection matrices (corresponding to reflections in a plane) are orthogonal matrices, for example the reflection across the $x$ -axis:
$\mathbf{R}_x = \begin{pmatrix} 1 & 0 \\ 0 & -1 \end{pmatrix}$
(43)
Rotation matrices, for example the matrix of a counterclockwise rotation of the plane through the angle $\theta$ :
$\mathbf{Q}_{\theta} = \begin{pmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{pmatrix}$
(44)
Givens rotation matrices, sparse orthogonal matrices that rotate only within two rows and two columns (the same as the simple rotation matrices mentioned in Chapter 3)
Householder reflection matrices, of the form $\mathbf{H} = \mathbf{I} - 2\mathbf{uu}^\top$ , where $\mathbf{u}$ is a unit vector
- The more general form is $\mathbf{H} = \mathbf{I} - \frac{2\mathbf{vv}^\top}{\mathbf{v}^\top\mathbf{v}}$ , where $\mathbf{v}$ is any nonzero vector

Every orthogonal matrix can be written as a product of several basic orthogonal transformations (such as rotations and reflections).

Exercise 1 (Verifying an Orthogonal Matrix)

Prove that the following matrix is orthogonal (Hint. Verify that $\mathbf{Q}^\top \mathbf{Q} = \mathbf{I}$ ):

\mathbf{Q} = \frac{1}{3} \begin{pmatrix} 1 & 2 & 2 \\ 2 & 1 & -2 \\ 2 & -2 & 1 \end{pmatrix}

(45)

9.2.2 The Geometric Meaning of Orthogonal and Unitary Transformations¶

Orthogonal and unitary transformations preserve the lengths of vectors and the angles between them, which gives them a clear geometric interpretation.

Definition of Orthogonal Transformations and Their Equivalence with Orthogonal Matrices¶

In a finite-dimensional Euclidean space, orthogonal transformations correspond one-to-one to orthogonal matrices:

Proof 6

We give a detailed proof:

( $\leftarrow$ ) An orthogonal matrix gives an orthogonal transformation.

Clearly an orthogonal matrix $\mathbf{Q}$ defines an orthogonal transformation $T$ . By Lemma 1 (the matrix inner product duality lemma), for all vectors $\mathbf{x}, \mathbf{y}$ :

⟨ T(\mathbf{x}), T(\mathbf{y}) ⟩ = ⟨ \mathbf{Qx}, \mathbf{Qy} ⟩ = \mathbf{x}^\top \mathbf{Q}^\top \mathbf{Q}\mathbf{y} = \mathbf{x}^\top \mathbf{y} = ⟨ \mathbf{x}, \mathbf{y} ⟩

(47)

( $\rightarrow$ ) An orthogonal transformation corresponds to an orthogonal matrix. Let $T$ be a linear transformation that preserves the inner product; we must prove that the matrix of $T$ with respect to the standard basis is an orthogonal matrix.

Let $\{\mathbf{e}_0, \mathbf{e}_1, \ldots, \mathbf{e}_{n-1}\}$ be the standard basis of $\mathbb{R}^n$ , and let $\mathbf{Q}$ be the matrix of $T$ with respect to the standard basis, that is, $T(\mathbf{x}) = \mathbf{Q}\mathbf{x}$ .

Column $j$ of $\mathbf{Q}$ is the coordinate vector of $T(\mathbf{e}_j)$ with respect to the standard basis. Since $T$ preserves the inner product,

⟨ T(\mathbf{e}_i), T(\mathbf{e}_j) ⟩ = ⟨ \mathbf{e}_i, \mathbf{e}_j ⟩ = \delta_{ij}

(48)

This means that the column vectors of $\mathbf{Q}$ form an orthonormal basis.

Now compute $\mathbf{Q}^\top \mathbf{Q}$ :

(\mathbf{Q}^\top \mathbf{Q})_{ij} = ⟨ T(\mathbf{e}_i), T(\mathbf{e}_j) ⟩ = \delta_{ij}

(49)

Therefore $\mathbf{Q}^\top \mathbf{Q} = \mathbf{I}$ , that is, $\mathbf{Q}$ is an orthogonal matrix.

The proof for unitary transformations on complex vector spaces is entirely analogous; simply replace the transpose by the conjugate transpose.

Proof 7

We use the fact that orthogonal transformations preserve the lengths of vectors and the angles between them.

1. Constructing an orthogonal transformation

Let $\mathbf{u}$ and $\mathbf{v}$ be two nonzero vectors in $\mathbb{R}^n$ . When they are linearly independent, they span a plane; when they are collinear, the question of their angle is trivial.

Using the Gram-Schmidt orthogonalization method, we can construct an orthogonal basis of $\mathbb{R}^n$ whose first two basis vectors $\mathbf{e}_0$ and $\mathbf{e}_1$ span exactly the plane $\text{span}\{\mathbf{u}, \mathbf{v}\}$ .

2. An orthogonal matrix rotates the plane onto the $xy$ -plane

Construct an orthogonal matrix $\mathbf{Q}$ (that is, one satisfying $\mathbf{Q}^\top \mathbf{Q} = \mathbf{I}$ ) such that

\mathbf{Q} \mathbf{e}_0 = \begin{pmatrix}1\\0\\0\\\vdots\\0\end{pmatrix}, \quad \mathbf{Q} \mathbf{e}_1 = \begin{pmatrix}0\\1\\0\\\vdots\\0\end{pmatrix}

(50)

Since $\mathbf{Q}$ maps each vector of the whole orthogonal basis to a new orthogonal basis, after the transformation by $\mathbf{Q}$ the plane originally spanned by $\mathbf{u}$ and $\mathbf{v}$ is mapped onto the $xy$ -plane.

3. Invariance of length and angle

Orthogonal matrices preserve the lengths and inner products of vectors:

\|\mathbf{Q}\mathbf{a}\| = \|\mathbf{a}\|, \quad (\mathbf{Q}\mathbf{a}) \cdot (\mathbf{Q}\mathbf{b}) = \mathbf{a} \cdot \mathbf{b}

(51)

Hence the angle $\theta$ between $\mathbf{u}$ and $\mathbf{v}$ (which satisfies $\cos\theta = \frac{\mathbf{u}\cdot \mathbf{v}}{\|\mathbf{u}\|\|\mathbf{v}\|}$ ) is unchanged by the transformation $\mathbf{Q}$ .

4. Reducing the problem to two dimensions

Since $\mathbf{Q}\mathbf{u}$ and $\mathbf{Q}\mathbf{v}$ both lie in the $xy$ -plane, they can be written as

\mathbf{Q}\mathbf{u} = \begin{pmatrix} u_0 \\ u_1 \\ 0 \\ \vdots \\ 0 \end{pmatrix}, \quad \mathbf{Q}\mathbf{v} = \begin{pmatrix} v_0 \\ v_1 \\ 0 \\ \vdots \\ 0 \end{pmatrix}

(52)

In the two-dimensional plane their inner product is $u_0v_0 + u_1v_1$ , and their lengths are $\sqrt{u_0^2 + u_1^2}$ and $\sqrt{v_0^2 + v_1^2}$ .

Since orthogonal transformations preserve inner products and lengths, the cosine formula in the original $n$ -dimensional space,

\cos\theta = \frac{\mathbf{u} \cdot \mathbf{v}}{\|\mathbf{u}\|\|\mathbf{v}\|}

(53)

is equivalent to the formula in the two-dimensional plane:

\cos\theta = \frac{u_0v_0 + u_1v_1}{\sqrt{u_0^2 + u_1^2}\sqrt{v_0^2 + v_1^2}}

(54)

5. Conclusion

By an orthogonal transformation, two vectors in $n$ -dimensional space can be rotated into a two-dimensional plane. Since orthogonal matrices preserve lengths and angles, the inner product and the lengths of the vectors are the same before and after the transformation, and so the angle between the two vectors stays the same. Therefore $\underline{\text{the cosine formula in } n \text{-dimensional space (defined via the inner product) has exactly the same form as the cosine formula in the two-dimensional plane}}$ .

Geometric Decomposition of Orthogonal Transformations¶

The geometric meaning of an orthogonal transformation depends on its eigenvalues and eigenvectors. On a real vector space, an orthogonal transformation can be decomposed into:

Rotations: corresponding to complex pairs of eigenvalues $e^{\pm i\theta}$ , acting as rotations in the corresponding two-dimensional subspaces
Reflections: corresponding to the eigenvalue -1, acting as reflections in the corresponding directions
The identity: corresponding to the eigenvalue +1, leaving the corresponding directions unchanged

◇9.2.3 Special Properties of Unitary Matrices and Quantum Computing¶

Unitary matrices are the generalization of orthogonal matrices to the complex field, and they play a central role in quantum mechanics and quantum computing.

The Importance of Unitary Matrices in Quantum Mechanics¶

This section considers pure states of closed systems and ideal quantum gates. Within this scope, a pure state of a physical system is described by a unit vector (a quantum state) in a complex vector space, and the evolution of the system is described by a unitary transformation. This is because:

Conservation of probability: unitary transformations preserve the norm of vectors, and therefore preserve the normalization condition $\|\psi\| = 1$ of quantum states
Reversibility: physical evolution (in the absence of measurement) is reversible, which corresponds to the invertibility of unitary matrices
Preservation of inner products: the inner product between normalized quantum states gives a probability amplitude, whose squared modulus gives the probability of the corresponding projective measurement; unitary transformations preserve inner products and therefore also preserve these probabilities

Common Unitary Matrices: Quantum Gates¶

Example 5 (Basic Unitary Matrices in Quantum Computing)

The most basic single-qubit gates in quantum computing are all $2 \times 2$ unitary matrices:

Pauli matrices (all unitary; $\sigma_x,\sigma_z$ are real orthogonal matrices):
$\sigma_x = \begin{pmatrix} 0 & 1 \\ 1 & 0 \end{pmatrix}, \quad \sigma_y = \begin{pmatrix} 0 & -i \\ i & 0 \end{pmatrix}, \quad \sigma_z = \begin{pmatrix} 1 & 0 \\ 0 & -1 \end{pmatrix}$
(56)
The Hadamard gate (a real unitary matrix):
$\mathbf{H} = \frac{1}{\sqrt{2}} \begin{pmatrix} 1 & 1 \\ 1 & -1 \end{pmatrix}$
(57)
To verify that $\mathbf{H}$ is unitary: since $\mathbf{H}$ is a real matrix, $\mathbf{H}^* = \mathbf{H}^\top$ :
$\mathbf{H}^\top \mathbf{H} = \frac{1}{2} \begin{pmatrix} 1 & 1 \\ 1 & -1 \end{pmatrix} \begin{pmatrix} 1 & 1 \\ 1 & -1 \end{pmatrix} = \frac{1}{2} \begin{pmatrix} 2 & 0 \\ 0 & 2 \end{pmatrix} = \mathbf{I}$
(58)
Phase gates (complex unitary matrices):
$\mathbf{S} = \begin{pmatrix} 1 & 0 \\ 0 & i \end{pmatrix}, \quad \mathbf{T} = \begin{pmatrix} 1 & 0 \\ 0 & e^{i\pi/4} \end{pmatrix}$
(59)
General rotation gates:
$\mathbf{R}_z(\theta) = \begin{pmatrix} e^{-i\theta/2} & 0 \\ 0 & e^{i\theta/2} \end{pmatrix}$
(60)

Note that the Hadamard gate is a real unitary matrix (and hence also an orthogonal matrix), while the phase gates $S,T$ and $R_z(\theta)$ for a general angle are non-real unitary matrices; for special angles $R_z$ may also be a real orthogonal matrix.

The Quantum Fourier Transform¶

Example 6 (The Quantum Fourier Transform Matrix)

The Hadamard gate can be viewed as the special case of the quantum Fourier transform (QFT) for a single qubit.

For $n$ qubits, the QFT matrix is a $2^n \times 2^n$ complex unitary matrix defined by

U_{QFT} = \frac{1}{\sqrt{2^n}} \begin{pmatrix} 1 & 1 & 1 & \cdots & 1 \\ 1 & \omega & \omega^2 & \cdots & \omega^{2^n-1} \\ 1 & \omega^2 & \omega^4 & \cdots & \omega^{2(2^n-1)} \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ 1 & \omega^{2^n-1} & \omega^{2(2^n-1)} & \cdots & \omega^{(2^n-1)^2} \end{pmatrix}

(61)

where $\omega = e^{2\pi i/2^n}$ is a $2^n$ -th root of unity.

The QFT plays a key role in many quantum algorithms, especially in solving periodicity problems, as in Shor’s algorithm (integer factorization) and the quantum phase estimation algorithm.

From the point of view of matrix theory, the QFT can be regarded as a special Vandermonde matrix and inherits many of the properties of Vandermonde matrices.

9.2.4 Numerical Stability and Applications¶

Orthogonal matrices have important stability advantages in matrix computations.

Why Orthogonal Matrices Are Numerically Stable¶

The condition number of an orthogonal matrix, $\kappa_2(\mathbf{Q}) = \|\mathbf{Q}\|_2\|\mathbf{Q}^{-1}\|_2 = 1$ , is the smallest possible value among all matrices. This means:

Errors are not amplified: when we compute $\mathbf{Qx}$ , any rounding error in the input $\mathbf{x}$ is not amplified
Numerical accuracy is maintained: in iterative algorithms, orthogonal transformations do not amplify the 2-norm of existing errors, although new rounding errors produced by repeated operations may still accumulate
Ill-conditioned problems are avoided: using orthogonal matrices avoids many ill-conditioned situations in numerical linear algebra

The Frobenius Norm and Orthogonal Invariance

The Frobenius norm of a matrix is defined as the square root of the sum of the squared moduli of all its entries,

\|\mathbf{A}\|_F = \sqrt{\sum_{i}\sum_{j} |a_{ij}|^2} = \sqrt{\operatorname{tr}(\mathbf{A}^*\mathbf{A})}

(62)

It regards an $m \times n$ matrix as a vector in $\mathbb{C}^{mn}$ and takes its $\ell_2$ length. Orthogonal (unitary) similarity transformations leave the Frobenius norm unchanged: for an orthogonal matrix $\mathbf{Q}$ , $\|\mathbf{Q}^\top\mathbf{A}\mathbf{Q}\|_F = \|\mathbf{A}\|_F$ (which follows immediately from the cyclic property of $\operatorname{tr}$ and $\mathbf{Q}^\top\mathbf{Q}=\mathbf{I}$ ). This is precisely how “orthogonal transformations do not amplify errors” shows up at the level of matrices. The full theory of matrix norms (including operator-induced norms) will be developed systematically in §11.2 of Chapter 11.

Main Applications¶

Orthogonal and unitary matrices play central roles in the following fields:

Numerical linear algebra:
- QR decomposition: factoring a matrix into the product of an orthogonal matrix and an upper triangular matrix
- Singular value decomposition (SVD): the subject of Chapter 11
- Iterative methods: such as Arnoldi iteration and the Lanczos method
Computer graphics:
- Rotation, scaling, and translation of 3D objects
- Viewing transformations and projection
- Interpolation in animation (SLERP: spherical linear interpolation)
Signal processing:
- The discrete Fourier transform (DFT) and its fast algorithm (FFT)
- The discrete cosine transform (DCT): the basis of JPEG compression
- Wavelet transforms: multiscale signal analysis
Quantum computing:
- Quantum gate operations (such as the various unitary matrices introduced in this section)
- Quantum algorithm design (such as Shor’s algorithm and Grover’s algorithm)
- Simulation of quantum state evolution
Data analysis and machine learning:
- Principal component analysis (PCA): discussed in detail in Chapter 11
- Independent component analysis (ICA)
- The whitening transformation

Orthogonal and unitary matrices are a natural extension of the theory of inner product spaces. They preserve inner products, lengths, and angles, and geometrically they correspond to rigid motions (rotations and reflections). An orthogonal transformation does not change the geometric structure of space, only the angle from which we view it. Understanding the properties of orthogonal transformations is essential for understanding many advanced concepts and applications in linear algebra; in particular, it lays the foundation for the spectral theorem for symmetric matrices in the next section, the introduction to quantum mechanics in Chapter 10, and the singular value decomposition in Chapter 11.

§9.2 Summary

The question this section answered is: in an inner product space, which matrices give linear transformations that leave the geometric structure completely unchanged? The answer is orthogonal matrices ( $\mathbf{Q}^\top\mathbf{Q}=\mathbf{I}$ ) and their complex generalization—unitary matrices ( $\mathbf{U}^*\mathbf{U}=\mathbf{I}$ ). Their inverse equals their transpose or conjugate transpose, their 2-norm condition number is always 1, and they are the most stable class of matrices in numerical computation.

The geometric core of this section is that an orthogonal transformation is a composition of rotations and reflections, and a unitary transformation is the analogue of a “rotation” in complex space. In both cases the eigenvalues have modulus 1; from the “measurement viewpoint,” they change only the coordinate system of observation, not the object itself. The columns and the rows of an orthogonal matrix are both orthonormal bases, a property that follows directly from the definition $\mathbf{Q}^\top\mathbf{Q}=\mathbf{I}$ , and from $(\det\mathbf{Q})^2=\det(\mathbf{Q}^\top\mathbf{Q})=1$ it follows that its determinant must have absolute value 1.

It is worth special attention that an orthogonal matrix itself need not be orthogonally diagonalizable—the complex eigenvalues of a rotation matrix reveal this detail. The special objects that truly can be “diagonalized by an orthogonal basis” are the protagonists of the next section—symmetric matrices and Hermitian matrices.

9.3 Symmetric Matrices and Hermitian Matrices¶

Symmetric matrices and Hermitian matrices are two special and important classes of matrices in linear algebra. Symmetric matrices play a central role in real vector spaces, and Hermitian matrices are the natural generalization of symmetric matrices to complex vector spaces. Both classes hold a fundamental place in physics, engineering, and data analysis. Their eigenvalues are all real, and an orthonormal basis of eigenvectors can be chosen for them, which gives them unique advantages in representing physical systems, analyzing the structure of data, and designing algorithms.

9.3.1 Eigenvalues and Eigenvectors of Symmetric and Hermitian Matrices¶

Definitions of Symmetric and Hermitian Matrices¶

Hermitian matrices are the natural generalization of symmetric matrices; for real matrices the two notions coincide (because the conjugate of a real number is the number itself).

Properties of the Eigenvalues and Eigenvectors¶

The eigenvalues and eigenvectors of symmetric and Hermitian matrices have special properties:

Proof 9

We prove that the eigenvalues of a Hermitian matrix $\mathbf{A}$ are all real (the symmetric case can be regarded as a special case).

Let $\lambda$ be an eigenvalue of $\mathbf{A}$ and $\mathbf{v}$ a corresponding eigenvector, that is, $\mathbf{A}\mathbf{v} = \lambda\mathbf{v}$ .

Compute $⟨ \mathbf{v}|\mathbf{A}|\mathbf{v} ⟩$
By the eigenvalue equation $\mathbf{A}\mathbf{v} = \lambda\mathbf{v}$ , we have

⟨ \mathbf{v}|\mathbf{A}|\mathbf{v} ⟩ = ⟨ \mathbf{v}|\lambda\mathbf{v} ⟩ = \lambda⟨ \mathbf{v}|\mathbf{v} ⟩ = \lambda\|\mathbf{v}\|^2

(65)

Use the Hermitian property and the duality lemma
Since $\mathbf{A}$ is a Hermitian matrix, it satisfies $\mathbf{A}^* = \mathbf{A}$ . By the matrix inner product duality lemma Lemma 1:

⟨ \mathbf{v}|\mathbf{A}|\mathbf{v} ⟩ = \overline{⟨ \mathbf{v}|\mathbf{A}^*|\mathbf{v} ⟩}

(66)

Since $\mathbf{A}^* = \mathbf{A}$ ,

⟨ \mathbf{v}|\mathbf{A}|\mathbf{v} ⟩ = \overline{⟨ \mathbf{v}|\mathbf{A}|\mathbf{v} ⟩}

(67)

Expand the inner product

Using $\mathbf{A}\mathbf{v} = \lambda\mathbf{v}$ , we can write the right-hand side as

\overline{⟨ \mathbf{v}|\mathbf{A}|\mathbf{v} ⟩} = \overline{⟨ \mathbf{v}|\lambda\mathbf{v} ⟩} = \overline{\lambda⟨ \mathbf{v}|\mathbf{v} ⟩} = \overline{\lambda}\|\mathbf{v}\|^2

(68)

Conclude that the eigenvalue is real
Combining the results of steps 1 and 3:

\lambda\|\mathbf{v}\|^2 = \overline{\lambda}\|\mathbf{v}\|^2

(69)

Since $\mathbf{v}$ is an eigenvector, $\mathbf{v} \neq \mathbf{0}$ , that is, $\|\mathbf{v}\|^2 > 0$ . Dividing both sides by $\|\mathbf{v}\|^2$ gives

\underline{\lambda = \overline{\lambda}}

(70)

This shows that $\lambda$ is real.

Eigenvectors belonging to different eigenvalues are orthogonal

Let $\mathbf{A}\mathbf{u} = \lambda\mathbf{u}$ and $\mathbf{A}\mathbf{v} = \mu\mathbf{v}$ with $\lambda \neq \mu$ . Then

\lambda\langle\mathbf{u},\mathbf{v}\rangle = \langle\mathbf{A}\mathbf{u},\mathbf{v}\rangle = \langle\mathbf{u},\mathbf{A}^*\mathbf{v}\rangle = \langle\mathbf{u},\mathbf{A}\mathbf{v}\rangle = \mu\langle\mathbf{u},\mathbf{v}\rangle

(71)

Since $\lambda \neq \mu$ , $\underline{\langle\mathbf{u},\mathbf{v}\rangle = 0}$ .

The Spectral Decomposition Theorem¶

Theorem 5 (Spectral Decomposition of Symmetric/Hermitian Matrices)

Let $\mathbf{A} \in \mathbb{R}^{n \times n}$ be a symmetric matrix. Then there exist an orthogonal matrix $\mathbf{P} \in \mathbb{R}^{n \times n}$ and a diagonal matrix $\mathbf{\Lambda} \in \mathbb{R}^{n \times n}$ such that

\mathbf{A} = \mathbf{P}\mathbf{\Lambda}\mathbf{P}^\top

(72)

For a Hermitian matrix $\mathbf{A} \in \mathbb{C}^{n \times n}$ , there exist a unitary matrix $\mathbf{U} \in \mathbb{C}^{n \times n}$ and a real diagonal matrix $\mathbf{\Lambda} \in \mathbb{R}^{n \times n}$ such that

\mathbf{A} = \mathbf{U}\mathbf{\Lambda}\mathbf{U}^*

(73)

This decomposition is called the spectral decomposition, where $\mathbf{\Lambda} = \text{diag}(\lambda_0, \lambda_1, \ldots, \lambda_{n-1})$ and the $\lambda_i$ are the eigenvalues of $\mathbf{A}$ .

The spectral decomposition can be rewritten in outer-product form:

\mathbf{A} = \sum_{i=0}^{n-1} \lambda_i \mathbf{q}_i \mathbf{q}_i^* = \sum_{i=0}^{n-1} \lambda_i |\mathbf{q}_i⟩⟨\mathbf{q}_i|

(74)

where $\mathbf{q}_i$ is the $i$ -th normalized eigenvector of $\mathbf{A}$ . This shows that a symmetric/Hermitian matrix can be expressed as a linear combination of rank-one matrices.

Unlike a general matrix, a symmetric matrix never has missing eigenvectors.

Proof 10

Prerequisites

The eigenvalues of a Hermitian matrix are all real.
Eigenvectors belonging to different eigenvalues are orthogonal.
A unitary matrix $\mathbf{V}$ satisfies $\mathbf{V}^{*}\mathbf{V} = \mathbf{V}\mathbf{V}^* = \mathbf{I}$ , where $\mathbf{I}$ is the identity matrix.

Proof In the inductive proof below, $\mathbf{V}$ and $\mathbf{D}$ denote the unitary matrix and the real diagonal matrix being constructed, that is, the $\mathbf{U}$ and $\mathbf{\Lambda}$ of the Hermitian case in the statement of the theorem ( $\mathbf{V} = \mathbf{U}$ , $\mathbf{D} = \mathbf{\Lambda}$ ).

We proceed by mathematical induction.

Base case When $n = 1$ , $\mathbf{A} = [a]$ with $a \in \mathbb{R}$ (because the diagonal entries of a Hermitian matrix must be real). Then $\mathbf{V} = [1]$ and $\mathbf{D} = [a]$ satisfy the theorem, that is, $\mathbf{A} = \mathbf{V}\mathbf{D}\mathbf{V}^{*}$ .

Induction hypothesis Assume that the theorem holds for every $j \times j$ Hermitian matrix; that is, every $j \times j$ Hermitian matrix $\mathbf{B}$ can be written as $\mathbf{B} = \mathbf{U}\mathbf{\Lambda}\mathbf{U}^{*}$ , where $\mathbf{U}$ is a unitary matrix and $\mathbf{\Lambda}$ is a diagonal matrix.

Inductive step Now consider a $(j+1) \times (j+1)$ Hermitian matrix $\mathbf{A}$ .

First, since $\mathbf{A}$ is a Hermitian matrix, by the fundamental theorem of algebra in §8.1, $\mathbf{A}$ has at least one eigenvalue $\lambda \in \mathbb{R}$ with a corresponding eigenvector $\mathbf{v}_{\lambda} \in \mathbb{C}^{j+1}$ . We may normalize $\mathbf{v}_{\lambda}$ to be a unit vector, that is, $\|\mathbf{v}_{\lambda}\| = 1$ . We can then construct a unitary matrix $\mathbf{P} = [\mathbf{v}_{\lambda} | \mathbf{v}_{\lambda}^{\perp}]$ , where $\mathbf{v}_{\lambda}^{\perp}$ consists of $j$ unit vectors orthogonal to $\mathbf{v}_{\lambda}$ that together with it form an orthonormal basis of $\mathbb{C}^{j+1}$ . (This can be constructed using the QR decomposition.)
Consider the matrix $\mathbf{P}^{*}\mathbf{A}\mathbf{P}$ :

\mathbf{P}^{*}\mathbf{A}\mathbf{P} = \begin{pmatrix} \mathbf{v}_{\lambda}^* \\ (\mathbf{v}_{\lambda}^{\perp})^{*} \end{pmatrix} \mathbf{A} \begin{pmatrix} \mathbf{v}_{\lambda} & \mathbf{v}_{\lambda}^{\perp} \end{pmatrix}

(75)

Compute each block of $\mathbf{P}^{*}\mathbf{A}\mathbf{P}$ :

a. The upper-left block: $\mathbf{v}_{\lambda}^{*}\mathbf{A}\mathbf{v}_{\lambda} = \lambda\mathbf{v}_{\lambda}^{*}\mathbf{v}_{\lambda} = \lambda$ , because $\mathbf{v}_{\lambda}$ is a unit vector.

b. The upper-right block: $\mathbf{v}_{\lambda}^{*}\mathbf{A}\mathbf{v}_{\lambda}^{\perp}$ . Note that for every $\mathbf{w} \in \text{Col}(\mathbf{v}_{\lambda}^{\perp})$ we have

\mathbf{v}_{\lambda}^{*}\mathbf{A}\mathbf{w} = (\mathbf{A}\mathbf{v}_{\lambda})^{*}\mathbf{w} = (\lambda\mathbf{v}_{\lambda})^{*}\mathbf{w} = \lambda^* \mathbf{v}_{\lambda}^{*}\mathbf{w} = \mathbf{0}

(76)

c. Similarly, the lower-left block: $(\mathbf{v}_{\lambda}^{\perp})^{*}\mathbf{A}\mathbf{v}_{\lambda} = (\mathbf{v}_{\lambda}^{\perp})^{*}\lambda\mathbf{v}_{\lambda} = \lambda(\mathbf{v}_{\lambda}^{\perp})^{*}\mathbf{v}_{\lambda} = \mathbf{0}$ , because $\mathbf{v}_{\lambda}^{\perp}$ is orthogonal to $\mathbf{v}_{\lambda}$ .

d. The lower-right block: $(\mathbf{v}_{\lambda}^{\perp})^{*}\mathbf{A}\mathbf{v}_{\lambda}^{\perp} = \mathbf{S}_j$ , where $\mathbf{S}_j$ is a $j \times j$ Hermitian matrix.

Therefore $\mathbf{P}^{*}\mathbf{A}\mathbf{P}$ has the form

\mathbf{P}^{*}\mathbf{A}\mathbf{P} = \begin{pmatrix} \lambda & \mathbf{0} \\ \mathbf{0} & \mathbf{S}_j \end{pmatrix}

(77)

By the induction hypothesis, the $j \times j$ Hermitian matrix $\mathbf{S}_j$ can be decomposed as $\mathbf{S}_j = \mathbf{U}_j \mathbf{\Lambda}_j \mathbf{U}^* _j$ , where $\mathbf{U}_j$ is a $j \times j$ unitary matrix and $\mathbf{\Lambda}_j$ is a $j \times j$ diagonal matrix.

Define the matrix $\mathbf{Q} = \begin{pmatrix} 1 & \mathbf{0} \\ \mathbf{0} & \mathbf{U}_j \end{pmatrix}$ . Then $\mathbf{Q}$ is a $(j+1) \times (j+1)$ unitary matrix, and

\mathbf{D} := \mathbf{Q}^{*}\mathbf{P}^{*}\mathbf{A}\mathbf{P}\mathbf{Q} = \begin{pmatrix} 1 & \mathbf{0} \\ \mathbf{0} & \mathbf{U}_j^{*} \end{pmatrix} \begin{pmatrix} \lambda & \mathbf{0} \\ \mathbf{0} & \mathbf{S}_j \end{pmatrix} \begin{pmatrix} 1 & \mathbf{0} \\ \mathbf{0} & \mathbf{U}_j \end{pmatrix} = \begin{pmatrix} \lambda & \mathbf{0} \\ \mathbf{0} & \mathbf{\Lambda}_j \end{pmatrix}

(78)

Therefore we obtain

\mathbf{A} = \mathbf{P}\mathbf{Q}\mathbf{D}\mathbf{Q}^{*}\mathbf{P}^* = \mathbf{V}\mathbf{D}\mathbf{V}^{*}

(79)

where $\mathbf{V} = \mathbf{P}\mathbf{Q}$ is a $(j+1) \times (j+1)$ unitary matrix and $\mathbf{D} =\begin{pmatrix} \lambda & \mathbf{0} \\ \mathbf{0} & \mathbf{\Lambda}_j \end{pmatrix}$ is a $(j+1) \times (j+1)$ diagonal matrix.

This completes the inductive step and proves that the theorem holds for every $(j+1) \times (j+1)$ Hermitian matrix.

Conclusion By mathematical induction, we have proved that for every $n \times n$ Hermitian matrix $\mathbf{A}$ there exist a unitary matrix $\mathbf{V}$ and a diagonal matrix $\mathbf{D}$ such that $\underline{\mathbf{A} = \mathbf{V}\mathbf{D}\mathbf{V}^{*}}$ .

Example 7 (Spectral Decomposition of a Symmetric Matrix)

Consider the symmetric matrix

\mathbf{A} = \begin{pmatrix} 4 & 1 \\ 1 & 4 \end{pmatrix}

(81)

Find its eigenvalues and eigenvectors:

Characteristic equation: $\det(\mathbf{A} - \lambda\mathbf{I}) = (4-\lambda)^2 - 1 = \lambda^2 - 8\lambda + 15 = 0$

Solving gives the eigenvalues $\lambda_0 = 5$ , $\lambda_1 = 3$

For $\lambda_0 = 5$ , the unit eigenvector is $\mathbf{q}_0 = \frac{1}{\sqrt{2}}\begin{pmatrix} 1 \\ 1 \end{pmatrix}$

For $\lambda_1 = 3$ , the unit eigenvector is $\mathbf{q}_1 = \frac{1}{\sqrt{2}}\begin{pmatrix} -1 \\ 1 \end{pmatrix}$

Construct the orthogonal matrix and the diagonal matrix:

\mathbf{P} = \begin{pmatrix} \frac{1}{\sqrt{2}} & -\frac{1}{\sqrt{2}} \\ \frac{1}{\sqrt{2}} & \frac{1}{\sqrt{2}} \end{pmatrix}, \quad \mathbf{\Lambda} = \begin{pmatrix} 5 & 0 \\ 0 & 3 \end{pmatrix}

(82)

Verify the spectral decomposition: $\mathbf{A} = \mathbf{P}\mathbf{\Lambda}\mathbf{P}^\top$

Example 8 (Spectral Decomposition of a Hermitian Matrix)

Consider the Hermitian matrix

\mathbf{A} = \begin{pmatrix} 2 & i \\ -i & 2 \end{pmatrix}

(83)

Find its eigenvalues and eigenvectors:

Characteristic equation: $\det(\mathbf{A} - \lambda\mathbf{I}) = (2-\lambda)^2 - (-i)(i) = \lambda^2 - 4\lambda + 3 = 0$

Solving gives the eigenvalues $\lambda_0 = 3$ , $\lambda_1 = 1$ (both real, as the properties of Hermitian matrices require)

For $\lambda_0 = 3$ , the unit eigenvector is $\mathbf{u}_0 = \frac{1}{\sqrt{2}}\begin{pmatrix} 1 \\ -i \end{pmatrix}$

For $\lambda_1 = 1$ , the unit eigenvector is $\mathbf{u}_1 = \frac{1}{\sqrt{2}}\begin{pmatrix} 1 \\ i \end{pmatrix}$

Construct the unitary matrix and the diagonal matrix:

\mathbf{U} = \frac{1}{\sqrt{2}}\begin{pmatrix} 1 & 1 \\ -i & i \end{pmatrix}, \quad \mathbf{\Lambda} = \begin{pmatrix} 3 & 0 \\ 0 & 1 \end{pmatrix}

(84)

Verify the spectral decomposition: $\mathbf{A} = \mathbf{U}\mathbf{\Lambda}\mathbf{U}^*$

import numpy as np

np.set_printoptions(precision=4, suppress=True, linewidth=100)

def print_header(title):
    print("=" * 60)
    print(f"  {title}")
    print("=" * 60)

def print_step(step, desc):
    print(f"\n▶ Step {step}: {desc}")
    print("-" * 40)

def compare_print(label, actual, expected_desc):
    print(f"\n[{label}]")
    print(f"  Computed value:\n{actual}")
    print(f"  Expected value: {expected_desc}")

# ============================================================
print_header("Example 9.3.1a | Spectral Decomposition of a Real Symmetric Matrix  A = PΛP^T")
# ============================================================

# --- Step 1 ---
print_step(1, "Define the symmetric matrix A")
A = np.array([[4, 1],
              [1, 4]], dtype=float)
print(A)

# --- Step 2 ---
print_step(2, "Compute the eigenvalues and eigenvectors (eigh guarantees real output)")
eigenvalues, P = np.linalg.eigh(A)
idx = np.argsort(eigenvalues)[::-1]   # descending order: λ₀ ≥ λ₁
eigenvalues = eigenvalues[idx]
P = P[:, idx]
print(f"  Eigenvalues Λ = diag({eigenvalues})")
print(f"  Expected from the text: λ₀ = 5,  λ₁ = 3  ✓")
print(f"\n  Orthogonal matrix P (columns = normalized eigenvectors):")
print(P)
print(f"  Expected from the text: q₀ = (1, 1)/√2,  q₁ = (-1, 1)/√2")

# --- Step 3 ---
print_step(3, "Verify that P is orthogonal: P^T P = I")
compare_print("P^T P", np.round(P.T @ P, 10), "the identity matrix I")

# --- Step 4 ---
print_step(4, "Reconstruct from the spectral decomposition: A_rec = P Λ P^T")
A_rec = P @ np.diag(eigenvalues) @ P.T
compare_print("A_rec = PΛP^T", A_rec,
              "the original matrix A =\n  [[4. 1.]\n   [1. 4.]]")
print(f"\n  Maximum reconstruction error: {np.max(np.abs(A - A_rec)):.2e}")

# --- Step 5 ---
print_step(5, "Outer-product expansion: A = λ₀ q₀q₀^T + λ₁ q₁q₁^T")
term0 = eigenvalues[0] * np.outer(P[:, 0], P[:, 0])
term1 = eigenvalues[1] * np.outer(P[:, 1], P[:, 1])
print(f"  λ₀·q₀q₀^T =\n{term0}")
print(f"  λ₁·q₁q₁^T =\n{term1}")
print(f"  Sum =\n{np.round(term0 + term1, 10)}")

# ============================================================
print_header("Example 9.3.1b | Spectral Decomposition of a Hermitian Matrix  A = UΛU*")
# ============================================================

# --- Step 1 ---
print_step(1, "Define the Hermitian matrix A and verify A = A*")
A_h = np.array([[ 2,  1j],
                [-1j,  2]], dtype=complex)
print(A_h)
print(f"  A = A*: {np.allclose(A_h, A_h.conj().T)}")

# --- Step 2 ---
print_step(2, "Compute the eigenvalues and eigenvectors (eigh guarantees real eigenvalues)")
eigenvalues_h, U = np.linalg.eigh(A_h)
idx = np.argsort(eigenvalues_h)[::-1]   # descending order: λ₀ ≥ λ₁
eigenvalues_h = eigenvalues_h[idx]
U = U[:, idx]
print(f"  Eigenvalues (all real): {eigenvalues_h}")
print(f"  Expected from the text: λ₀ = 3,  λ₁ = 1  ✓")
print(f"\n  Unitary matrix U (columns = normalized eigenvectors):")
print(U)
print(f"  Expected from the text, ordered by λ=3,1: u₀ = (1, -i)/√2, u₁ = (1, i)/√2; eigh returns ascending order and may give different phases")

# --- Step 3 ---
print_step(3, "Verify that U is unitary: U*U = I")
compare_print("U*U", np.round(U.conj().T @ U, 10), "the identity matrix I")

# --- Step 4 ---
print_step(4, "Reconstruct from the spectral decomposition: A_rec = U Λ U*")
A_rec_h = U @ np.diag(eigenvalues_h) @ U.conj().T
compare_print("A_rec = UΛU*", A_rec_h, "the original matrix A_h")
print(f"\n  Maximum reconstruction error: {np.max(np.abs(A_h - A_rec_h)):.2e}")

# --- Step 5 ---
print_step(5, "Verify that eigenvectors for different eigenvalues are orthogonal: ⟨u₀|u₁⟩ = u₀*·u₁")
inner = np.vdot(U[:, 0], U[:, 1])
print(f"  ⟨u₀|u₁⟩ = {inner:.2e}  (should be 0, by property 2 of the theorem)")

9.3.2 Positive Definite and Positive Semidefinite Matrices¶

Definition of Positive Definiteness¶

Equivalent Conditions for Positive Definiteness¶

Positive definiteness can be characterized in several ways:

Theorem 6 (◆Equivalent Conditions for Positive Definiteness)

For a symmetric matrix $\mathbf{A} \in \mathbb{R}^{n \times n}$ , the following conditions are equivalent:

$\mathbf{A}$ is positive definite
All eigenvalues of $\mathbf{A}$ are positive
All leading principal minors of $\mathbf{A}$ are positive (the $k$ -th leading principal minor is the determinant of the upper-left $(k+1) \times (k+1)$ submatrix, $k \in \{0, \ldots, n-1\}$ ; Sylvester’s criterion)
$\mathbf{A}$ can be factored as $\mathbf{A} = \mathbf{L}\mathbf{L}^\top$ , where $\mathbf{L}$ is lower triangular with positive diagonal entries (the Cholesky decomposition)
There exists a nonsingular square matrix $\mathbf{C}$ such that $\mathbf{A} = \mathbf{C}^\top \mathbf{C}$
$\mathbf{A} = \mathbf{B}^\top \mathbf{B}$ , where $\mathbf{B}$ is a matrix with linearly independent columns

Proof 11

The proof uses a cyclic chain of implications:

(1) \leftrightarrow (2) \rightarrow (3) \rightarrow (4) \rightarrow (5) \rightarrow (6) \rightarrow (1)

(85)

That is, we first prove $(1) \leftrightarrow (2)$ , then proceed in order along $(2) \rightarrow (3) \rightarrow (4) \rightarrow (5) \rightarrow (6) \rightarrow (1)$ , finally closing the cycle of equivalences.

Step 1: (1) $\leftrightarrow$ (2)

By the spectral theorem for real symmetric matrices, there exists an orthogonal matrix $\mathbf{Q}$ such that

\mathbf{A} = \mathbf{Q} \mathbf{\Lambda} \mathbf{Q}^\top, \quad \mathbf{\Lambda} = \operatorname{diag}(\lambda_0, \lambda_1, \ldots, \lambda_{n-1})

(86)

For any $\mathbf{x} \neq \mathbf{0}$ , let $\mathbf{y} = \mathbf{Q}^\top \mathbf{x} \neq \mathbf{0}$ (since $\mathbf{Q}$ is orthogonal and hence invertible); then

\mathbf{x}^\top \mathbf{A} \mathbf{x} = \mathbf{y}^\top \mathbf{\Lambda} \mathbf{y} = \sum_{i=0}^{n-1} \lambda_i y_i^2

(87)

$(1) \rightarrow (2)$ : if some $\lambda_k \leq 0$ , take $\mathbf{x} = \mathbf{Q}\mathbf{e}_k \neq \mathbf{0}$ ; then $\mathbf{x}^\top \mathbf{A}\mathbf{x} = \lambda_k \leq 0$ , contradicting positive definiteness.
$(2) \rightarrow (1)$ : if all $\lambda_i > 0$ , then $\sum \lambda_i y_i^2 > 0$ for every $\mathbf{y} \neq \mathbf{0}$ , so $\mathbf{A}$ is positive definite.

Step 2: (2) $\rightarrow$ (3)

For any $k \in \{0, \ldots, n-1\}$ , let $\mathbf{A}_k$ be the upper-left $(k+1) \times (k+1)$ leading principal submatrix of $\mathbf{A}$ . For any $\mathbf{y} \in \mathbb{R}^{k+1}, \mathbf{y} \neq \mathbf{0}$ , let

\mathbf{x} = \begin{pmatrix} \mathbf{y} \\ \mathbf{0} \end{pmatrix} \in \mathbb{R}^n

(88)

Then $\mathbf{x} \neq \mathbf{0}$ , and since $\mathbf{A}$ is positive definite (by (1) $\leftrightarrow$ (2)),

\mathbf{y}^\top \mathbf{A}_k \mathbf{y} = \mathbf{x}^\top \mathbf{A} \mathbf{x} > 0

(89)

Hence $\mathbf{A}_k$ is positive definite and its eigenvalues are all positive, so

\det(\mathbf{A}_k) = \prod_i \lambda_i(\mathbf{A}_k) > 0

(90)

Step 3: (3) $\rightarrow$ (4) (existence of the Cholesky decomposition)

We use induction on $n$ . In the induction below, the subscripts in $\mathbf{A}_{n-1}$ and $\mathbf{L}_{n-1}$ denote the size $n-1$ , not the 0-based numbering of the leading principal minors in the previous step.

Base case $n = 1$ : condition (3) gives $a_{00} > 0$ ; take $\mathbf{L} = [\sqrt{a_{00}}]$ .
Induction hypothesis: the Cholesky decomposition exists for every $(n-1) \times (n-1)$ symmetric matrix satisfying condition (3).
Inductive step: partition $\mathbf{A}$ as

\mathbf{A} = \begin{pmatrix} \mathbf{A}_{n-1} & \mathbf{b} \\ \mathbf{b}^\top & a_{n-1,n-1} \end{pmatrix}

(91)

All the leading principal minors of $\mathbf{A}_{n-1}$ are inherited from $\mathbf{A}$ , so by the induction hypothesis $\mathbf{A}_{n-1} = \mathbf{L}_{n-1}\mathbf{L}_{n-1}^\top$ .

Let $\mathbf{l} = \mathbf{L}_{n-1}^{-1}\mathbf{b}$ and consider the Schur complement (see Chapter 5)

s = a_{n-1,n-1} - \mathbf{b}^\top \mathbf{A}_{n-1}^{-1} \mathbf{b} = a_{n-1,n-1} - \mathbf{l}^\top \mathbf{l}

(92)

The block determinant formula gives $\det(\mathbf{A}) = \det(\mathbf{A}_{n-1}) \cdot s$ , and since $\det(\mathbf{A}) > 0$ and $\det(\mathbf{A}_{n-1}) > 0$ , we get $s > 0$ . Take

\mathbf{L} = \begin{pmatrix} \mathbf{L}_{n-1} & \mathbf{0} \\ \mathbf{l}^\top & \sqrt{s} \end{pmatrix}

(93)

A direct computation verifies $\mathbf{L}\mathbf{L}^\top = \mathbf{A}$ , and the diagonal entries of $\mathbf{L}$ are all positive. This completes the induction.

Step 4: (4) $\rightarrow$ (5)

Let $\mathbf{C} = \mathbf{L}^\top$ . This is an upper triangular square matrix whose diagonal entries are all positive, so $\mathbf{C}$ is nonsingular. A direct computation gives

\mathbf{C}^\top \mathbf{C} = \mathbf{L} \mathbf{L}^\top = \mathbf{A}

(94)

Step 5: (5) $\rightarrow$ (6)

Let $\mathbf{B} = \mathbf{C}$ . Since $\mathbf{C}$ is a nonsingular square matrix, $\ker(\mathbf{C}) = \{\mathbf{0}\}$ , so its columns are linearly independent. Thus

\mathbf{A} = \mathbf{C}^\top \mathbf{C} = \mathbf{B}^\top \mathbf{B}

(95)

and the columns of $\mathbf{B}$ are linearly independent, so condition (6) holds.

Step 6: (6) $\rightarrow$ (1)

The columns of $\mathbf{B}$ being linearly independent is equivalent to $\ker(\mathbf{B}) = \{\mathbf{0}\}$ . For any $\mathbf{x} \neq \mathbf{0}$ we have $\mathbf{B}\mathbf{x} \neq \mathbf{0}$ , and hence

\mathbf{x}^\top \mathbf{A} \mathbf{x} = \mathbf{x}^\top \mathbf{B}^\top \mathbf{B} \mathbf{x} = \|\mathbf{B}\mathbf{x}\|^2 > 0

(96)

So $\mathbf{A}$ is positive definite, which brings us back to condition (1).

\underline{(1) \leftrightarrow (2) \leftrightarrow (3) \leftrightarrow (4) \leftrightarrow (5) \leftrightarrow (6)}

(97)

Positive semidefinite matrices have similar equivalent conditions.

Theorem 7 (◆Equivalent Conditions for Positive Semidefiniteness)

For a symmetric matrix $\mathbf{A} \in \mathbb{R}^{n \times n}$ , the following conditions are equivalent:

$\mathbf{A}$ is positive semidefinite, that is, $\mathbf{x}^\top \mathbf{A} \mathbf{x} \geq 0$ for all $\mathbf{x} \in \mathbb{R}^n$
All eigenvalues of $\mathbf{A}$ are nonnegative
All principal minors of $\mathbf{A}$ are nonnegative (for every index subset $S \subseteq \{0,\ldots,n-1\}$ , the determinant of the submatrix formed by the corresponding rows and columns is $\geq 0$ )
$\mathbf{A}$ can be factored as $\mathbf{A} = \mathbf{L}\mathbf{L}^\top$ , where $\mathbf{L}$ is lower triangular with nonnegative diagonal entries (the generalized Cholesky decomposition)
There exists a (possibly singular) square matrix $\mathbf{C}$ such that $\mathbf{A} = \mathbf{C}^\top \mathbf{C}$
$\mathbf{A} = \mathbf{B}^\top \mathbf{B}$ , where $\mathbf{B}$ is any matrix (its columns need not be linearly independent)

The comparison with the positive definite version is as follows:

Condition	Positive definite	Positive semidefinite
(2) Eigenvalues	$> 0$	$\geq 0$
(3) Principal minors	Leading principal minors $> 0$	All principal minors $\geq 0$
(4) Cholesky	Diagonal entries $> 0$ , decomposition unique	Diagonal entries $\geq 0$ , decomposition not necessarily unique
(5) Square-matrix factorization	$\mathbf{C}$ nonsingular	$\mathbf{C}$ may be singular
(6) General factorization	Columns of $\mathbf{B}$ linearly independent	No restriction on $\mathbf{B}$

Example 9 (Testing for Positive Definiteness)

Consider the symmetric matrix

\mathbf{A} = \begin{pmatrix} 2 & -1 & 0 \\ -1 & 2 & -1 \\ 0 & -1 & 2 \end{pmatrix}

(98)

Test whether it is positive definite:

Compute the eigenvalues:
- The characteristic equation is $\det(\mathbf{A} - \lambda\mathbf{I}) = 0$
- Solving gives the eigenvalues $\lambda_0 = 2 - \sqrt{2}$ , $\lambda_1 = 2$ , $\lambda_2 = 2 + \sqrt{2}$
- Since all the eigenvalues are greater than 0, $\mathbf{A}$ is positive definite
Test the leading principal minors:
- The first leading principal minor: $\det([2]) = 2 > 0$
- The second leading principal minor: $\det\begin{pmatrix} 2 & -1 \\ -1 & 2 \end{pmatrix} = 3 > 0$
- The third leading principal minor: $\det(\mathbf{A}) = 4 > 0$
- All the leading principal minors are positive, so $\mathbf{A}$ is positive definite

Properties and Applications of Positive Definite Matrices¶

Positive definite matrices have many important properties:

A positive definite matrix is invertible, and its inverse is also positive definite
The main diagonal entries of a positive definite matrix are all positive
The determinant of a positive definite matrix is positive
The sum of two positive definite matrices is positive definite
The Schur complement of a positive definite matrix is also positive definite

A positive definite matrix can define an inner product on a vector space:

9.3.3 Positive Definite Hermitian Matrices and Their Applications¶

Definition of Positive Definite Hermitian Matrices¶

For complex vector spaces, the notion of positive definiteness extends naturally:

It should be emphasized that although the entries of $\mathbf{A}$ may be complex, the value of the expression $\mathbf{z}^* \mathbf{A}\mathbf{z}$ is always real (because $\mathbf{A}$ is a Hermitian matrix).

Equivalent Conditions for Positive Definite Hermitian Matrices¶

Example 10 (A Positive Definite Hermitian Matrix)

Consider the Hermitian matrix

\mathbf{A} = \begin{pmatrix} 3 & 2-i \\ 2+i & 4 \end{pmatrix}

(102)

Verify that it is positive definite:

Compute the eigenvalues:
- Characteristic equation: $\det(\mathbf{A} - \lambda\mathbf{I}) = (3-\lambda)(4-\lambda) - |2-i|^2 = \lambda^2 - 7\lambda + 7 = 0$
- Solving gives the eigenvalues $\lambda_0 = \frac{7-\sqrt{21}}{2} \approx 1.21$ , $\lambda_1 = \frac{7+\sqrt{21}}{2} \approx 5.79$
Since all the eigenvalues are positive real numbers, $\mathbf{A}$ is a positive definite Hermitian matrix.

Compute its Cholesky decomposition $\mathbf{A} = \mathbf{L}\mathbf{L}^*$ :

\mathbf{L} = \begin{pmatrix} \sqrt{3} & 0 \\ \frac{2+i}{\sqrt{3}} & \sqrt{\frac{7}{3}} \end{pmatrix} \approx \begin{pmatrix} 1.73 & 0 \\ 1.15+0.58i & 1.53 \end{pmatrix}

(103)

One can verify that $\mathbf{L}\mathbf{L}^* = \mathbf{A}$ .

import numpy as np
from scipy.linalg import cholesky

# Define the Hermitian matrix A
A = np.array([[3, 2-1j],
              [2+1j, 4]], dtype=complex)

print("Original matrix A:")
print(A)
print()

# Verify the Hermitian property
print("Hermitian property check (A = A^*):", np.allclose(A, A.conj().T))
print()

# Compute the eigenvalues
eigenvalues = np.linalg.eigvalsh(A)
print("Eigenvalues:")
print(f"λ₀ ≈ {eigenvalues[0]:.2f}  (theoretical value: (7-√21)/2 ≈ 1.21)")
print(f"λ₁ ≈ {eigenvalues[1]:.2f}  (theoretical value: (7+√21)/2 ≈ 5.79)")
print(f"All eigenvalues are positive: {np.all(eigenvalues > 0)} → the matrix is positive definite")
print()

# Cholesky decomposition
L = cholesky(A, lower=True)
print("Cholesky factor L:")
print(L)
print()

# Verify LL^* = A
print("Verify LL^* = A:")
A_reconstructed = L @ L.conj().T
print("Reconstructed matrix:")
print(A_reconstructed)
print(f"Maximum error: {np.max(np.abs(A - A_reconstructed)):.2e}")

A positive definite Hermitian matrix can define an inner product on a complex vector space:

Applications in Signal Processing¶

In signal processing and communications, positive definite Hermitian matrices are also widely used:

Covariance matrices: the autocorrelation matrix of a signal is a positive semidefinite Hermitian matrix; it is positive definite only when there is no direction with zero second moment
Multiple-input multiple-output (MIMO) systems: the Gram matrix $\mathbf{H}^*\mathbf{H}$ of the channel matrix $\mathbf{H}$ is a positive semidefinite Hermitian matrix; it is positive definite only when $\mathbf{H}$ has full column rank. Calling it a covariance matrix requires a separately specified stochastic model
Beamforming and spatial filtering: optimization based on positive definite Hermitian matrices

Exercise 2 (Cholesky Decomposition and Square Root of a Positive Definite Hermitian Matrix)

Consider the Hermitian matrix $\mathbf{A} = \begin{pmatrix} 5 & 3+2i \\ 3-2i & 5 \end{pmatrix}$ :

Verify that $\mathbf{A}$ is a Hermitian matrix
Test whether $\mathbf{A}$ is positive definite
Compute the Cholesky decomposition $\mathbf{A} = \mathbf{L}\mathbf{L}^*$ of $\mathbf{A}$
Compute the square root $\mathbf{A}^{1/2}$ of $\mathbf{A}$ (that is, the positive definite Hermitian matrix satisfying $(\mathbf{A}^{1/2})^2 = \mathbf{A}$ )

Solution to Exercise 2

Hermitian property. $\mathbf{A}^* = \overline{\mathbf{A}}^\top$ . Take the conjugate of $\mathbf{A}$ entry by entry and then transpose: $(3+2i)$ lies in row 0, column 1; its conjugate is $3-2i$ , which after transposition lands in row 1, column 0, matching the entry of the original $\mathbf{A}$ in that position; the diagonal entries 5 are real. Hence $\mathbf{A}^* = \mathbf{A}$ , and $\mathbf{A}$ is a Hermitian matrix.
Positive definiteness (by Theorem 8, check that all eigenvalues are positive). The characteristic equation $\det(\mathbf{A} - \lambda\mathbf{I}) = 0$ is
$(5-\lambda)^2 - (3+2i)(3-2i) = (5-\lambda)^2 - (9 + 4) = (5-\lambda)^2 - 13 = 0$
(105)
so $\lambda = 5 \pm \sqrt{13}$ , that is, $\lambda_0 = 5 - \sqrt{13} \approx 1.394 > 0$ and $\lambda_1 = 5 + \sqrt{13} \approx 8.606 > 0$ . Both eigenvalues are positive, so $\mathbf{A}$ is positive definite.
Cholesky decomposition $\mathbf{A} = \mathbf{L}\mathbf{L}^*$ . Let $\mathbf{L} = \begin{pmatrix} \ell_{00} & 0 \\ \ell_{10} & \ell_{11} \end{pmatrix}$ ( $\ell_{00}, \ell_{11} > 0$ ). Comparing entries:
- $(0,0)$ : $\ell_{00}^2 = 5 \Rightarrow \ell_{00} = \sqrt{5}$
- $(1,0)$ : $\ell_{10}\,\overline{\ell_{00}} = 3 - 2i \Rightarrow \ell_{10} = \dfrac{3-2i}{\sqrt{5}}$
- $(1,1)$ : $|\ell_{10}|^2 + \ell_{11}^2 = 5 \Rightarrow \ell_{11}^2 = 5 - \dfrac{13}{5} = \dfrac{12}{5} \Rightarrow \ell_{11} = \sqrt{\dfrac{12}{5}} = \dfrac{2\sqrt{15}}{5}$
Therefore $\mathbf{L} = \begin{pmatrix} \sqrt{5} & 0 \\ \dfrac{3-2i}{\sqrt{5}} & \sqrt{\dfrac{12}{5}} \end{pmatrix}$ . (One can verify that $\mathbf{L}\mathbf{L}^* = \mathbf{A}$ .)
The square root is constructed from the spectral decomposition: if $\mathbf{A} = \mathbf{U}\mathbf{\Lambda}\mathbf{U}^*$ , then $\mathbf{A}^{1/2} = \mathbf{U}\mathbf{\Lambda}^{1/2}\mathbf{U}^*$ , where $\mathbf{\Lambda}^{1/2} = \operatorname{diag}(\sqrt{\lambda_0}, \sqrt{\lambda_1})$ . The orthonormal eigenvectors are
$|\mathbf{u}_0\rangle = \frac{1}{\sqrt{2}}\begin{pmatrix} 1 \\ -\frac{3-2i}{\sqrt{13}} \end{pmatrix}, \quad |\mathbf{u}_1\rangle = \frac{1}{\sqrt{2}}\begin{pmatrix} 1 \\ \frac{3-2i}{\sqrt{13}} \end{pmatrix}$
(106)
Substituting into the outer-product form $\mathbf{A}^{1/2} = \sqrt{\lambda_0}\,|\mathbf{u}_0\rangle\langle\mathbf{u}_0| + \sqrt{\lambda_1}\,|\mathbf{u}_1\rangle\langle\mathbf{u}_1|$ gives
$\mathbf{A}^{1/2} = \begin{pmatrix} \dfrac{\sqrt{\lambda_0}+\sqrt{\lambda_1}}{2} & \dfrac{3+2i}{2\sqrt{13}}\left(\sqrt{\lambda_1}-\sqrt{\lambda_0}\right) \\[2mm] \dfrac{3-2i}{2\sqrt{13}}\left(\sqrt{\lambda_1}-\sqrt{\lambda_0}\right) & \dfrac{\sqrt{\lambda_0}+\sqrt{\lambda_1}}{2} \end{pmatrix} \approx \begin{pmatrix} 2.057 & 0.729+0.486i \\ 0.729-0.486i & 2.057 \end{pmatrix}$
(107)
where $\sqrt{\lambda_0} \approx 1.181$ and $\sqrt{\lambda_1} \approx 2.933$ . This matrix is positive definite Hermitian, and squaring it recovers $\mathbf{A}$ . $\square$

§9.3 Summary

This section answered a central question left open by Chapter 8: what condition guarantees that a matrix has all real eigenvalues and that an orthonormal basis can be chosen from its eigenvectors? The answer is precisely symmetry—a real symmetric matrix satisfies $\mathbf{A}=\mathbf{A}^\top$ , a complex Hermitian matrix satisfies $\mathbf{A}=\mathbf{A}^*$ , and both are self-adjoint operators that “respect the inner product structure.”

The spectral decomposition theorem is the summit of this section: every symmetric/Hermitian matrix can be written as $\mathbf{A}=\mathbf{P}\mathbf{\Lambda}\mathbf{P}^\top$ (or $\mathbf{U}\mathbf{\Lambda}\mathbf{U}^*$ ), where $\mathbf{P}$ (or $\mathbf{U}$ ) is an orthogonal (unitary) matrix and $\mathbf{\Lambda}$ is a real diagonal matrix. This decomposition embodies both the theory of orthogonal matrices from §9.2 and the eigenvalue theory of §8, and it is the most complete matrix structure theorem in the book so far.

Positive definiteness further refines the hierarchy of symmetric matrices: positive definite ↔ all eigenvalues positive ↔ all leading principal minors positive ↔ a Cholesky decomposition exists. A positive definite matrix can be regarded as the generator of a generalized inner product on a vector space, and such matrices appear widely as Hessian matrices in optimization, covariance matrices in statistics, and stiffness matrices in engineering.

Once we have mastered the spectral decomposition of symmetric matrices, we can understand the geometric essence of quadratic forms—the next section establishes this connection systematically.

9.4 Quadratic Forms and the Principal Axis Theorem¶

A quadratic form is a linear combination of squares and products of variables, widely used in optimization, statistics, quantum mechanics, computer graphics, and other fields. The principal axis theorem shows that a quadratic form can be simplified to a standard form by a suitable change of coordinates, a process that geometrically corresponds to finding the principal directions of a quadric surface. This section discusses the theory of quadratic forms and its applications in real and complex spaces in full, revealing the essential geometric meaning of symmetric and Hermitian matrices.

9.4.1 Definition and Basic Properties of Quadratic Forms¶

Definition of a Quadratic Form¶

Geometrically, quadratic forms correspond to conics and quadric surfaces, such as ellipses, hyperbolas, and ellipsoids. In physics, quadratic forms can represent key quantities such as energy functions and moments of inertia.

Quadratic Forms and Symmetric Matrices¶

Every quadratic form can be represented by a symmetric matrix:

Classifying Quadratic Forms by Definiteness¶

Quadratic forms can be classified according to the signs of the values they take:

The definiteness of a quadratic form can be determined from the eigenvalues of the corresponding symmetric matrix:

import numpy as np

np.set_printoptions(precision=4, suppress=True, linewidth=100)

def print_header(title):
    print("=" * 60)
    print(f"  {title}")
    print("=" * 60)

def print_step(step, desc):
    print(f"\n▶ Step {step}: {desc}")
    print("-" * 40)

# ============================================================
print_header("Example 9.4.1 | Determining the Definiteness of a Quadratic Form")
# ============================================================

# --- Step 1 ---
print_step(1, "Read off the symmetric matrix A from the coefficients of the quadratic form")
# Q(x,y,z) = 2x² + 3y² + 4z² - 2xy + 4xz - 6yz
# Off-diagonal terms are halved to symmetrize: a_ij = a_ji = coefficient/2
A = np.array([[ 2, -1,  2],
              [-1,  3, -3],
              [ 2, -3,  4]], dtype=float)
print(A)
print(f"  Symmetry check A = A^T: {np.allclose(A, A.T)}")

# --- Step 2 ---
print_step(2, "Check the value of the quadratic form at a specific vector")
x = np.array([1, 1, 1], dtype=float)
Q_val = x @ A @ x
print(f"  Q(1,1,1) = x^T Ax = {Q_val}")
print(f"  Hand check: 2+3+4−2+4−6 = {2+3+4-2+4-6}  ✓")

# --- Step 3 ---
print_step(3, "Compute the eigenvalues (eigh guarantees real output, in ascending order)")
eigenvalues = np.linalg.eigvalsh(A)
print(f"  λ₀ ≈ {eigenvalues[0]:.4f}")
print(f"  λ₁ ≈ {eigenvalues[1]:.4f}")
print(f"  λ₂ ≈ {eigenvalues[2]:.4f}")

# --- Step 4 ---
print_step(4, "Determine the definiteness")
is_pd = bool(np.all(eigenvalues > 0))
print(f"  All eigenvalues > 0: {is_pd}")
print(f"  → the quadratic form is positive definite  ✓")

9.4.2 Diagonalization of Quadratic Forms in Real Space and the Principal Axis Theorem¶

Algebraic Derivation of the Diagonalization¶

By a suitable change of coordinates, every quadratic form can be simplified to a form containing only square terms:

Theorem 10 (Diagonalization of Quadratic Forms (the Principal Axis Theorem))

For every quadratic form $Q(\mathbf{x}) = \mathbf{x}^\top \mathbf{A}\mathbf{x}$ with $\mathbf{A}$ a symmetric matrix, there exists an orthogonal change of variables $\mathbf{x} = \mathbf{P}\mathbf{y}$ such that in the new coordinates $\mathbf{y}$ the quadratic form becomes

Q(\mathbf{x}) = \lambda_0 y_0^2 + \lambda_1 y_1^2 + \cdots + \lambda_{n-1} y_{n-1}^2

(115)

where $\lambda_0, \lambda_1, \ldots, \lambda_{n-1}$ are the eigenvalues of the matrix $\mathbf{A}$ and the columns of $\mathbf{P}$ are the corresponding orthonormal eigenvectors. The coordinate axes of the new coordinate system are called the principal axes of the quadric surface.

This theorem is in fact an application of the spectral theorem for real symmetric matrices. It shows that every real symmetric matrix can be diagonalized by an orthogonal matrix:

\mathbf{A} = \mathbf{P}\mathbf{\Lambda}\mathbf{P}^\top

(116)

where $\mathbf{\Lambda} = \text{diag}(\lambda_0, \lambda_1, \ldots, \lambda_{n-1})$ is the diagonal matrix of eigenvalues.

Proof 13

Let $\mathbf{x} = \mathbf{P}\mathbf{y}$ , where $\mathbf{P}$ is an orthogonal matrix; then the quadratic form becomes

\begin{align} Q(\mathbf{x}) &= \mathbf{x}^\top \mathbf{A}\mathbf{x} \\ &= (\mathbf{P}\mathbf{y})^\top \mathbf{A}(\mathbf{P}\mathbf{y}) \\ &= \mathbf{y}^\top \mathbf{P}^\top \mathbf{A}\mathbf{P}\mathbf{y} \end{align}

(117)

By the spectral theorem we can choose $\mathbf{P}$ so that $\mathbf{P}^\top \mathbf{A}\mathbf{P} = \mathbf{\Lambda}$ , that is, so that the columns of $\mathbf{P}$ are orthonormal eigenvectors of $\mathbf{A}$ . Then

\underline{Q(\mathbf{x}) = \mathbf{y}^\top \mathbf{\Lambda}\mathbf{y} = \sum_{i=0}^{n-1} \lambda_i y_i^2}

(118)

This theorem has an important geometric meaning:

The eigenvalues determine how much the quadric surface “stretches” along each principal axis
The eigenvectors determine the principal directions (principal axes) of the quadric surface
The orthogonal transformation can be regarded as a rotation of the coordinate system that aligns the new coordinate system with the principal axes of the quadric surface

Example 12 (Principal Axes of an Ellipse)

Consider the quadratic form

Q(x, y) = 3x^2 + 2\sqrt{3}xy + 3y^2

(119)

Its matrix form is

\mathbf{A} = \begin{pmatrix} 3 & \sqrt{3} \\ \sqrt{3} & 3 \end{pmatrix}

(120)

Computing the eigenvalues and eigenvectors:

Eigenvalues: $\lambda_0 = 3 + \sqrt{3}$ , $\lambda_1 = 3 - \sqrt{3}$
Orthonormal eigenvectors: $\mathbf{p}_0 = \frac{1}{\sqrt{2}}(1, 1)^\top$ , $\mathbf{p}_1 = \frac{1}{\sqrt{2}}(1, -1)^\top$

Construct the orthogonal matrix:

\mathbf{P} = \frac{1}{\sqrt{2}}\begin{pmatrix} 1 & 1 \\ 1 & -1 \end{pmatrix}

(121)

The canonical form in the new coordinate system is

Q(u, v) = (3 + \sqrt{3})u^2 + (3 - \sqrt{3})v^2

(122)

This represents an ellipse whose principal axes lie along the directions of the eigenvectors $\mathbf{p}_0$ and $\mathbf{p}_1$ ; the length of the semi-major axis is proportional to $\frac{1}{\sqrt{3 - \sqrt{3}}}$ and the length of the semi-minor axis is proportional to $\frac{1}{\sqrt{3 + \sqrt{3}}}$ .

The Law of Inertia for Quadratic Forms and the Classification of Quadric Surfaces¶

Although a quadratic form can be given different representations by different changes of coordinates, some properties are invariant:

The indices of inertia are directly related to the eigenvalues of the symmetric matrix: the positive index of inertia equals the number of positive eigenvalues, the negative index of inertia equals the number of negative eigenvalues, and the zero index of inertia equals the number of zero eigenvalues.

Below we classify only the positive level surfaces $Q(\mathbf{x})=1$ of nondegenerate quadratic forms in three-dimensional real space, with no linear terms.

9.4.3 Hermitian Forms in Complex Space¶

From Quadratic Forms to Hermitian Forms¶

In a complex vector space, the notion of a quadratic form must be extended to that of a Hermitian form in order to keep its positive definiteness and geometric meaning:

The values of a Hermitian form are always real, which is guaranteed by the properties of Hermitian matrices. In fact, for every complex vector $\mathbf{z}$ , the Hermitian form $H(\mathbf{z}) = \mathbf{z}^* \mathbf{A}\mathbf{z}$ satisfies

H(\mathbf{z}) = \overline{H(\mathbf{z})}

(124)

which guarantees that $H(\mathbf{z})$ is real.

Comparing Hermitian Matrices and Symmetric Matrices¶

Hermitian matrices (defined in Definition 10) are the natural extension of symmetric matrices to complex vector spaces:

Hermitian matrices share many properties with real symmetric matrices:

All eigenvalues are real
Eigenvectors belonging to different eigenvalues are mutually orthogonal
They can be diagonalized by a unitary matrix: $\mathbf{A} = \mathbf{U}\mathbf{\Lambda}\mathbf{U}^*$ , where $\mathbf{U}$ is a unitary matrix and $\mathbf{\Lambda}$ is a real diagonal matrix

Positive Definiteness in Complex Vector Spaces¶

The definiteness of a Hermitian form is analogous to that of a real quadratic form:

As in real space, the definiteness of a Hermitian form can be determined from the eigenvalues of the Hermitian matrix:

9.4.4 The Principal Axis Theorem in Complex Space¶

Statement of the Principal Axis Theorem for Complex Vector Spaces¶

The principal axis theorem extends naturally to complex vector spaces:

Note that in complex space the canonical form involves the squared moduli $|w_i|^2 = \overline{w_i} w_i$ rather than simple squares. This reflects the essential difference between Hermitian forms and quadratic forms.

Comparing Unitary and Orthogonal Transformations¶

Unitary transformations are the counterparts of orthogonal transformations in complex space:

Orthogonal transformation: $\mathbf{x} = \mathbf{P}\mathbf{y}$ , where $\mathbf{P}^\top \mathbf{P} = \mathbf{I}$ ; it preserves lengths and inner products
Unitary transformation: $\mathbf{z} = \mathbf{U}\mathbf{w}$ , where $\mathbf{U}^* \mathbf{U} = \mathbf{I}$ ; it preserves lengths and inner products

Geometrically, orthogonal transformations represent rotations and reflections, while unitary transformations can be regarded as “rotations” in complex space. Neither changes the lengths of vectors or the “angles” between them.

Applying the Spectral Decomposition to the Principal Axis Theorem¶

The proof of the principal axis theorem for complex vector spaces relies on the spectral decomposition of Hermitian matrices:

Proof 14

By the spectral decomposition theorem for Hermitian matrices, a Hermitian matrix $\mathbf{A}$ can be decomposed as

\mathbf{A} = \mathbf{U}\mathbf{\Lambda}\mathbf{U}^*

(128)

where $\mathbf{U}$ is a unitary matrix and $\mathbf{\Lambda} = \text{diag}(\lambda_0, \lambda_1, \ldots, \lambda_{n-1})$ is a real diagonal matrix.

Let $\mathbf{z} = \mathbf{U}\mathbf{w}$ ; then the Hermitian form becomes

\begin{align} H(\mathbf{z}) &= \mathbf{z}^* \mathbf{A}\mathbf{z} \\ &= (\mathbf{U}\mathbf{w})^* \mathbf{A}(\mathbf{U}\mathbf{w}) \\ &= \mathbf{w}^* \mathbf{U}^* \mathbf{A}\mathbf{U}\mathbf{w} \\ &= \mathbf{w}^* \mathbf{\Lambda}\mathbf{w} \\ &= \sum_{i=0}^{n-1} \lambda_i |w_i|^2 \end{align}

(129)

\underline{H(\mathbf{z}) = \sum_{i=0}^{n-1} \lambda_i |w_i|^2}

(130)

This gives the canonical form of the Hermitian form.

The Geometric Meaning of the Canonical Form in Complex Space¶

The positive level set $H(\mathbf{z})=1$ of the canonical form $H(\mathbf{z}) = \sum_{i=0}^{n-1} \lambda_i |w_i|^2$ of a Hermitian form corresponds geometrically to a hypersurface in complex space. Although it is hard to visualize directly, it can be understood as follows:

A positive definite Hermitian form (all $\lambda_i > 0$ ) corresponds to a “hyperellipsoid” in complex space
An indefinite Hermitian form (with both positive and negative eigenvalues) corresponds to a “hyperboloid” in complex space
A singular positive semidefinite Hermitian form (all $\lambda_i \geq 0$ , with at least one equal to 0) corresponds to a degenerate hypersurface

Example 14 (A Hermitian Form on the Complex Plane)

Consider the Hermitian form on $\mathbb{C}^2$ defined by the Hermitian matrix $\mathbf{A} = \begin{pmatrix} 3 & i \\ -i & 2 \end{pmatrix}$ :

H(\mathbf{z}) = \mathbf{z}^* \mathbf{A}\mathbf{z} = 3|z_0|^2 + 2|z_1|^2 + iz_0^*z_1 - iz_1^*z_0

(131)

Computing the eigenvalues and eigenvectors:

Eigenvalues: $\lambda_0 = \frac{5 + \sqrt{5}}{2} \approx 3.618$ , $\lambda_1 = \frac{5 - \sqrt{5}}{2} \approx 1.382$
Normalized eigenvectors (obtained by computation): $\mathbf{u}_0 \approx \begin{pmatrix}0.850651\\-0.525731i\end{pmatrix}$ , $\mathbf{u}_1 \approx \begin{pmatrix}0.525731\\0.850651i\end{pmatrix}$ ; they correspond in order to the eigenvalues above and are assembled as column vectors into $\mathbf{U}=[\mathbf{u}_0\;\mathbf{u}_1]$

By the unitary change of variables $\mathbf{z} = \mathbf{U}\mathbf{w}$ , where $\mathbf{U} = [\mathbf{u}_0, \mathbf{u}_1]$ , the Hermitian form is brought to canonical form:

H(\mathbf{z}) \approx 3.618|w_0|^2 + 1.382|w_1|^2

(132)

This represents a “hyperellipsoid” in complex space whose “principal axes” lie along the directions of $\mathbf{u}_0$ and $\mathbf{u}_1$ .

Real vectors and imaginary vectors are handled differently. For example, consider a purely imaginary vector $\mathbf{z} = i\mathbf{x}$ , where $\mathbf{x}$ is a real vector, and substitute it into the Hermitian form:

H(i\mathbf{x}) = (i\mathbf{x})^* \mathbf{A}(i\mathbf{x}) = \mathbf{x}^\top (-i)(\mathbf{A})(i\mathbf{x}) = \mathbf{x}^\top \mathbf{A}\mathbf{x} = H(\mathbf{x})

(133)

This shows that a Hermitian form behaves in the same way on purely real vectors and purely imaginary vectors, which is a manifestation of unitary invariance.

Exercise 3 (The Complex Principal Axis Theorem and the Canonical Form of a Hermitian Form)

Consider the Hermitian matrix $\mathbf{A} = \begin{pmatrix} 2 & 1-i \\ 1+i & 3 \end{pmatrix}$ on $\mathbb{C}^2$ :

Verify that $\mathbf{A}$ is indeed a Hermitian matrix
Compute the eigenvalues and eigenvectors of $\mathbf{A}$
Write down the canonical form of the corresponding Hermitian form $H(\mathbf{z}) = \mathbf{z}^*\mathbf{A}\mathbf{z}$
Explain the geometric meaning of the canonical form

Solution to Exercise 3

Hermitian property. The off-diagonal entry $(0,1)$ is $1-i$ , and its conjugate $1+i$ is exactly the $(1,0)$ entry; the diagonal entries $2, 3$ are real. Hence $\mathbf{A}^* = \mathbf{A}$ .
Eigenvalues. $\det(\mathbf{A}-\lambda\mathbf{I}) = (2-\lambda)(3-\lambda) - (1-i)(1+i) = (2-\lambda)(3-\lambda) - 2 = \lambda^2 - 5\lambda + 4 = 0$ , which gives $\lambda_0 = 1$ and $\lambda_1 = 4$ (both real, as the properties of Hermitian matrices require).
Eigenvectors.
- $\lambda_0 = 1$ : solve $(\mathbf{A}-\mathbf{I})\mathbf{v} = \mathbf{0}$ , that is, $\begin{pmatrix} 1 & 1-i \\ 1+i & 2 \end{pmatrix}\mathbf{v} = \mathbf{0}$ , which gives $\mathbf{v}_0 \propto \begin{pmatrix} 1-i \\ -1 \end{pmatrix}$ ; normalizing, $|\mathbf{u}_0\rangle = \frac{1}{\sqrt{3}}\begin{pmatrix} 1-i \\ -1 \end{pmatrix}$ .
- $\lambda_1 = 4$ : solve $(\mathbf{A}-4\mathbf{I})\mathbf{v} = \mathbf{0}$ , which gives $\mathbf{v}_1 \propto \begin{pmatrix} 1-i \\ 2 \end{pmatrix}$ ; normalizing, $|\mathbf{u}_1\rangle = \frac{1}{\sqrt{6}}\begin{pmatrix} 1-i \\ 2 \end{pmatrix}$ .
- Check orthogonality: $\langle\mathbf{u}_0|\mathbf{u}_1\rangle = \frac{1}{\sqrt{18}}\big[\overline{(1-i)}(1-i) + \overline{(-1)}\cdot 2\big] = \frac{1}{\sqrt{18}}[|1-i|^2 - 2] = \frac{1}{\sqrt{18}}[2-2] = 0$ ✓
Canonical form. Let $\mathbf{U} = \big(|\mathbf{u}_0\rangle\ \ |\mathbf{u}_1\rangle\big)$ be the unitary matrix and make the change of variables $\mathbf{z} = \mathbf{U}\mathbf{w}$ (that is, $\mathbf{w} = \mathbf{U}^*\mathbf{z}$ ). By the spectral decomposition $\mathbf{A} = \mathbf{U}\mathbf{\Lambda}\mathbf{U}^*$ (by Theorem 13):
$H(\mathbf{z}) = \mathbf{z}^*\mathbf{A}\mathbf{z} = \mathbf{w}^*\mathbf{\Lambda}\mathbf{w} = \lambda_0|w_0|^2 + \lambda_1|w_1|^2 = |w_0|^2 + 4|w_1|^2$
(134)
This is the canonical form of the Hermitian form—a real-valued quadratic expression with real coefficients and no cross terms.
Geometric meaning. In the unitary coordinates $\mathbf{w}$ , $H$ becomes independent scaling along two mutually orthogonal principal axes (the directions of $|\mathbf{u}_0\rangle$ and $|\mathbf{u}_1\rangle$ ), with the eigenvalues 1 and 4 as scaling factors. Since both eigenvalues are positive, $H(\mathbf{z}) > 0$ for all $\mathbf{z} \neq \mathbf{0}$ , so $\mathbf{A}$ is positive definite; in complex coordinates $H(\mathbf{z}) = c$ ( $c>0$ ) describes an “ellipsoid”-type level set whose shape is determined by the axis ratio $1:4$ . This is completely parallel in structure to the principal axis theorem for real quadratic forms (Theorem 10), reflecting the consistency of the real and complex theories. $\square$

Quadratic forms and the principal axis theorem are a model of how linear algebra binds algebra and geometry tightly together. The geometric interpretation of the eigenvalues and eigenvectors of symmetric matrices makes abstract algebraic structures visible and intuitive. The extension from real vector spaces to complex vector spaces shows the consistency and universality of mathematical concepts, and provides a solid mathematical foundation for modern physical theories such as quantum mechanics.

# ==================== Visualizing the quadratic form of a Hermitian matrix ====================
import numpy as np
import matplotlib.pyplot as plt


# Create a 2×2 Hermitian matrix
A_hermitian = np.array([
[2,1+1j],[1-1j,3]
], dtype=complex)

print("\n" + "=" * 50)
print("Hermitian matrix example (complex case)")
print("=" * 50)

# Verify the Hermitian property: A = A^†
is_hermitian = np.allclose(A_hermitian, A_hermitian.conj().T)
print(f"Is the matrix Hermitian: {is_hermitian}")
print("\nOriginal matrix A:")
print(A_hermitian)

# Compute the eigenvalues and eigenvectors
# For symmetric matrices, `eigh` is more efficient and more accurate than `eig`, and returns the eigenvalues in ascending order.
eigenvalues_herm, eigenvectors_herm = np.linalg.eigh(A_hermitian)
print("\nEigenvalues (note: the eigenvalues of a Hermitian matrix must be real):")
print(eigenvalues_herm)
print(f"Imaginary parts of the eigenvalues (should be close to zero): {np.max(np.abs(eigenvalues_herm.imag)):.2e}")

print("\nEigenvector matrix (complex vectors):")
print(eigenvectors_herm)

# Verify that the eigenvectors are unitary (using the conjugate transpose for complex vectors)
unitarity_check = eigenvectors_herm.conj().T @ eigenvectors_herm
print("\nInner-product matrix of the eigenvectors Q^† Q (should be the identity matrix):")
print(np.round(unitarity_check, 10))

# Verify the eigendecomposition: A = Q Λ Q^†
A_herm_reconstructed = eigenvectors_herm @ np.diag(eigenvalues_herm) @ eigenvectors_herm.conj().T
reconstruction_error_herm = np.linalg.norm(A_hermitian - A_herm_reconstructed)
print(f"\nReconstruction error ||A - QΛQ^†||: {reconstruction_error_herm:.2e}")

# Check positive definiteness
is_positive_definite_herm = np.all(eigenvalues_herm > 0)
print(f"\nIs the matrix positive definite: {is_positive_definite_herm}")

if is_positive_definite_herm:
    # Compute the square root of the Hermitian matrix
    A_herm_sqrt = eigenvectors_herm @ np.diag(np.sqrt(eigenvalues_herm)) @ eigenvectors_herm.conj().T
    print("\nSquare root of the Hermitian matrix A^(1/2):")
    print(np.round(A_herm_sqrt, 4))
    
    # Verify
    verification_herm = A_herm_sqrt @ A_herm_sqrt
    print("\nCheck (A^(1/2))² ≈ A:")
    print(np.round(verification_herm, 10))
    print(f"Check error: {np.linalg.norm(verification_herm - A_hermitian):.2e}")

# Range of the eigenvalues of the Hermitian matrix
print(f"Eigenvalue range of the Hermitian matrix: [{eigenvalues_herm[0]:.4f}, {eigenvalues_herm[-1]:.4f}]")
print(f"Condition number of the Hermitian matrix: {eigenvalues_herm[-1]/eigenvalues_herm[0]:.4f}")

# Compute the eigenvalues and eigenvectors
eigenvalues_2d_herm, eigenvectors_2d_herm = np.linalg.eigh(A_hermitian )

print("\n" + "=" * 50)
print("Visualizing a 2D Hermitian matrix")
print("=" * 50)
print("Eigenvalues (real):", eigenvalues_2d_herm)
print("Eigenvector matrix (complex):")
print(eigenvectors_2d_herm)




# 3. Create a real coordinate grid (x, y are the two components of the vector)
x = np.linspace(-2, 2, 200)
y = np.linspace(-2, 2, 200)
X, Y = np.meshgrid(x, y)

# 4. Compute the quadratic form Q(v) = v^H A v, where v = [x, y]^T is a real vector
# Note: when v is a real vector, v^H A v = v^T Re(A) v
# because the imaginary parts at symmetric positions cancel (1j * xy - 1j * yx = 0)
Z = A_hermitian[0,0].real * X**2 + 2 * A_hermitian[0,1].real * X * Y + A_hermitian[1,1].real * Y**2

# 5. Plot
plt.figure(figsize=(8, 8))
# Draw the contour lines
contour = plt.contour(X, Y, Z, levels=15, cmap='plasma')
plt.clabel(contour, inline=True, fontsize=8)

# Draw the unit-energy ellipse (Q(v) = 1)
ellipse = plt.contour(X, Y, Z, levels=[1.0], colors='red', linewidths=3)

# On the real section z∈R², the principal axes are determined by Re(A)
section_values, section_vectors = np.linalg.eigh(A_hermitian.real)
for i in range(len(section_values)):
    v = section_vectors[:, i]
    # Scale the eigenvectors to fit the plot
    scale = 1 / np.sqrt(section_values[i])
    plt.quiver(0, 0, v[0]*scale, v[1]*scale, angles='xy', scale_units='xy', 
               scale=1, color=['blue', 'green'][i], label=f'EV {i+1} axis')

plt.axhline(0, color='black', lw=1)
plt.axvline(0, color='black', lw=1)
plt.gca().set_aspect('equal')
plt.title('Real Section: z in R^2\n$Q(x,y) = x^T Re(A) x = 1$')
plt.xlabel('$x_1$')
plt.ylabel('$x_2$')
plt.legend()
plt.grid(True, alpha=0.3)
plt.show()

print("\nKey observations:")
print("• A Hermitian matrix guarantees that z^† A z is always real")
print("• The eigenvalues are all real, even though the eigenvectors are complex vectors")
print("• The contour lines show the section restricted to R²; the full C² has four real dimensions")
print(f"• Eigenvalue range of the matrix A: [{eigenvalues_2d_herm[0]:.4f}, {eigenvalues_2d_herm[-1]:.4f}]")

§9.4 Summary

This section answered a classical geometric question: why can the level surfaces of a quadratic form be brought to principal-axis form by an orthogonal change of basis? The answer is given by the principal axis theorem: every quadratic form $Q(\mathbf{x})=\mathbf{x}^\top\mathbf{A}\mathbf{x}$ ( $\mathbf{A}$ a real symmetric matrix) can be brought by an orthogonal change of variables $\mathbf{x}=\mathbf{P}\mathbf{y}$ to $\sum_i\lambda_i y_i^2$ , where the $\lambda_i$ are the eigenvalues of $\mathbf{A}$ and the columns of $\mathbf{P}$ are the corresponding orthonormal eigenvectors—the “principal axis directions.”

The geometric core of this section is that the eigenvalues determine the shape of the quadric surface (ellipsoid or hyperboloid) and the eigenvectors determine the directions of the principal axes. For a nondegenerate $Q(\mathbf{x})=1$ in three dimensions, three positive eigenvalues give an ellipsoid, two positive and one negative a hyperboloid of one sheet, one positive and two negative a hyperboloid of two sheets, and three negative the empty set; one cannot look only at the sign of the determinant or ignore the level value. This classification theory unifies results of analytic geometry that had long been scattered.

Over the complex field, the Hermitian form $H(\mathbf{z})=\mathbf{z}^*\mathbf{A}\mathbf{z}$ becomes $\sum_i\lambda_i|w_i|^2$ after a unitary change of variables; what appears is the squared modulus $|w_i|^2$ , not the square of a real part. This detail reflects the essential difference between Hermitian forms and real quadratic forms, and corresponds directly to the probabilistic interpretation of expanding an observable in its eigenstate basis in quantum mechanics.

As the theoretical end point of Chapter 9, this section fuses the inner product, the spectral decomposition, and geometric classification into one, providing a complete mathematical language for the operator theory of quantum mechanics in Chapter 10.

9.5 Summary¶

Review of the Theoretical Thread¶

This chapter started from a question: what are the costs and the rewards of extending the geometric language of real vector spaces to the complex field? §9.1 laid the foundation of the answer. With conjugation introduced, the complex inner product $⟨\mathbf{u}|\mathbf{v}⟩=\mathbf{u}^*\mathbf{v}$ regains positive definiteness, thereby giving $\mathbb{C}^n$ fully meaningful notions of length and angle. Within this framework, Gram-Schmidt orthogonalization provides a systematic algorithm for constructing orthonormal bases, which is precisely the geometric interpretation of the QR decomposition.

§9.2 then asked: which linear transformations preserve this geometric structure exactly as it is? Orthogonal matrices and unitary matrices are the answer—their 2-norm condition number is always 1, making them the most stable class of matrices in numerical computation. Geometrically they correspond to rotations and reflections, which change only the viewing angle, not the shape of the object. An orthogonal matrix itself, however, need not be orthogonally diagonalizable, and this detail leads to another important class defined by a different condition: real symmetric matrices.

§9.3 is the heart of the chapter. Symmetric matrices ( $\mathbf{A}=\mathbf{A}^\top$ ) and Hermitian matrices ( $\mathbf{A}=\mathbf{A}^*$ ), as self-adjoint operators, are naturally compatible with the inner product structure, so their eigenvalues must be real and eigenvectors belonging to different eigenvalues must be orthogonal. The spectral decomposition theorem $\mathbf{A}=\mathbf{P}\mathbf{\Lambda}\mathbf{P}^\top$ (or $\mathbf{U}\mathbf{\Lambda}\mathbf{U}^*$ ) fuses the theory of orthogonal matrices from §9.2 with the eigenvalue theory of §8, and is the most complete matrix structure theorem so far. Positive definiteness adds a further scale to this structure: Sylvester’s criterion, the Cholesky decomposition, and the interpretation of covariance matrices form a bridge between theory and application.

§9.4 closes with quadratic forms. The principal axis theorem says that every quadratic form can be brought to a diagonal canonical form by an orthogonal change of variables, with the eigenvalues determining the shape of the surface and the eigenvectors the directions of the principal axes. Geometrically, this result unifies the classification theory of classical curves and surfaces such as ellipses, hyperbolas, and ellipsoids; over the complex field, the canonical form $\sum\lambda_i|w_i|^2$ of a Hermitian form echoes directly the eigenstate expansion of observables in quantum mechanics.

Connections to Other Chapters¶

This chapter deepens and completes the eigenvalue theory of Chapter 8. §8 built the eigenvalue framework for general matrices, but could give only conditional answers to questions such as “when is a matrix diagonalizable” and “when are the eigenvalues real.” This chapter takes symmetry (self-adjointness) as a sufficient condition and obtains a perfect special case of the general theory of Chapter 8: under this condition, diagonalization is not only possible but can be achieved by an orthogonal (unitary) matrix, the eigenvalues are automatically all real, eigenvectors belonging to different eigenvalues are automatically orthogonal, and within the eigenspace of a repeated root the eigenvectors can be orthogonalized. The spectral decomposition theorem of §9.3 can be regarded as the strongest form of Chapter 8’s “diagonalization by similarity” for the class of symmetric matrices.

This chapter also builds a double bridge to the introduction to quantum mechanics in Chapter 10 and the singular value decomposition in Chapter 11. In quantum mechanics, observables correspond to Hermitian operators, the evolution of quantum states corresponds to unitary transformations, and measurement outcomes (which must be real) are precisely the physical realization of the spectral theorem of this chapter. The singular value decomposition (SVD), for its part, can be seen as asking, for a general matrix (non-square, non-symmetric), “what is the geometric decomposition closest to that of a symmetric matrix?”; it generalizes both the orthogonal matrices and the spectral decomposition of this chapter and forms the final unification of the theory of linear algebra.

The Role of This Chapter in the Book¶

This chapter resolves a central question: once a vector space has been given a geometric structure (an inner product), which class of matrices is the most natural and the most regular? The answer given by symmetric and Hermitian matrices is: all real eigenvalues, orthogonal eigenvectors, and a perfect spectral decomposition. This is not an isolated mathematical coincidence but the full expression, at the level of linear algebra, of a deep symmetry—self-adjointness. In nature, a self-adjoint Hamiltonian corresponds to real energies; in a closed system whose Hamiltonian does not depend explicitly on time, the expectation value of the energy is conserved; physical observables correspond to Hermitian operators, and the covariance structure of data corresponds to positive semidefinite matrices—symmetry is everywhere, and the theory of this chapter therefore has a universal significance that goes beyond linear algebra itself.

As you read on with the perspective of this chapter, you will find that the singular value decomposition of Chapter 11 is in fact the “non-square version” of the spectral decomposition of this chapter: the left and right singular vectors are the orthogonal eigenvectors of $\mathbf{A}\mathbf{A}^*$ and $\mathbf{A}^*\mathbf{A}$ , respectively. Once you understand this chapter, you already hold the most essential geometric intuition of the SVD—decomposing any linear transformation into the three steps “rotate—stretch—rotate,” each of which is rooted in the inner-product geometry of this chapter.

Chapter 9 Inner Product Spaces and Orthogonal Structure

Chapter Structure and Learning Objectives¶

9.1 Inner Product Spaces¶

Why Do We Need Inner Product Spaces?¶

9.1.1 Definition and Properties of the Inner Product¶

9.1.2 Orthonormal Bases¶

9.1.3 The Gram-Schmidt Orthogonalization Method¶

9.2 Orthogonal Matrices and Unitary Matrices¶

9.2.1 Definitions and Basic Properties of Orthogonal and Unitary Matrices¶

Definitions¶

Basic Properties¶

Orthogonality of Columns and Rows¶

Examples of Orthogonal Matrices¶

9.2.2 The Geometric Meaning of Orthogonal and Unitary Transformations¶

Definition of Orthogonal Transformations and Their Equivalence with Orthogonal Matrices¶

Geometric Decomposition of Orthogonal Transformations¶

◇9.2.3 Special Properties of Unitary Matrices and Quantum Computing¶

The Importance of Unitary Matrices in Quantum Mechanics¶

Common Unitary Matrices: Quantum Gates¶

The Quantum Fourier Transform¶

9.2.4 Numerical Stability and Applications¶

Why Orthogonal Matrices Are Numerically Stable¶

Main Applications¶

9.3 Symmetric Matrices and Hermitian Matrices¶

9.3.1 Eigenvalues and Eigenvectors of Symmetric and Hermitian Matrices¶

Definitions of Symmetric and Hermitian Matrices¶

Properties of the Eigenvalues and Eigenvectors¶

The Spectral Decomposition Theorem¶

9.3.2 Positive Definite and Positive Semidefinite Matrices¶

Definition of Positive Definiteness¶

Equivalent Conditions for Positive Definiteness¶

Properties and Applications of Positive Definite Matrices¶

9.3.3 Positive Definite Hermitian Matrices and Their Applications¶

Definition of Positive Definite Hermitian Matrices¶

Equivalent Conditions for Positive Definite Hermitian Matrices¶

Applications in Signal Processing¶

9.4 Quadratic Forms and the Principal Axis Theorem¶

9.4.1 Definition and Basic Properties of Quadratic Forms¶

Definition of a Quadratic Form¶

Quadratic Forms and Symmetric Matrices¶

Classifying Quadratic Forms by Definiteness¶

9.4.2 Diagonalization of Quadratic Forms in Real Space and the Principal Axis Theorem¶

Algebraic Derivation of the Diagonalization¶

The Law of Inertia for Quadratic Forms and the Classification of Quadric Surfaces¶

9.4.3 Hermitian Forms in Complex Space¶

From Quadratic Forms to Hermitian Forms¶

Comparing Hermitian Matrices and Symmetric Matrices¶

Positive Definiteness in Complex Vector Spaces¶

9.4.4 The Principal Axis Theorem in Complex Space¶

Statement of the Principal Axis Theorem for Complex Vector Spaces¶

Comparing Unitary and Orthogonal Transformations¶

Applying the Spectral Decomposition to the Principal Axis Theorem¶

The Geometric Meaning of the Canonical Form in Complex Space¶

9.5 Summary¶

Review of the Theoretical Thread¶

Connections to Other Chapters¶

The Role of This Chapter in the Book¶

Concept Map¶