Quantitative Finance — the Mathematics of Markets · chapter 07

07Eigenvectors, SVD & Cholesky

Chapter 6 handed us a machine and left the picture hard to hold. A matrix grabs every arrow on the plane at once and swings it somewhere new, and once you stack a rotation on top of a shear, nobody can keep the whole motion in their head. This chapter is one move that makes it simple. Hidden inside almost every such transformation are a few special directions that come out pointing exactly the way they went in, only longer or shorter. Find those directions, rebuild your coordinate system out of them, and the tangle collapses into pure independent stretching. That single change of viewpoint is the entire chapter, and everything else is it wearing a costume. Diagonalization is the move applied. SVD is the move rescued for the matrices that refuse it. The condition number is the move read off as a single number, and Cholesky is the move run backwards to manufacture a shape you want. By the end you will be able to say in one sentence why a regression on two look-alike stocks blows up, and then build the machine that does the fit anyway.

Before the first eigenvector, a map. Chapter 6 gave us the transformation, the rank and the collapse that makes a matrix impossible to undo. This chapter takes those objects apart along their own axes, and a surprising amount of what comes later is waiting on it. Click any downstream node and watch the road run back to this page.

A map, not a lesson — who downstream borrows this page, and the one question it exists to answer.

① Tap a blue node on the road above — a gold road traces its debt back to this page.

② Flip the switch — do all the arrows turn?

flip the switch — watch every arrow

What you're looking at — the eigen-road: who downstream owes this page, and the one question it answers.

Ch 07, you are here — and the two arrows that refuse to turn

later chapters/methods that borrow the result — tap any node

under Ch 6's shear, most arrows swing to a new heading

Fig. 1. You are the gold stop on the course spine. Tap any blue chapter downstream — covariance (18), OLS (20), PCA (24), correlated Monte-Carlo (18/24), risk (31) — and a gold road traces its debt back here, naming the one thing it borrows: PCA is nothing but the eigen-directions of the covariance matrix. That's why Ch 6 could only hand this chapter a question. Flip the switch below and watch it: a matrix moves all of space at once, most arrows swing to a new heading — but two point exactly where they started. Finding those two, and how far they stretch, is the whole chapter.

Four of those roads matter enough to name now. The covariance matrix in Chapter 18 is symmetric and needs its eigenvalues to be well behaved. PCA in Chapter 24 is nothing but the eigen-directions of that matrix. Risk decomposition in Chapter 31 reads a portfolio through the same axes. And the correlated random draws that drive every Monte-Carlo in Part 4 come out of a Cholesky factor built here. So this is a small chapter with a long reach.

01★★ The directions that don't turn

Let's pick up exactly where Chapter 6 put us down. Take the matrix A = [2 1 ; 1 2], which by the keystone of that chapter means e₁ lands on (2, 1) and e₂ lands on (1, 2). Now feed it a whole circle of unit arrows and apply it to every one of them at once.

Most of the arrows turn. That is the ordinary behaviour of a transformation, and it is exactly what made the picture hard to hold at the end of the last chapter. An arrow points one way going in and a different way coming out, and the amount it swings depends on where it started. But watch the panel below carefully, because that word most is doing real work.

The parallel hunt — steer an arrow until the matrix stops turning it

Under A = [[2,1],[1,2]], drag the blue arrow x round the ring. Its output Ax swings to a new heading — hunt for the two where it stops turning.

drag the dot · tap the ring · ← → keys nudge

TURNED — off by 26.6°

What you’re looking at — a matrix turns almost every arrow, but two directions it only stretches.

the input arrow x you steer (length 1)

its output Ax — where A sends it

locked: Ax lies dead‑parallel to x — an eigenvector

the stretch at a lock is the eigenvalue λ (×3 and ×1 here)

Fig. 2. — Drag the blue arrow around the ring under A = [[2,1],[1,2]]. Almost everywhere its output Ax points somewhere new; at just two headings it comes out dead‑parallel — (1,1) stretches ×3, (1,−1) holds at ×1. Those fixed directions are the eigenvectors, the stretches are the eigenvalues, and Av = λv is simply the picture you just found by hand.

Drag the input arrow slowly around the circle and watch the angle-mismatch meter beside it. For nearly every direction the meter sits well off zero, meaning input and output point different ways. Then swing through the 45° diagonal and the meter drops dead to zero. The input (1, 1) fires out to (3, 3): same line, same direction, three times as long. Keep going and it happens once more at 135°, where (1, −1) comes out as (1, −1), completely unmoved.

Check the first one by hand, because it should not feel like a trick. A(1,1) is 1 times column one plus 1 times column two, which is (2,1) + (1,2) = (3,3). That really is 3 times the arrow we started with. The transformation did not turn it at all. It only stretched it.

Now the names, arriving late on purpose so each one is a label for something you already saw. A direction the transformation only stretches, and never turns, is an eigenvector of the matrix. The factor it gets stretched by is that eigenvector's eigenvalue, written λ. Our matrix has two of them: λ = 3 along (1,1), and λ = 1 along (1,−1).

Only the word is intimidating. Eigen is German for "own", so an eigenvector is one of the matrix's own directions, in the same sense that a bell has its own note. Hit a bell any way you like and it rings at the frequencies the bell itself decides. Push a matrix any way you like and these are the directions it will not fight you on.

And the whole idea compresses to one line, which now says something you have already watched:

A v = λ v — on this special vector, the entire matrix acts like a single number

Read the two sides carefully, because that is where the surprise lives. The left side is a matrix doing its full job on v: four numbers, a mix of columns, a transformation of all of space. The right side is one number multiplying v. For almost every vector those two things cannot possibly agree. On an eigenvector they agree exactly.

It's a line, not an arrow — and λ is allowed to be negative

walk the point t = 2.0

λ = +3.00 · same line, longer

What you're looking at — one vector walked up and down a whole direction, and what M does to it

v — the input, walked along the line by the slider

Mv — the output, and λ = how many times longer

the dashed eigen-lines — directions M only stretches

drag off the line → Mv points a new way, no λ

Fig. 3. Walk the blue vector up and down a line: its output never leaves that line, and the ratio stays pinned at λ — so every scalar multiple is the same eigenvector. Switch to the swap matrix and walk (1,−1): the output flips to the far end of the same line, λ = −1. Negative isn't broken; the line survived, you just walk it backwards. Drag the dot off the line and Mv swings away — no λ, because it was never an eigenvector.

One more thing to nail down before we go hunting properly, because it trips up everyone once. An eigenvector is not a particular arrow. It is a whole line through the origin. Drag the length slider in that panel and watch (2,2) come out as (6,6), and (−3,−3) come out as (−9,−9). Every one of them is stretched by the same 3, because the length was never what the matrix was preserving. The direction was.

The second card in that panel handles the case people find unsettling. Take the swap matrix [0 1 ; 1 0], which exchanges your two coordinates. Along (1,1) nothing happens at all, so λ = 1. Along (1,−1) the output is (−1, 1), which is the same line pointing the other way, so λ = −1. A negative eigenvalue is not a broken one. It means the direction survived as a line and got flipped end for end.

02Hunting stretches with a determinant

Finding those directions by dragging an arrow around a circle was a good way to meet them. It is a terrible way to compute them. Real matrices are bigger than 2×2, they do not live on a screen, and squinting at an angle meter does not scale past the picture. So we need a way to find the eigenvalues directly, and the route runs entirely through tools Chapter 6 already built.

Start with the most boring transformation there is. The identity matrix I has columns (1,0) and (0,1), which are e₁ and e₂ themselves. By the keystone of Chapter 6, a matrix's columns are where the basis lands, so I lands every basis arrow exactly where it already was. It is the do-nothing map, and Iv = v for every vector on the plane.

One λ dial, three panels — watch what I, λI, and A − λI each do to the same little square.

0 off the diagonal — so subtracting it has nothing to touch there. Only the diagonal can ever move.

λ drag → one dial drives all three

λ = 1.00 · A = [2 1 ; 1 2] fixed

A − λI det = 0.00

What you're looking at — the same unit square, put through three matrices, with one λ dial

I · its columns are e₁,e₂, so it lands every vector on itself — the do-nothing map (left, fixed).

λ down the diagonal — λI is a uniform scale (middle); the only entries subtraction can reach on the right.

A − λI drawn live; it goes flat (det = 0) exactly at λ = 1 and λ = 3 — those are A's eigenvalues.

Fig. 4. One dial, three honest pictures. I (left) is the do-nothing map: its columns are e₁ and e₂, so by Ch 6's keystone it lands every vector on itself — the test vector v and its image are the same arrow, lamp green. λI (middle) is that same idea scaled: drag λ and the whole grid inflates (λ=2 doubles), deflates (λ=½ halves), or collapses to the origin (λ=0) — and notice its matrix is λ on the diagonal, 0 everywhere else. So A − λI (right) is a real transformation you can watch, and dragging λ slides only the two diagonal entries: [2 1;1 2] at λ=0, [1 1;1 1] at λ=1, [−1 1;1 −1] at λ=3. That's the whole trick — "subtract λ down the diagonal" is not a ritual, because λI has nothing off the diagonal to subtract. Push further and the square goes flat exactly at λ=1 and λ=3: det = 0, the map collapses a direction — and those λ are the eigenvalues the rest of the chapter is chasing.

Now scale it. The matrix λI has columns (λ, 0) and (0, λ), so it multiplies everything by λ without turning anything. Drag the λ dial in that panel and the whole grid inflates and deflates uniformly. That is all λI ever means: a pure uniform scaling, written as a matrix so we can subtract it from another matrix.

Which is the move. Subtract that scaled identity from our A and you get A − λI, a real transformation with its own grid, and the panel draws it while you drag. At λ = 0 it is just A. Push λ up and the two diagonal entries slide down together, because subtracting λI only touches the diagonal. Nothing mysterious happened. We shifted a matrix by a uniform scaling.

Peel the keystone line, one honest algebra beat at a time — until it becomes a statement about collapse.

v = 0 into Av = λv and it reads 0 = 0 — true for every λ you like. so it singles out no stretch and names no direction — it says nothing. an eigenvalue needs a real, nonzero v to point at. (tap the rail again to close)

Step the 4 beats → each one changes only the equation and its picture.

standing rule: v ≠ 0 · tap to see why it's required

Av and λv: the same arrow, twice

What you're looking at — four algebra beats, each with its own picture, turning the keystone line into a fact about collapse

v is the eigenvector (blue); Av and λv land as the same arrow — three times as long, so λ = 3

beat 2 swaps λv for λIv: the identity I does nothing, so the picture doesn't move

subtract (beat 3) and the arrows cancel to a dot at the origin; factor out v (beat 4) and A−λI crushes a nonzero v to zero

aha — that's where (A−λI)v = 0 comes from: one subtraction. An eigenvector is a nonzero direction the shifted matrix sends straight to the origin — a collapse, det = 0.

Fig. 5. Peel the equation — from Av = λv to a vector crushed to zero: four honest algebra beats, each carrying its own picture, ending at (A − λI)v = 0 with v nonzero.

Now peel the equation, one honest step at a time. We start from the line the first section earned, Av = λv. Rewrite the right side using the do-nothing matrix, which changes nothing at all: Av = λIv. Move it across: Av − λIv = 0. Factor the v out of both terms, exactly as you would with numbers:

(A − λI) v = 0, where v is not the zero vector

Stop and read what that sentence claims, because it is the whole derivation. It says the shifted matrix A − λI takes a nonzero vector and sends it to the origin. Chapter 6 already told us what kind of transformation does that. A map that crushes a whole nonzero direction down to a single point has destroyed a dimension. It has collapsed space.

And collapse has a number attached. Chapter 6's determinant measures how much area survives a transformation, and it reads exactly zero when the plane folds flat onto a line. That matrix is called singular. So the eigenvalues are precisely the values of λ that make the shifted matrix singular:

det(A − λI) = 0 — the characteristic equation

The collapse dial — sweep λ until the grid goes flat. Where it flattens, the determinant is zero.

A−λI = [ 2.00 1 ; 1 2.00 ]

λ = 0.00

drag λ → · watch a gold arrow vanish

(λ−1)(λ−3) = 3.00

healthy grid · area 3.00

What you're looking at — the eigenvalue hunt is just watching for the flatten.

the unit square's image — its area is exactly |det(A−λI)|

the two eigen-directions; each arrow is crushed to a point at its own λ — that λ is an eigenvalue

the determinant curve; the dot touches zero at the very λ where the grid goes flat

between the roots (1<λ<3) det<0 — the square is flipped inside-out

Fig. 6. Ch 6 proved it: a matrix is singular — it flattens the grid — exactly when its determinant is zero. So the eigenvalue hunt needs no memorised formula. Just slide λ and watch A − λI: the shaded square breathes, flips inside-out between the roots, and at λ = 1 and λ = 3 it goes dead flat — area zero — while the determinant curve on the right touches zero at the very same instant. The gold direction it crushes to a single point is the eigenvector: the one line A leaves pointing where it was. That is why det(A − λI) = 0 isn't a rule to memorise — it is the sound of the grid folding flat.

That panel is where the derivation stops being words. Sweep λ and two things move together. On the left, the grid of A − λI deforms. On the right, the determinant traces out the curve (2−λ)² − 1, which multiplies out to λ² − 4λ + 3. Start at λ = 0 and the determinant reads 3, so area triples and the grid is healthy. Push to λ = 2 and it reads −1, meaning the plane got flipped over on the way past.

Now go slowly through λ = 1 and λ = 3. At each of those the grid flattens onto a single line and the determinant curve touches zero. Those are the two eigenvalues, and you found them without dragging a single arrow. Better than that: the line the grid collapses onto tells you the direction that got crushed, which is the eigenvector itself.

Factor the polynomial and it agrees: λ² − 4λ + 3 = (λ − 3)(λ − 1), with roots 3 and 1. Those are the same two stretch factors we found by hand in the arrow hunt, and nothing about the picture had to be trusted to get them.

Two honesty notes on scope. First, an n × n matrix gives a polynomial of degree n, so it has at most n eigenvalues. Second, some real matrices have no real eigenvalues at all, because that polynomial can fail to cross zero. We meet the cleanest example of that shortly, and it is the reason the second half of this chapter exists.

03★ Eigen-graph-paper

Now we cash them in. We have two directions this matrix only stretches, and we know the stretches are 3 and 1. What you do with that pair of facts is one of the best ideas in linear algebra.

Throw away the horizontal and vertical grid. It was never handed down from anywhere. It is just the paper we happened to draw on, and there is nothing about A that respects it. Rule new paper whose lines run along (1,1) and (1,−1) instead, and then ask what the same transformation looks like.

Change the paper — the same matrix A on eigen graph-paper

Predict first — how does A move the blob? Then Commit.

Pick a card, then press Commit

What you're looking at — one matrix A, two sheets of graph paper.

the blob — dashed = before A, solid = after A (same shape, moved)

the ×3 eigen-direction (1,1) — A only stretches here

the ×1 eigen-direction (1,−1) — A leaves it untouched

Fig. 7. The mess was never in the matrix — it was in the paper. On a square grid the blob A produces looks rotated and smeared, an ugly tangle. But re-rule the page along A's own eigen-directions — (1,1) and (1,−1) — and replay the identical transformation: no shear, no spin, one axis simply ×3.00 and the other ×1.00. Hit overlay both to confirm nothing about A changed. That independent per-axis scaling, with zero cross-talk, is exactly what diagonal means — before a single line of algebra.

Predict before you flip that switch. On the square grid, A takes a blob and produces something rotated and smeared, and if you had to describe the motion in words you would need several. Now flip to the eigen-grid and look at the identical action. The grid does not shear. It does not rotate. It pulls out to 3× along one diagonal and leaves the other diagonal exactly as it was.

Sit with that for a second, because it is the reveal the chapter is built on. The mess was never in the transformation. It was in the paper. On its own axes, this matrix does nothing more complicated than stretch two directions by two numbers, with no cross-talk between them at all.

Write those two numbers as a matrix and you get the simplest kind there is:

D = [3 0 ; 0 1] — a diagonal matrix

A diagonal matrix is one whose only nonzero entries sit on the top-left to bottom-right diagonal. Chapter 6 makes that concrete. Its first column is (3, 0), so the first axis is scaled by 3 and stays on the first axis. Its second column is (0, 1), so the second axis is scaled by 1 and stays put. Each axis is scaled on its own, and no coordinate ever leaks into another. That is the entire meaning of the word diagonal.

But there is a catch, and it is the one that makes people memorise instead of understand. The reader hands you a vector in ordinary coordinates, not in eigen-coordinates. So to use D at all, you need a round trip.

The round trip: A = P D P⁻¹ walks x = (3, 1) through three stations and lands on (7, 5)

step through the belt — and reverse it

On the belt — press Next ▶

P⁻¹ doesn't move the arrow — it only renames it. Predict: which station actually stretches it?

What you're looking at — one arrow, two languages. P⁻¹ and P are just translators between them; only D stretches.

the vector x — same arrow, read as (3,1) in the grid or (2,1) in the eigen-basis

its e₁=(1,1) part — the direction D stretches ×λ₁=3 (2→6)

its e₂=(1,−1) part — stretched ×λ₂=1, so it's left alone

the eigen readout & the answer — both routes reach (7,5)

Fig. 8. — A = P D P⁻¹ is three moves, not an algebra sandwich. P⁻¹ just renames x from grid-coords (3,1) to eigen-coords (2,1); D stretches each eigen-coord by its own λ (2→6, 1→1); P reads the result back into the grid as (7,5). That renaming is change of basis — the same arrow, two languages. CHECK reaches the same (7,5) straight from the columns of A.

Step the three stations in that panel with x = (3, 1) and watch the coordinates change meaning. Station one translates in: rewrite (3,1) in eigen-language by asking how much of each eigen-direction it contains. The answer is 2 parts (1,1) plus 1 part (1,−1), since 2(1,1) + 1(1,−1) = (3,1). So in eigen-coordinates the vector reads (2, 1).

Station two is the easy one. Apply D, which multiplies the first coordinate by 3 and the second by 1, giving (6, 1). Station three translates back: rebuild the actual vector from those parts, 6(1,1) + 1(1,−1) = (7, 5).

Check it against the matrix directly. A(3,1) = 3(2,1) + 1(1,2) = (7, 5). Identical. The round trip is not an approximation or an alternative method. It is the same transformation, taken apart into the only three things it was ever doing.

Both translations are themselves matrices, and Chapter 6 told us that doing one transformation after another is matrix multiplication. Build P by putting the eigenvectors in as columns, P = [1 1 ; 1 −1]. Then P⁻¹ is the map into eigen-language and P is the map back out, so the three stations in order are:

A = P D P⁻¹ — translate in, scale, translate back

That is diagonalization, and the sandwich hides no cleverness at all. Read right to left, following the vector: P⁻¹ puts you in the good coordinates, D does the only interesting work, and P puts you back in the coordinates you asked the question in.

Break symmetry and watch the eigen-axes lean — line b up with c and they snap to a right angle

drag both — line b up with c to snap 90°

b (top-right of A) c (bottom-left of A)

skew — axes at 53.1°

What you're looking at — the two eigen-directions of A = [2 b; c 2], found by looking

e₁, first eigen-axis

e₂, second eigen-axis

angle between them

b = c → axes ⟂ → Q⁻¹ equals Qᵀ

Fig. 9. The matrix A = [2 b; c 2] stretches space along two special directions — its eigen-axes, found here by looking, not solving. Start broken (b=1, c=4): the axes lean together, the angle reads 53.1°, and the red skew lamp is lit. Drag c back toward b and the angle climbs; the instant b = c the matrix turns symmetric, the axes snap to 90.0°, and the green lamp fires. That right angle is the whole payoff: translating back becomes a pure un-rotation, so Q⁻¹ and Qᵀ print the same grid — the inverse is just the transpose. That's why this course cares about symmetric matrices: every covariance matrix later lands in exactly this case.

Our A is symmetric, meaning it equals its own transpose, so the entry in row 1 column 2 matches the one in row 2 column 1. That single property runs through the rest of this course. Drag the off-diagonal knob in that panel and watch the two eigen-axes lean toward each other as the symmetry breaks. Snap the entries back into agreement and the axes lock at a right angle.

That perpendicularity buys something concrete. When your new axes are perpendicular and unit length, translating back is just an un-rotation, and the inverse of a rotation is its transpose. So for symmetric matrices the sandwich simplifies, with Q holding the perpendicular eigen-directions as columns:

A = Q D Qᵀ — for symmetric A, the eigen-axes are perpendicular

An honesty note before we move on, because it sets up the next section. Not every square matrix can be diagonalized. The shear [1 1 ; 0 1] has λ = 1 twice, and only one direction to go with it, namely (1,0). One axis is not enough to rule paper with, so the round trip cannot be built. That is a real failure and we are about to hit two worse ones.

04★ When nothing comes out parallel

The eigen-story has two holes in it, and they are not edge cases. They are the ordinary situation in finance.

Break the eigen-hunt: two matrices where the search that always worked simply can't.

spin the input arrow

drag the blue arrow, or slide

spin it — hunt an arrow that stays put

What you're looking at — the chapter's own eigen-hunt, rerun on two matrices where it is doomed.

Tab 1, the quarter-turn: the input arrow you spin and the output it maps to are always 90° apart — no arrow ever comes out parallel, so λ²+1=0 has no real root and the count sticks at 0/2.

Tab 2, the 252×2: a 2-D input maps into a 252-D output. “Is the output parallel to the input?” can't even be asked — they live in different-sized spaces.

The reframe: the demand “come out parallel to yourself” was too strict. That's why a second decomposition — the SVD — must exist. It is not redundancy.

Fig. 10. Two honest failures. Tab 1 reruns the keystone hunt on the quarter-turn [0 −1; 1 0]: spin the input arrow anywhere on the circle and the output comes out stubbornly 90° off — no direction is ever held, the count stays 0/2, and λ²+1=0 has no real root (its curve floats entirely above zero). Tab 2 takes a 252×2 data matrix: drag the 2-D input and its 252-D output reshapes, but press “is the output parallel to the input?” and the panel refuses — the two arrows live in different-sized spaces, so the eigen-question can't even be asked. The demand was too strict — which is exactly why the SVD has to exist.

The first hole is a pure rotation. Set that panel to the quarter-turn matrix [0 −1 ; 1 0] and run the arrow hunt from the start of the chapter again. Drag all the way around the circle and the angle-mismatch meter never touches zero, because every arrow comes out turned by 90°. There is no unturned direction, and the characteristic equation agrees: λ² + 1 = 0 has no real root.

The second hole is worse and more important. Chapter 6's real object was a 252 × 2 data matrix, two predictors against a year of trading days. That matrix takes a vector with 2 entries and returns one with 252. Flip the panel to that case and try to ask the question. "Does the output point the same way as the input?" is not a question you can answer, because the input and the output do not even live in the same space. Rectangular matrices cannot have eigenvectors at all.

So the demand was too strict. We insisted that output be parallel to input, and that insistence is what is failing. Drop it, and ask the one question that works for any matrix of any shape: feed in the whole unit circle, and see what shape comes out.

Circle in, ellipse out — drag the two columns anywhere; the output is always an ellipse, and its axes hand you σ, U and V.

Grab either accent dot and drag it anywhere — squash, shear, spin, mirror. The image on the right stays an ellipse every time.

σ₁=1.78 σ₂=0.92 · an ellipse

What you're looking at — every matrix sends the circle to an ellipse, and its axes are U, Σ, V.

blue = the unit circle and its image (same points, moved)

gold = the long axis σ₁; its two preimages on the circle are a column of V

cyan = the short axis σ₂, always ⟂ to gold — that's why U and V are orthogonal

drag the two columns · collapse puts col 2 on col 1 → σ₂→0, the singular case of Ch 6

Fig. 11. Stop asking for one held direction and ask what the whole unit circle becomes. Drag the two columns anywhere — squash them, shear them, spin them, mirror them — and the image on the right stays an ellipse, every single time; the lamp never goes out. Its long axis is σ₁, its short axis σ₂ (printed as half-lengths), the two are always perpendicular, and the circle points that landed on them light up in matching colours — that pair of perpendicular sets is exactly V (in) and U (out), with the stretches Σ between them. Hit collapse it to slide column 2 onto column 1: the ellipse flattens to a line and σ₂ → 0.000 — the singular, collapsed case from Ch 6, now with a number on it. That's the rescue, and it costs nothing: every matrix of every shape has natural axes — just two sets instead of one.

Drag the columns to any matrix you like and the answer never changes. The circle always comes out as an ellipse. Squash it, shear it, rotate it, make it as ugly as you can, and the output is still an ellipse. The only degenerate case is the singular one, where the ellipse flattens into a straight segment, and that is Chapter 6's collapse showing up again.

That ellipse hands you three facts for free, and they are all readable off the picture. It has two perpendicular principal axes, which is true of every ellipse. Its half-lengths are two numbers. And the input directions that landed on them are two more perpendicular directions, back on the circle. Give them names:

σ₁, σ₂ the singular values · V the input axes · U the output axes

Rotate, stretch, rotate — and where SVD swallows diagonalization

▸ or tap a station

σ₁ 1.618σ₂ 0.618

σ₁·σ₂ = 1.000det A = 1.000

σ₁·σ₂ = |det A| — SVD survives the shear

U ≠ V — two different turns

start: the unit circle

What you're looking at — every matrix A is three moves: rotate (Vᵀ), stretch (Σ), rotate (U).

the circle → ellipse: the whole space being carried through A

the two input points that report each hidden rotation

σ₁, σ₂ — the stretch factors, and σ₁σ₂ = |det A|

ghost of A·(circle): the three moves land on it. Symmetric → U=V, so diagonalization is the special case.

Fig. 12. SVD in one sentence: rotate, stretch, rotate. Step the unit circle through A = UΣVᵀ — Vᵀ turns the input directions onto the axes (a circle looks unchanged, so the cyan input points do the reporting), Σ pulls it into an ellipse with the two singular values on their own axes, and U swings it into place, landing on the ghost of A·(circle). On the shear that has no eigenbasis it still works, with σ = 1.618 / 0.618 whose product is det A = 1. Switch to symmetric and the two rotations lock to one Q — the “they coincide” lamp fires and the σ become the eigenvalues 3, 1. That is why diagonalization was only ever the special case.

Now step the three stations in that panel and the decomposition writes itself. Station one rotates the input so that V's perpendicular directions line up with the coordinate axes. Station two stretches along those axes by σ₁ and σ₂, which is a diagonal matrix and nothing more. Station three rotates the result so it sits on the output axes U. Three moves: rotate, stretch, rotate.

A = U Σ Vᵀ — the singular value decomposition

Run it on the shear that defeated diagonalization and watch it work anyway. For [1 1 ; 0 1] the panel reports σ₁ = 1.618 and σ₂ = 0.618. Multiply them and you get 1.000, which is exactly this matrix's determinant, and that is not a coincidence. The determinant is the area factor, and stretching by σ₁ one way and σ₂ the other scales area by their product. The two rotations do not change area at all.

Two things, said plainly. First, singular values are always real and non-negative, because they are half-lengths of an ellipse. So this decomposition exists for every matrix ever written — square, rectangular, rotating, collapsing, all of them. Second, there are now two sets of axes rather than one, and that is not a complication for its own sake. Input and output simply need not agree, and for a rectangular matrix they cannot, since they live in spaces of different sizes.

Then set the panel to a symmetric matrix and watch the two stories become one. For our original A, the ellipse's axes are the eigen-axes, so U and V become the same rotation Q, and the singular values 3 and 1 are the eigenvalues. SVD is not a rival to diagonalization. It is diagonalization with the one impossible demand removed.

05How thin is your ellipse

The ellipse carries a health check, and reading it is the most practically useful thing in this chapter. Take the longest half-axis σ_max and the shortest σ_min, and form their ratio:

κ = σ_max / σ_min — the condition number, which is just "how far from round"

A matrix with κ near 1 maps the circle to something nearly circular. It treats every direction roughly alike, and we call it well-conditioned. The shear we just measured has κ = 1.618 / 0.618 = 2.618, which is perfectly healthy. A matrix with a huge κ maps the circle to a thin sliver, and we call it ill-conditioned.

Thin the ellipse — drag two look-alike stocks toward collinear and watch κ = σ_max/σ_min blow up

how alike are your two predictors?

alike → collinear drag ε →

well-conditioned — safe to invert

σ_max × σ_min1.00000

|det| = ε1.00000✓

What you're looking at — κ is how far from round the matrix acts

blue ellipse = the unit circle pushed through the matrix (round input shown as the faint ring)

σ_max = the long stretch — barely moves as the columns close in

σ_min = the short stretch — falls 10× per 10× squeeze; that fall is κ

Fig. 13. It is never σ_max that fails you. Squeeze the columns 10× and σ_min falls 10× while σ_max holds — and κ is just that fall, read as one number. The cross-check σ_max×σ_min = |det| = ε quietly proves the ellipse's shrinking area.

Now build the sliver deliberately, using Chapter 6's own failure. Set the columns to (1, 0) and (1, ε) and drag ε down toward zero, so the two columns slowly become the same arrow. This is the two-look-alike-stocks matrix, drawn honestly. At ε = 0.1 the readout gives σ_max = 1.416 and σ_min = 0.0706, so κ = 20.05. Tighten to ε = 0.01 and you get σ_max = 1.414, σ_min = 0.00707, and κ = 200.0.

Watch which number is doing the damage. The long axis barely moves, drifting from 1.416 to 1.414. It is σ_min that crashes, falling by a factor of ten each time the columns get ten times closer. Chapter 6 called this near-singular and could only point at it. Now it has a size.

Why is a small σ_min dangerous? The three stations answer it with nothing added. To undo a transformation you reverse each step: un-rotate, un-stretch, un-rotate. Un-stretching a direction that was multiplied by σ means dividing by σ. So the inverse divides by every singular value, and a tiny σ_min becomes an enormous 1/σ_min.

The error amplifier — a 0.001 nudge becomes 0.1414, and the blow-up is exactly 1/σ_min

nudge direction (drag the dial, or slide)

worst! amp = 1/σ_min = 141.42×

Steer the blue nudge; watch the gold answer-error. Vertical = the squashed direction.

What you're looking at — the same 0.001 nudge, and how big an answer-error it becomes.

Δb — the nudge on b, always 0.001 (a speck)

Δx — the error it makes in the solved answer

the worst direction · blow-up = 1/σ_min

the safe direction · barely amplified (≈1×)

Fig. 14. Solving Ax = b means multiplying by A⁻¹, and inverting un-stretches — it divides by every singular value σ. Send in the tiny nudge Δb (always 0.001, a speck on the dial) and the answer moves by Δx = A⁻¹Δb, drawn beside it at the same true scale so the blow-up is physical, not claimed. Steer along the safe direction and the answer barely twitches (≈1×); rotate onto the squashed direction and the same speck is multiplied by 1/σ_min before you ever see it — at ε = 0.01 that's 141.42×, turning 0.001 into 0.1414. The worst case is exactly 1/σ_min; that is why ill-conditioned fits explode, and it is the same number an amplifier calls its noise gain.

That panel makes the consequence physical. Solve Ax = b at ε = 0.01, then nudge b by 0.001 in its second slot, which is a rounding error you would never notice. The answer moves by 0.1414. That is a 141.42× amplification of your nudge, and the number is not arbitrary: 1/σ_min = 1/0.00707 = 141.42, exactly. The worst-case amplification a matrix can inflict is 1/σ_min, and this nudge found it.

So now the story from Chapter 6 closes properly. Regress a stock on two nearly identical predictors and your data matrix has two nearly identical columns. Its ellipse is a sliver, σ_min sits near zero, and κ is huge. The fit still looks fine, because the reachable sheet barely moved. But the coefficients are computed by dividing through that tiny number, so the smallest speck of noise in your returns swings them wildly. This is exactly why Chapter 6's weights jumped from (−28, 30) to (−298, 300) on a tenfold tightening, and now you can predict it before running anything.

A note on the word, since it sounds like dry numerical analysis and is not. An engineer would call κ the noise gain: the factor by which the worst-case input error gets multiplied on the way to the output. An amplifier with a gain of 200 amplifies the hiss as faithfully as the music, and a matrix with κ = 200 does the same thing to your data.

06★ Inverting the un-invertible

The un-stretching idea has one obvious hole, and walking straight into it gives us the last tool of the linear-algebra chapters. What happens when a singular value is exactly zero?

Chapter 6 was blunt about this case. A matrix that collapses space merged two different inputs into one output, so the information about where you started is gone, and no formula recovers it. Dividing by σ = 0 is not a technical difficulty. It is the arithmetic complaining about a real, physical loss.

Build the undo — assemble A⁺ one station at a time, and meet the honest refusal in the middle

② invert each stretch

pick a matrix · build ① → ③

What you're looking at — the pseudo-inverse A⁺ built as three reversed stations, with one honest refusal.

the stations — Uᵀ, Σ⁺, V, the undo read right→left

the inverted stretch 1/σ — the number to watch

free rotations, and the coordinate recovered exactly

σ = 0: dividing is refused; that direction is lost

Fig. 15. The pseudo-inverse A⁺ with nothing to memorise — it is the three stations of the SVD run backwards. Uᵀ and V just un-rotate, always free; the only real work is the middle, where every nonzero stretch flips to 1/σ. When a stretch is σ = 0 the widget refuses the division: dividing there would invent information Chapter 6 proved was destroyed, so you leave it at 0. Feed (5, 7) and it comes back (5, 0) — the 5 recovered exactly, the 7 gone because it lived in the crushed direction. Flip to full rank and nothing is refused: every stretch inverts and A⁺ is just the ordinary inverse. That one refusal in the middle is the whole idea.

So build the best possible undo, one station at a time, and let the panel refuse where it must. Reverse the last rotation, which is free, because rotations are always undo-able. Reverse the stretches, and for every nonzero σ that means multiplying by 1/σ. Then reach a zero σ and stop: leave that direction at zero rather than inventing a number for it. Reverse the first rotation and you are done.

That assembled map is the pseudo-inverse, written A⁺. Notice there was nothing to memorise. It is three stations reversed with one honest refusal in the middle, and the refusal is the interesting part. The panel's rank-1 example makes it concrete: feed [1 0 ; 0 0] the target (5, 7) and it returns (5, 0). The 5 was reachable, so it is recovered exactly. The 7 lived in the crushed direction, so the honest answer is to claim nothing.

Which sets up the payoff Chapter 6 promised and deliberately left open.

Chapter 6's unanswered question, now in 3-D: find the closest reachable point to b — by hand, then by the pseudo-inverse.

① drag the blue point on the sheet — hunt the closest spot to b

point (1.0, 1.0, 0)

distance to b 4.583

your best so far 4.583

② stop guessing — let the machine do it →

Drag to hunt the closest point.

What you're looking at — the closest reachable point to b, found two ways that turn out to be one

least squares — drag to get as close as the reachable sheet allows

pseudo-inverse — reverse what can be reversed, stay silent about what was destroyed: A⁺b = (2,3)

both land on the identical point (2,3,0) — the foot of the perpendicular from b

Fig. 16. The payoff Chapter 6 promised and could not compute. The reachable sheet is every point A can build; the target b = (2,3,4) floats above it, out of reach. Drag the blue point and the distance bottoms out near 4.000 at (2,3,0) — its neighbour (2,4,0) already reads √17 ≈ 4.123. You can get close by hand. Now press run the pseudo-inverse: it returns the weights (2,3), pushes them back through A, and drops a gold marker on the exact same point — with the perpendicular drawn. Least squares and the pseudo-inverse are not two methods that agree; they are one object described twice: "get as close as the set allows" and "undo what can be undone." (Chapter 20 adds the statistics — the reason this is the right thing to minimise — not the computation, which is done.)

That panel replays the overdetermined setup from the end of the last chapter. The reachable sheet is the flat plane at z = 0, and the target b = (2, 3, 4) floats above it, unreachable, exactly as real returns always do. Chapter 6 argued geometrically that the best you can do is drop a perpendicular, landing on (2, 3, 0) at distance 4, and that any neighbour like (2, 4, 0) is worse at √17 ≈ 4.123.

Now press the button and let the pseudo-inverse answer instead. It returns the weights (2, 3), and running those back through the matrix lands on (2, 3, 0). The same point. Not close to it, not an approximation of it, but the identical point the perpendicular found.

So the two ideas are one object seen from two sides. Least squares is the geometric description: get as close to b as the reachable set allows. The pseudo-inverse is the machine that computes it: reverse what can be reversed, and stay silent about what was destroyed. Chapter 20 will put the statistics on top of this, but the computation is finished right here.

07The well-shaped matrices

One family of symmetric matrices behaves so well that the rest of this course leans on it constantly. Symmetric already bought us a lot: real stretches, along perpendicular axes. Now add one more condition and watch what it rules out.

Three circles, three fates — when is a symmetric matrix positive‑definite?

tap any card ▶ to run its matrix

tap a card ▶ run its matrix

What you're looking at — three symmetric matrices, one unit circle, three fates.

the (1, 1) axis — always stretches outward (λ>0)

the (1,−1) axis — the fate‑carrier: stretch, flip, or vanish

arc where xᵀAx > 0 — that direction is pushed out

arc where xᵀAx < 0 — that direction is folded through 0

The rule in your words: every λ > 0 means no flip and no collapse — the circle can only inflate. That is positive-definite, and the formal test xᵀAx > 0 (Ch 18) is the very same statement read off the coloured circle. Card 2's entries are all positive, yet its −1 eigenvalue still folds an axis red — positive entries are not positive-definite.

Fig. 17. Three symmetric matrices, one unit circle, three fates. Tap a card and watch the circle transform: matrix 1 inflates it into a full ellipse and every direction is pushed outward — positive-definite. Matrix 2 has an eigenvalue of −1, so at the halfway mark the circle pinches to a line and reopens with the (1,−1) axis flipped through the origin — indefinite, even though every entry is positive. Matrix 3 has an eigenvalue of 0, so that axis vanishes and the circle flattens onto the line through (1, 1) — semi-definite / collapsed. The reader's rule: all λ>0 means no flip and no collapse. A negative λ is a flip, a zero λ is a collapse — so “all positive” is exactly “pushes every direction outward,” which is what xᵀAx>0 (Ch 18) says in symbols.

Three symmetric matrices act on the same unit circle in that map, and the three fates are the whole definition. Our original A has eigenvalues 3 and 1, both positive, and it inflates the circle into a full ellipse that pushes outward in every direction. Swap the entries to [1 2 ; 2 1] and the eigenvalues are 3 and −1, so one direction gets flipped inside out. Take [1 1 ; 1 1] and the eigenvalues are 2 and 0, so the circle flattens onto a line.

A symmetric matrix whose eigenvalues are all positive is called positive-definite. In the pictures, that means it stretches every single direction outward: no flips, because a flip needs a negative eigenvalue, and no collapse, because a collapse needs a zero one. It is the well-shaped case, and the name is worse than the idea.

You will meet the formal definition as xᵀAx > 0 for every nonzero x, and it is worth flagging now so it is not a shock later. In plain words that expression says the matrix never turns a vector back against itself. Grounding it properly needs the dot product doing work we have not asked of it yet, so we take it on the pictures here and Chapter 18 makes it formal. One more honest warning: positive-definite does not mean positive entries. [1 2 ; 2 1] is all positive numbers and it is not positive-definite.

Halve the stretches — a matrix square root, demystified: on its own axes A is two numbers, so take the root of each

tap ③ to push a circle through √A, twice ↻

√A = 1.3660.3660.3661.366

1.366² + 0.366² = 2.000
2·(1.366)(0.366) = 1.000

A's stretches: ×3 and ×1

positive ⇒ every stretch has a real √

What you're looking at — a square root of A is any map that, done twice, gives A; on the eigen-axes A is just two numbers, so root each.

gold = A — its two eigen-stretches ×3 and ×1; A·(circle) is the target ellipse

blue = √A — each stretch square-rooted (√3, √1) → assembled [1.366 0.366; 0.366 1.366]

violet = A straight — the independent check; twice through √A lands on it → green lamp

Fig. 18. A "square root" of A is any map that, done twice, gives A back — and on A's own eigen-axes that stops being exotic. There A is just two stretch numbers, ×3 and ×1; to undo it in two equal steps you need the number that times itself gives each — the square root, √3 ≈ 1.732 and √1 = 1. Keep the axes fixed and reassemble: √A = [1.366 0.366; 0.366 1.366]. Press ③ apply twice and watch the unit circle go through √A (blue), then √A again (gold) — landing exactly on A·(circle), the match lamp firing. Check it by hand: 1.366² + 0.366² = 2.000 and 2(1.366)(0.366) = 1.000 — the entries of A. Switch to the indefinite card and the second axis asks for √(−1): the halving leaves the real plane, and no real square root exists. That's the whole trick — halving a positive number is something you've done since school.

Now collect the reward, because positive stretches are exactly what a square root needs. A square root of a matrix is any transformation that, applied twice, gives you the original. That sounds impossible until you look at it on the eigen-axes, where the matrix is only two numbers.

Halve them. If the eigen-stretches are 3 and 1, take √3 and √1, keep the same axes, and build √A = Q √D Qᵀ. Apply it twice and each axis gets stretched by √3 and then by √3 again, which is 3. That panel prints the result for our matrix: √A = [1.366 0.366 ; 0.366 1.366]. Square it and the entries come back as 1.366² + 0.366² = 2.000 on the diagonal and 2(1.366)(0.366) = 1.000 off it, which is A exactly.

The positivity was doing real work back there, not decorating the theorem. A negative eigenvalue would ask you for the square root of a negative number, and the transformation would walk straight off the real plane. Positive-definite is the condition that keeps every halved stretch real.

08Cholesky, the shape-maker

We already have one square root, so the last tool in this chapter is not a new idea at all. It is the same object, bought cheaper. And it arrives with a job description that makes it worth carrying.

The eigen-square-root Q √D Qᵀ is honest work, but look at what it cost. You had to find every eigenvector first, and for a matrix of any real size that is expensive. So here is the trade: give up on making the square root symmetric, and demand instead that it be lower-triangular, meaning every entry above the diagonal is zero. That shape can be computed directly, entry by entry, without an eigenvector anywhere.

M = L Lᵀ — the Cholesky decomposition, with L lower-triangular

For our familiar M = [2 1 ; 1 2] the factor comes out as L = [1.4142 0 ; 0.7071 1.2247]. Multiply L by its own transpose and you get 2, 1, 1, 2 back. This only works because M is positive-definite, for exactly the reason we just met. The recipe takes a square root at every step down the diagonal. A flip or a collapse would hand it a negative or a zero to root, and the factor would leave the real plane.

The shape-maker — push a round cloud through L and out comes your ellipse

reshape the target (the correlation baked into M)

press PUSH — cloud becomes M

What you're looking at — L is not bookkeeping; it is the machine that manufactures a shape.

the ellipse M encodes (axes 1.73 & 1.00)

Cholesky L — each point's path, no eigenvector used

symmetric √M — different paths, same ellipse

the one point you follow

Fig. 19. A round cloud of points, deliberately shapeless. Press PUSH THROUGH L and every point flows along its line to its image — out comes the exact ellipse that M = [2 1; 1 2] encodes, with semi-axes 1.73 and 1.00 along (1,1) and (1,−1). The panel shows L, computed with no eigenvector anywhere, and L Lᵀ printing the target M straight back. Reshape the target and all three nonzero entries of L recompute live. The two journey lines are why L is lower-triangular: the first coordinate is scaled only (that's the 0 up top), the second picks up a share of the first. Now hit compare √M: the symmetric square root [1.37 0.37; 0.37 1.37] drives every point down a different path — yet lands on the identical ellipse. A square root of a matrix is no more unique than √4; the shape is what's real. This is the standard way correlated scenarios are manufactured (Ch 18 & 24).

Now give L a visible job, because a factorization with no purpose is just bookkeeping. Start with a round cloud of points, a circle's worth of directions with no preference in any of them. Push every one of them through L and watch what the cloud becomes.

It becomes the ellipse that M encodes, with semi-axes 1.732 and 1 lying along (1,1) and (1,−1). Drag the target shape in the panel and L's three numbers update to match it. That is the whole point of Cholesky in one sentence: L is the transformation that turns a shapeless round cloud into any ellipse you name.

Flip the panel's switch and something worth noticing happens. The symmetric square root [1.366 0.366 ; 0.366 1.366] produces the identical ellipse, by a different route. A matrix does not have one square root any more than 4 has one. Both are correct, and Cholesky is the one you can compute cheaply.

An honest flag on where this is going, because the real motivation is not available yet. In Chapters 18 and 24, M will be a covariance matrix, and this exact move becomes the standard way to manufacture correlated random returns out of independent ones. Draw a round cloud of independent noise, push it through L, and out comes noise carrying the correlation structure you asked for. That sentence needs variance, which we have not defined. So here it stays pure geometry, and the geometry is already complete.

The payoff map — one move, six costumes: every tool here is the same change of axes

Tap the hub or any spoke → watch its chain light and its one line print.

the map so far

One move — change to a matrix’s own axes.

Tap any node to replay it in a costume.

What you’re looking at — the whole chapter as one map. Every tool is the same move: turn to a matrix’s own axes, where the tangle becomes plain stretching.

the keystone — eigenvectors, the directions a matrix only stretches; a lit chain glows gold.

the six tools — eigen-hunt, diagonalize, condition κ, pseudo-inverse, Cholesky: that one move in a costume.

SVD — the honest general one: every matrix, always exists (eigen-decomposition can fail — the dashed line marks the split).

the forward ring — what this move becomes in Ch 18/24/31; tap «show where it goes».

Fig. 20. · One move, six costumes — tap a spoke to light the minimal chain it needs and print its one-line result; reveal the forward ring to see what the move becomes in later chapters, or grey it all to the single sentence at the centre.

So here is the chapter as one image. Every matrix has directions it treats simply, and finding them turns a tangle into independent scaling. When those directions come out parallel to where they went in, they are eigenvectors, the stretches are eigenvalues, and the round trip through them is diagonalization. When no such directions exist, the circle-to-ellipse picture gives you two sets of axes instead of one, and SVD works on every matrix there is. The ratio of the biggest stretch to the smallest tells you how badly inverting will amplify your noise. Inverting what you can and refusing what you cannot gives least squares. And running the stretches backwards, halving each one, manufactures a shape.

Chapter 8 picks this up from an angle you can already feel coming. A positive-definite matrix turns a circle into a full ellipse, which is the same thing as saying a certain surface curves upward in every direction like a bowl. Optimising a portfolio means finding the bottom of exactly that bowl, and doing it means differentiating a function of many variables at once. That is the gradient, and its behaviour is governed by the eigen-structure you just spent a chapter building.

‹ prevVectors & Matrices next ›Multivariable Calculus & Optimization

iolinked.com

Written by Ajai Raj