06Vectors & Matrices

Chapter 5 closed on a grid it could not handle. Every number in that chapter was a scalar, one price or one rate at a time, and the machinery worked beautifully for exactly one asset. Nobody holds one asset. A year of daily returns for forty stocks is 10,080 numbers arranged in rows and columns, and no amount of careful scalar algebra will make that tractable. So here we build the two objects that will carry the rest of the course. A vector is an ordered list of numbers, and a matrix is what transforms one. I want to give away the punchline now, because the punchline is the reason to walk the road. A matrix is not a box of numbers with a strange multiplication rule attached. It is a function that moves all of space at once, and its four little entries are nothing more than a record of where two arrows landed. Once you see that, the whole vocabulary that usually has to be memorised collapses into one question. Did the transformation keep space full, or did it crush space flat? Everything else in this chapter, and the reason a regression on two nearly identical stocks explodes, is that one question asked in different words.

Here is where this chapter sits. Chapter 5 gave us e^{rT}, e^{−rT} and additive log returns, and every one of them tracks a single stream of money. Now the objects get wider. Click any downstream node and watch the road run back to this page.

Let's start with the smallest object that holds more than one number.

01One list, two pictures

Take two numbers and write them in a fixed order: (3, 2). Chapter 1 already gave you one way to read that. It is a point on the plane, three steps right and two steps up, and you can put your finger on it. Draw an arrow from the origin out to that point and you have the second reading, which is the same thing wearing a different coat. The arrow has a direction and a length, and both of those are fully determined by the two numbers.

Add a third number and nothing breaks. (3, 2, 5) is a point in the room you are sitting in: three steps along the floor, two steps across it, five steps up. You can still picture it, just about.

Now the second picture, and it comes from the last chapter. A stock's Monday log return is one number, and Tuesday's is another. Stack a whole trading year in order and you are holding 252 numbers in a fixed sequence. That is exactly the same kind of object: an ordered list. So a year of Apple is a single point in 252-dimensional space, and I want to be blunt about what that sentence does and does not ask of you.

It does not ask you to see it. Nobody can picture 252 dimensions, including the people who work in them every day, and the honest thing is to say so out loud. What it asks is much smaller. Every operation we are about to define works entry by entry, so whatever worked on the two-number list works letter for letter on the 252-number one. Drag the dimension dial in that panel and watch the drawing give up while the arithmetic strip beside it keeps running. That gap between what you can draw and what you can compute is the whole trick, and it never causes trouble again.

Now the definition, with both pictures in hand. A vector is an ordered list of numbers, and its dimension is how long the list is. Order matters, because (3, 2) and (2, 3) are different points, and each slot means a specific thing: a coordinate, or a Tuesday.

02Scale it, add it, mix it

There are only two things you can do to a list of numbers, and both are obvious once written down. You can scale it, which means multiply every entry by the same number. You can add two lists of the same length, which means add them entry by entry. That is the complete toolkit.

Each move has a picture on the plane, and the pictures are worth holding. Scaling by 2 stretches the arrow to double its length without turning it at all. Scaling by −1 flips it to point the opposite way. Adding two vectors lays the second arrow's tail on the first arrow's head, and the sum is the arrow from the origin to where you finish. Toggle the panel between the arrow view and the entry view and the two stay locked together, because they are the same arithmetic.

Now do both at once, on something real. You put 60% of your money into Apple and 40% into Google. Apple's week of returns is one vector, Google's is another, and your portfolio's week is 0.6 times the first plus 0.4 times the second. Scale each one, then add them up.

Drag the weight slider and watch the blended stream form. At 60/40, Monday's two returns of +1.2% and +0.5% become 0.6(1.2) + 0.4(0.5) = 0.92%, and the whole week comes out as 0.92, 0.16, 0.90, 0.68, −0.82. Slide the weight to 100/0 and the blend snaps onto Apple's own line, which is the sanity check you should demand of any mixing rule.

That move has a name, and I am naming it late on purpose so it arrives as a label for something you already did. Scale each vector by a weight, then add the results, and you have formed a linear combination. Write it once:

c₁v₁ + c₂v₂ + … + cₙvₙ

Flag this one hard, because it is the operation the entire chapter is built on. Every matrix, every column space, every rank and every least-squares fit ahead of us is a linear combination in a costume. A portfolio is one. When you meet the same move inside matrix multiplication in a few pages, it will feel like recognition rather than a new rule.

03★ What "linear" actually means

If mixing is the operation we care about, the next question is which functions respect it. Chapter 1 defined a function as a map from a domain to a codomain, taking single numbers in and giving single numbers out. Now the inputs and outputs are vectors. A function takes a point of the plane and sends it somewhere else on the plane.

And here we have to fight a word. School burns in "linear means the straight line y = mx + b", and that meaning will actively mislead you for the rest of this chapter. A shear that slants the whole plane sideways is linear. A curve-free map that shifts everything two units right is not. So forget the graph and look at what happens to space itself.

Watch the grid, because the grid is the test. Apply the transformation and three things must survive. Straight lines stay straight. Parallel lines stay parallel, and the spacing between them stays even. The origin does not move. Run the rotation and all three hold. Run the shear and all three still hold, even though the picture looks slanted and wrong to the eye. Then hit the bend and predict before you look: the gridlines curve, so the first condition dies, and the map is not linear.

A quick honesty note on the origin. A map that shifts everything by a fixed amount preserves straightness and parallelism perfectly, and it still fails, because it moves the origin. That kind of map has its own name, affine, and it is a linear map with a shift bolted on afterwards. We will meet it again as the intercept in Chapter 20's regression.

Now write the visual test as arithmetic, because arithmetic is what we can compute with. If mixing is the operation and the grid stays a grid, mixing before the map and mixing after it must agree. In symbols, for any vectors u and v and any number c:

T(u + v) = T(u) + T(v) and T(cv) = c·T(v)

Take both roads in that panel and watch them land on the same dot. Road one adds u and v first, then pushes the single sum through the transformation. Road two pushes u and v through separately, then adds the two outputs. The endpoints coincide exactly, and when you switch the panel to the bending map they visibly separate. Those two cold equations are not a definition to memorise. They are the grid staying a grid, written down.

A map that passes is called a linear transformation. Hold that phrase loosely for one more page, because we are about to discover something surprising about how little information one of them actually contains.

04★★ A matrix is where the basis lands

This is the rung the whole chapter turns on, and almost every course sprints past it. Most people meet matrices as a box of numbers with a bizarre multiplication rule to memorise, and nothing about that box ever means anything. We are going to close that gap now, and we are going to close it by making you do the computation before any rule exists.

Two vectors get special names because everything is built out of them. Write e₁ = (1, 0) and e₂ = (0, 1), the one-step-right arrow and the one-step-up arrow. Together they are the standard basis. Notice that every vector on the plane is already a mix of those two, and you have been writing it that way your whole life: (3, 2) = 3e₁ + 2e₂. Three steps right, two steps up.

Now the setup. I am going to tell you two facts about a linear transformation and nothing else. e₁ lands on (2, 1). e₂ lands on (−1, 1). That is all you get, and the panel will draw those two arrows and nothing else. Where does (3, 2) go? Commit to an answer, or sit in not knowing for a moment, because both are useful.

You can answer it, and the answer is forced. Here is the reasoning in one line. The input is 3e₁ + 2e₂, the transformation respects mixes, so the output must be 3 times where e₁ went plus 2 times where e₂ went. That is 3(2, 1) + 2(−1, 1), which is (6 − 2, 3 + 2) = (4, 5). Nothing was looked up, because the two arrows you were handed decided it entirely.

Sit with how strong that is. There are infinitely many points on the plane, and a linear transformation has to say where every one of them goes. Knowing two arrows fixes all of it. So a linear transformation of the plane contains exactly four numbers of information, and not one drop more.

Write those four numbers down and you have built a matrix. Put where e₁ went in the first column, and where e₂ went in the second:

A = [ 2  −1 ;  1   1 ], first column (2, 1), second column (−1, 1)

That is the keystone of this chapter, and it deserves a flat sentence. The columns of a matrix are the landing spots of the basis vectors. The box was never arbitrary. It is a lookup table with two entries, and reading a matrix means reading where the arrows went.

Now steer it. Drag either landing arrow in that panel and the entire grid deforms to follow, live, because those four numbers are the transformation and nothing else is stored anywhere. Park the first column on (1, 0) and the second on (0, 1) and the grid snaps back to rest, which is the identity matrix. Push the first column down below the axis and the whole plane flips over. You are not adjusting a picture of the matrix. You are adjusting the matrix.

Which gives us multiplication for free, as a description of something we already did.

To compute Ax, take the entries of x as weights and form that linear combination of A's columns. With x = (3, 2) that is 3(2, 1) + 2(−1, 1) = (4, 5). The same arithmetic you just did by hand. That is the meaning of matrix-vector multiplication, and it is the definition worth carrying.

You have probably also been taught a recipe: dot each row with the vector. Step the panel's second road and watch where it comes from. Row one is (2, −1), so row one gives 2(3) + (−1)(2) = 4. Row two is (1, 1), so row two gives 1(3) + 1(2) = 5. Same answer, (4, 5), and it is not a coincidence. The four products 2·3, −1·2, 1·3 and 1·2 appear in both roads. The columns road groups them by input, and the rows road groups them by output coordinate. It is one total, added up in two different orders.

Lead with the columns and the recipe stays a bookkeeping convenience. Lead with the recipe and the meaning is gone, which is exactly how most of us lost it the first time.

05Do one, then the other

If a matrix is a transformation, then two matrices are two transformations, and the obvious thing to do is run them one after the other. Rotate the plane a quarter turn, then flip it upside down. That pair of moves is itself a transformation, and it is linear, because doing two grid-preserving things in a row preserves the grid.

Before we write any rule, do it in the wrong order and see what happens.

Take the point (1, 0), which is a flag planted on the positive x-axis. Rotate a quarter turn counter-clockwise and it swings up to (0, 1). Now flip vertically, which negates the y-coordinate, and it drops to (0, −1). Reset and swap the order, then flip first: (1, 0) does not move at all, since it sits on the mirror line. Then rotate, and it swings up to (0, 1). Two different destinations from the same start, and you can see both on the panel at once.

So order matters in the world. That means it has to matter in the arithmetic that records the world, and the notorious fact that AB ≠ BA stops being arbitrary cruelty. It is the plainest observation in the chapter, written in symbols.

Now define the product, and define it by what we want it to do. Applying B then A to a vector x means computing A(Bx), which we already know how to do twice. We want a single matrix that does the whole job in one step, and we name that matrix AB:

(AB)x = A(Bx) — apply B first, then A

That is also why AB reads right to left, which trips up everybody once. The matrix nearest the vector touches it first.

Building AB is now a job you can do with only the keystone. The columns of any matrix are where the basis vectors land, so the columns of AB are where AB sends e₁ and e₂. Send each of B's columns through A and collect the answers. With A our old [2 −1; 1 1] and B the quarter-turn [0 −1; 1 0], B's columns are (0, 1) and (−1, 0). Push the first through A: 0(2,1) + 1(−1,1) = (−1, 1). Push the second: −1(2,1) + 0(−1,1) = (−2, −1). Stack those as columns and AB = [−1 −2; 1 −1].

Check that against the row-times-column drill and it agrees entry for entry, because it is the same sum regrouped. The rule was never invented. It is composition, worked out.

06Everywhere you can reach

Here is the question that turns this chapter from mechanics into insight. Every output of A is some mix of A's columns, with the input supplying the weights. So what is the set of all outputs? It is the set of all mixes of the columns, and that set has a name.

The span of a collection of vectors is every linear combination you can build from them. The span of a matrix's columns is called its column space, and it is precisely the set of vectors b for which Ax = b has a solution. If b is in there, some weighting reaches it. If it is not, no weighting ever will.

Sweep the two weight sliders and paint. With columns at (2, 1) and (−1, 1), the painted region fills the entire plane, and there is no point you cannot reach. Now drag the second column slowly around until it lies along the first, at (−4, −2), which is just −2 times the first column. Watch what happens to the paint.

The whole plane snaps down to a single line. Both sliders still move, and they still do something, but everything they produce lands on that one line through the origin. That is not a glitch. No weighted mix of two arrows lying on the same line can ever leave that line. You just watched a dimension die, and you caused it.

Two cases, two words. Columns pointing genuinely different ways are independent. Columns lying along one line are dependent or collinear, because one is just a scaled copy of the other and it contributes nothing new.

07Rank, counted honestly

The reachable set you just painted was sometimes a plane and sometimes a line. Those are different dimensions, and measuring that dimension is all that rank is.

You will usually be taught rank as "the number of pivots left after you row-reduce the matrix". Correct procedure, terrible definition. It tells you how to compute the number without ever telling you what the number is. So here is the meaning, with no algorithm in it. The rank of a matrix is the dimension of its column space, the count of genuinely independent directions its columns give you.

Three matrices sit side by side in that map, and you can read all three by eye. The first has columns (2, 1) and (−1, 1), pointing different ways, so it reaches the whole plane and its rank is 2. The second has columns (2, 1) and (−4, −2), which lie on one line, so it reaches only that line and its rank is 1. The second column added no new direction, so it did not raise the count. The third matrix is all zeros, it reaches only the origin, and its rank is 0.

A matrix with the most rank it could possibly have is full rank. One that falls short is rank-deficient. Both words do heavy work in a moment, and now they mean something you can see.

08★ Collapse, and an area that reads zero

Something strange happened in the span panel and we walked past it. When the rank dropped to 1, the transformation stopped merely moving points around the plane. It started destroying a dimension, taking the entire two-dimensional plane and squashing it onto a one-dimensional line.

Sit with that for a second, because the gut expects a transformation to shuffle space, not to flatten it. Every point still goes somewhere. The strange part is that a whole sheet of points all arrive at the same line.

We need a number that detects this, and there is one hiding in plain sight. Watch what a transformation does to area.

Start with the unit square, the little box with corners at (0,0), (1,0), (1,1) and (0,1), whose area is exactly 1. Apply the transformation and the square becomes a parallelogram spanned by the two columns. With columns (2, 1) and (−1, 1), that parallelogram has area 3, so this map triples the area of anything you feed it. That factor has a name: it is the determinant.

Now drag the second column toward the first, and predict what the area readout will do as they line up. Watch it thin. The parallelogram gets narrower and narrower, and the instant the second column lands anywhere on the line through the first, the area is exactly 0. Not small. Zero. The parallelogram has no width left, because the whole plane has just folded onto one line.

So the determinant reading zero is not a piece of trivia about a formula. It is the arithmetic signature of collapse. A matrix whose determinant is zero is called singular, and singular, rank-deficient and collapses-space are three names for the event you just triggered with your mouse.

One honesty note on scope. You have probably seen det = ad − bc for a 2×2 matrix, and it is the same number the panel prints. I am deliberately not deriving that formula here, because the meaning is what carries into Chapter 7 and the algebra is a detail you can look up. In three dimensions the determinant scales volume instead of area, and everything said here holds word for word.

09Undo, or don't

Now the question that all of this was built for: given a transformation, can you undo it? If I run every point of the plane through A and hand you the result, is there another transformation that puts every point back exactly where it started?

Most people think this is a question about whether a formula for the inverse exists. It is not. It is a question about information, and the cleanest way to see that is to try to undo a collapse.

Take the singular matrix with columns (2, 1) and (−4, −2), the collinear pair from the span panel. Feed it the input (2, 0) and the output is 2(2,1) + 0(−4,−2) = (4, 2). Now feed it a completely different input, (0, −1). The output is 0(2,1) + (−1)(−4,−2) = (4, 2). The same landing spot, from two genuinely different starting points.

So here is the demand. I hand you the output (4, 2) and ask which input produced it. There is no honest answer. It could have been (2, 0), or (0, −1), or any of the infinitely many points on the line joining them. The transformation did not merely scramble the information about where you started. It threw it away, and no cleverness recovers what is gone.

That is the real content of the word invertible. A transformation can be undone exactly when it never merged two inputs into one output, and it merges inputs only when it collapses space. Nothing lost, everything recoverable.

Which lets us do the most useful piece of tidying in the chapter. Drag the columns in that panel and watch five lamps flip at the same instant, all of them wired to the same event. The matrix is invertible. It has full rank. Its determinant is not zero. It is non-singular. Its columns are independent. Slide the columns onto one line and all five go dark together.

Those are not five facts to memorise. They are five ways of saying that space stayed full, and the five words are exactly the ones that get thrown around in interviews and papers as though they were separate ideas. They are one picture, and you now own the picture.

10When you cannot reach b

Everything so far assumed that Ax = b has an answer. In real data it almost never does, and this last section is where the chapter pays for itself.

Here is the setup, and it is the shape of every regression you will ever run. You have a year of a stock's returns, so 252 numbers, and you want to explain them with two knobs: a market exposure and a baseline. That is 252 equations and 2 unknowns, which is called an overdetermined system. Far more demands than freedom.

What can your two knobs actually produce? Every output of A is a mix of its two columns, so the reachable set is a flat two-dimensional sheet sitting inside a 252-dimensional space. The vector b of actual returns is a point somewhere in that huge space. The chance it happens to land exactly on your thin sheet is essentially zero.

So Ax = b has no solution, and that is normal rather than a failure. It says your two knobs cannot reproduce a year of reality exactly, which was never a reasonable hope. The panel draws this in three dimensions because three is what you can see. The honesty note from the start of the chapter still holds: at 252 the picture is unavailable and the arithmetic is identical.

If you cannot hit b, aim for the closest point you can hit. That means the point of the sheet nearest to b, and "nearest" turns out to have a completely concrete answer.

Drag b around above the sheet and watch the nearest point track it. The connecting segment is always perpendicular to the sheet, and that is geometry rather than convention. To reach a plane from a point outside it by the shortest path, you go straight at it. Any slanted path is the hypotenuse of a right triangle whose short leg is that perpendicular, so it is longer. With the sheet at z = 0 and b = (2, 3, 4), the nearest point is (2, 3, 0) at distance 4. Try the neighbour (2, 4, 0) and the distance is √17 ≈ 4.123, which is worse, as every other candidate will be.

That perpendicular drop is called an orthogonal projection, and choosing the reachable point closest to b is exactly what least squares does. This is the setup only, and Chapter 20 is where it becomes the normal equations and the fitted coefficients. I would rather flag that plainly than pretend we finished the job here.

But there is one thing we can finish, and it is the promise made in the chapter's first paragraph. Flip the panel's second switch and tilt the two columns nearly collinear.

Watch two readouts move in completely different ways. The fitted point on the sheet barely stirs, because the sheet is still nearly the same sheet. The coefficients, meanwhile, go berserk. At a column separation of ε = 0.1 the weights needed to reach that point are (−28, 30). Tighten to ε = 0.01 and they are (−298, 300). Two enormous numbers, cancelling each other almost perfectly, to produce a modest answer.

That is collinearity, and it is the failure the keystone promised. Regress a stock on two nearly identical inputs and your data matrix has two nearly identical columns. Its column space is nearly a line, its determinant is nearly zero, and it is nearly singular. Nearly non-invertible means the coefficients are nearly undefined, which is why they swing wildly, flip sign on a tiny change in the data, and mean nothing at all. Your fit still looks fine, and every individual coefficient is garbage. That is not a bug in the regression software. It is a rank-deficiency you can now diagnose in one sentence.

So here is the chapter as one picture. A vector is an ordered list, read as a point or as a bundle of data. A matrix is a function that moves all of space at once, and its columns are simply where the basis arrows landed. Multiplying is mixing those columns, and multiplying two matrices is doing one transformation after another. Everything after that was one question. Did space stay full, or did it get crushed flat? Full means independent columns, full rank, nonzero determinant, invertible, undo-able. Flat means the information is gone.

And that sets up the next question exactly. A transformation grabs most directions and turns them, which is what makes the picture confusing to hold. But look at the shear in the grid panel again and you will notice that some directions do not turn at all. They only get stretched. Those special directions are where a matrix becomes simple, and they are called eigenvectors. They are Chapter 7, and they are the language of PCA, of numerical stability, and of generating correlated random noise.