18Covariance, Correlation & the Causation Trap

Chapter 17 left you holding a machine for judging one number. You had an estimate, the cloud a boring world would have produced, and a way to ask how far out your dot landed. Now point that same machine at two things at once and a crack opens immediately. Since Chapter 16 you have been adding variances together as though that were a law of nature. It is not a law. It is the one special case where a third term quietly vanished, and nobody ever told you the term was standing there. That term has a name, covariance, and this whole chapter is the bill for it. Then we do one small piece of division and something far better than tidiness happens. Divide covariance by the two standard deviations and the formula turns into Chapter 6's cosine, character for character. Correlation is the angle between two arrows. Once you can see that, |ρ| ≤ 1 stops being a bound you memorize and zero correlation stops meaning unrelated. The ugliest phrase in quant finance becomes a picture you can draw, and three assets turn out to be forbidden from all hedging each other. We close on a real market correlation of about −0.7 whose cause the field still has not settled.

Look at what this page stands on, because every piece of it is already yours. Chapter 6 gave us the dot product, the norm, and the cosine formula that connects them. Chapter 7 gave us eigenvalues and the Cholesky decomposition, Chapter 11 gave us variance as the expected squared deviation, and Chapter 16 gave us Bessel's n−1. And Chapter 17 gave us the habit this chapter will need most, which is asking what a number looks like in a world where nothing at all is going on.

That is the seam. Every tool on the left of that picture handled one variable at a time, and the moment you hold two of them the machinery reaches for a quantity that has never been defined.

01The term that was always there

Here is a position. You buy one unit of an asset whose daily volatility is 1.6%, then you short one unit of the same asset, at the same moment, in the same size.

Before you compute anything, answer the question in your head. What is the variance of that position?

Everybody gets this right, because you can see the flat line. The two legs cancel exactly, every single day, forever. Whatever the asset does, you hold a rock worth precisely nothing, and a rock has zero variance.

Now run the rule you have been using since Chapter 16, where each leg on its own is genuinely risky with variance σ² apiece. Variance adds, so the position's variance is σ² + σ² = 2σ², which in this case is a daily volatility of √2 × 1.6% = 2.2627%.

The formula says the flat line is the riskiest thing in the room. So the formula is not a law, and the contradiction is not subtle, because it came out of your own tool in one line with no new notation anywhere.

So go back and expand it honestly. Write a = X − μₓ and b = Y − μᵧ for the two deviations. Chapter 11 defined variance as Var(Z) = E[(Z−μ_Z)²], and linearity of expectation gives μ_Z = μₓ + μᵧ when Z = X + Y. So the variance of the sum is E[(a+b)²], and Chapter 1 already told us how to multiply that out.

Var(X+Y) = E[a²] + E[b²] + 2E[ab] = Var(X) + Var(Y) + 2E[(X−μₓ)(Y−μᵧ)]

There is a third term, and nothing exotic was used to find it. It came out of ordinary algebra applied to a formula you already held, which means it has been sitting there the entire time. In Chapter 16 every example was built from independent draws, and for independent draws that term happens to be zero. That is the only reason variance appeared to add.

Notice how far the damage reaches, because SE = σ/√n is built on variance adding across n independent observations. Every standard error in Chapter 16 and every test in Chapter 17 has this term hiding inside it, set silently to zero. This chapter is the bill.

02The average signed rectangle

Give the thing a name. Covariance is Cov(X,Y) = E[(X−μₓ)(Y−μᵧ)], the average of the product of the two deviations. That sentence is symbol soup on a first reading, so let us not read it at all. Let us draw it instead.

Take five days of two assets, small enough to do entirely by hand. Asset A returns +1.4, +0.9, +0.4, −0.2, −0.5 percent, so its mean is +0.4. Asset B returns +0.7, +0.6, −0.3, +0.4, −0.4 percent, so its mean is +0.2. Plot the five points, then draw a horizontal line at B's mean and a vertical line at A's mean.

Those two lines cut the plane into four quadrants, and every point now has a rectangle attached to it. Its width is that day's deviation in A, and its height is that day's deviation in B. Its sign is positive if the point sits up-right or down-left of the crossing, and negative if it sits up-left or down-right. Covariance is the average of those signed areas.

The five signed products are +0.50, +0.20, 0.00, −0.12 and +0.54. Look at what each one is doing. Day 1 overshot in both directions at once, so it votes strongly positive. Day 4 missed in opposite directions, so it votes against. Day 3 sits exactly on A's mean line, so its rectangle has zero width and it does not vote at all.

They sum to +1.12, so divide by n−1 = 4, for the reason Chapter 16 gave, and the sample covariance is 0.28. That is the whole definition, enacted rather than stated, and it is worth holding onto the phrase: covariance is the average signed rectangle.

Three famous facts now fall out of that one picture, with no separate proofs.

Set Y = X and every rectangle becomes a square, because the width and the height are the same deviation. So Cov(X,X) = Var(X). Variance was self-covariance all along, and that fact will build the whole covariance matrix later on this page.

Swap the two arguments and nothing at all moves, because a rectangle turned on its side has the same area. So Cov(X,Y) = Cov(Y,X), which is to say that covariance is symmetric. Remember that, because at the end of this chapter that innocent little fact becomes a hard limit on what correlation can ever tell you.

Now make X and Y genuinely independent. Chapter 10 said independence means the joint distribution factorizes, so the double sum for E[XY] splits into (Σx·p(x))(Σy·p(y)) = E[X]E[Y]. Multiply out the definition and you get the computational form Cov(X,Y) = E[XY] − E[X]E[Y], which is therefore zero. On the picture the same thing looks like positive and negative rectangles arriving in balanced pairs and annihilating.

That closes the loop from the opening, and it is worth saying flatly. Independence was never why variance added. Independence caused a cancellation, and the cancellation is why variance added.

One honesty note about E[XY] − E[X]E[Y], because it is the form every library computes and the form that quietly ruins intuition. It is fast, and it is also the numerically fragile one, since it subtracts two nearly equal large numbers. That is Chapter 7's condition-number instinct applied to plain arithmetic. Use it to compute. Do not let it become the definition in your head, because the four quadrants are what you actually need.

03A number with no size

Now break the number on purpose. Keep the same five days, the same two assets and the same picture, and change only the units the returns are written in.

Quote them as decimals and the covariance is 0.000028. Quote the identical returns as percentages and it is 0.28, and quote them in basis points and it is 2,800.

Nothing about the two assets changed, and the scatter has the same shape in all three panels, so the relationship between them is exactly as strong. Yet the covariance moved by a factor of one hundred million.

The reason is one line of algebra. Scale X by a and Y by b, and the mean scales too, so each deviation picks up its own factor. Linearity of expectation pulls both constants out, and you get Cov(aX, bY) = ab·Cov(X,Y). Covariance is bilinear, which means it carries the product of the two units. The unit here is percent times percent, and nobody has an intuition for that, nor should anyone try to build one.

This is not a flaw to apologise for, because it is a specification rather than a defect. A covariance of 0.000028 is not big and it is not small, because it is not anything at all until you know both scales. If you have ever squinted at a covariance and wondered whether it counted as a strong relationship, that is why you got no feedback. The number was never wrong. It was meaningless.

And the same line of algebra points straight at the fix. Each variable already carries its own natural size, which is its standard deviation, and that scales the same way: SD(aX) = |a|·SD(X). So divide the covariance by both standard deviations and every scale factor cancels dead. What we want is a version of this number that does not move when we relabel the axes.

04Two columns, two arrows

Before doing that division, stop and look at what you are actually holding. This is the step that turns the rest of the chapter from formulas into pictures, and almost every course skips it.

You have two columns of numbers, which are 252 daily returns of asset A and 252 daily returns of asset B. Chapter 6 was explicit that a list of n numbers is a vector in ℝⁿ, and it used exactly this example to say so. So you are holding two arrows, living in a 252-dimensional space you will never draw and never need to.

Do one thing to them, and subtract each column's own mean from every one of its entries. That is centering, and the result is the deviation vector — the same deviations from the rectangle picture, now stacked into a column.

Centering is not cosmetic, and here is the trap that proves it. Take two stocks that both drifted upward all year while wiggling completely independently. Their raw columns are mostly positive, so the raw dot product is a big positive number, and it looks like strong agreement. It is nothing of the kind. What that number reports is that both columns were usually above zero, which is drift rather than co-movement. Centre them and the number collapses into noise.

Now take the plain Chapter 6 dot product of the two centered columns, which is aᶜ·bᶜ = Σ(aᵢ−ā)(bᵢ−b̄). Put that next to the sample covariance formula and read both lines slowly, because they are the same sum of the same products.

So we can write down two identifications, and neither of them is an analogy.

First, covariance is a dot product: Cov̂(a,b) = (aᶜ·bᶜ)/(n−1). The n−1 is Bessel's correction from Chapter 16, unchanged, and it is there for the same reason as always, since both means were estimated from the same data and one degree of freedom is already spent.

Second, standard deviation is a length: σ̂ₐ = ‖aᶜ‖/√(n−1). Chapter 6 defined the norm as ‖a‖ = √(a·a), and the sample standard deviation is √(Σ(aᵢ−ā)²/(n−1)). Those are the same square root with the constant moved.

Nothing was proved here. Two formulas were placed side by side and recognised as one formula, which is exactly why the step costs you nothing and buys the entire chapter.

05Correlation is a cosine

Now do the division we demanded, and give the result its name. Correlation is ρ = Cov(X,Y)/(σₓσᵧ), which is covariance measured in units of the two standard deviations. Every rescaling cancels, so the number no longer cares whether you quote returns in decimals, percent or basis points. That was the point.

But substitute the two identifications and something much better than tidiness happens. Covariance is (aᶜ·bᶜ)/(n−1), and the two standard deviations are ‖aᶜ‖/√(n−1) and ‖bᶜ‖/√(n−1), so their product carries a 1/(n−1) as well. The n−1 cancels top and bottom, and what is left standing is this.

ρ = (aᶜ·bᶜ) / (‖aᶜ‖ · ‖bᶜ‖)

That is Chapter 6's cosine formula, and not something that merely resembles it. It is the same formula, character for character, unchanged. Correlation is the cosine of the angle between the two centered data vectors.

Everything you were ever told to memorize about correlation now follows without further work. A cosine cannot leave [−1, 1], so |ρ| ≤ 1 is not a bound that needs proving but a plain fact about angles. Most courses reach it with a discriminant trick on E[(tX+Y)²] ≥ 0, which compels agreement and produces no understanding at all.

ρ = 1 means the two arrows point the same way, so one column is literally a positive rescaling of the other. That is perfect linear agreement, which is a much narrower claim than "they always move together". ρ = −1 means antiparallel.

And ρ = 0.7 finally has an honest reading. Project one deviation vector onto the other, and the shadow keeps 70% of the original length, while the rest sticks out perpendicular and is simply not seen. Our five days scored r = 0.6993, which is an angle of 45.63°. So a correlation of 0.7, a number that sounds like a strong verdict, is two arrows sitting about forty-six degrees apart.

Hold on to one more thing, because it is where the chapter goes next. If correlations are angles between arrows, then a set of pairwise correlations is a set of pairwise angles. Angles among several arrows are not independent dials you can turn freely.

Test that on yourself before reading on. Here are three assets, and every pair moves against both of the others, so each one hedges the other two. Set all three correlations to −0.9. Possible?

Almost everyone says yes, because nothing in the definition of a pair mentions a third asset. Now take the algebra away and ask it in English. If A moves exactly opposite to B, and B moves exactly opposite to C, what is A doing relative to C?

It is moving with it, which means A and C are the same trade. The three dials were fighting each other the whole time, and you did not need a single formula to see it.

The geometry says the same thing with a number attached. Three arrows from one origin can be pushed apart until each pair sits at 120°, and after that there is nowhere left to go. cos 120° = −0.5. That is the floor for three mutually equal correlations, and it is a fact about space rather than a fact about finance.

Watch what happens either side of it. At ρ = −0.5 the three arrows form a flat tripod and the equal-weight portfolio's variance sits at exactly zero. Nudge one dial to −0.51 and the picture cannot be drawn at all, while the variance readout goes negative. A portfolio with less risk than nothing.

06The one line it can see

Push on ρ = 0, because that is where the technical word and the English word finally come apart.

ρ = 0 means cos θ = 0, which means the two centered vectors are perpendicular. Perpendicular is a statement about one line's worth of information. It says B's deviation vector has no component lying along A's, so A's shadow on B is empty. It says nothing whatsoever about whether B is determined by A.

Here is the smallest example that forces the point. Let X be equally likely to be −2, −1, +1 or +2, and let Y = X². Y is a deterministic function of X, so tell me X and I tell you Y exactly, which makes these two as dependent as two variables can get.

Now compute, with four numbers and no theory. E[X] = 0, and E[XY] = E[X³] = (−8 − 1 + 1 + 8)/4 = 0. So Cov(X,Y) = E[XY] − E[X]E[Y] = 0, and ρ = 0.

Draw the cloud and the reason is on the page. It is a U, and a U leans left exactly as much as it leans right. Every positive signed rectangle is met by a negative one of the same size, and they annihilate in pairs.

Correlation did not fail here. It reported honestly that there is no linear component, and there genuinely is not one. The instrument is pointing at the place where the information is not.

So the logic runs one way only, and it is worth writing as an arrow. Independence ⇒ ρ = 0, which we proved with the rectangles. ρ = 0 ⇏ independence, which the U just demonstrated.

One exception deserves naming out loud, because it is why the confusion is so sticky. If X and Y are jointly Normal, then ρ = 0 really does imply independence. Most of your mental library of scatter pictures is Gaussian, so the implication feels true in general. It is not. And the place that exception fails hardest is the joint tails, which is exactly why Chapter 32 needs copulas.

The blind spot has a milder form too, and it is far more common than a perfect parabola.

A single number cannot carry the shape of a cloud. A clean line, a widening fan, and one outlier dragging everything can all report the same r. Always look at the scatter. The number is a summary of one specific thing, and it was never a substitute for the picture.

07Somewhere to put the pairs

Scale up, and the container builds itself. Two assets have one covariance, and three assets have three pairs. Five hundred assets have 500·499/2 = 124,750 pairs, plus 500 variances. A quant does not hold two assets.

You need somewhere to put those numbers, and the shape of the container is forced rather than chosen. Index the rows and columns by asset, and define Σᵢⱼ = Cov(Rᵢ,Rⱼ). That is the covariance matrix.

A notation warning first, said once, plainly. That capital sigma is not the summation sign you have used since Chapter 2, and the two of them will appear in the same formulas. Var(Rₚ) = ΣᵢΣⱼ wᵢwⱼ Σᵢⱼ is a genuinely hostile line of text. Read it by position rather than by glyph, because it is an accident of history and not a concept.

Two features of the container are already ours. The diagonal holds Σᵢᵢ = Cov(Rᵢ,Rᵢ) = Var(Rᵢ), because variance is self-covariance. So the variances are not stored somewhere separate, because they were always the diagonal of this object. And the matrix is symmetric, because covariance is symmetric, so the upper and lower triangles mirror each other.

Divide every entry by the two relevant standard deviations and you get the correlation matrix. Same object, rescaled, with ones down the diagonal, and every off-diagonal entry a cosine.

Computing it is one matrix product rather than a loop over pairs, so stack the centered data columns into a matrix Xᶜ with n rows for days and k columns for assets. Then XᶜᵀXᶜ is the matrix of all pairwise dot products, so Σ̂ = (1/(n−1))·XᶜᵀXᶜ.

Check that on a toy small enough to multiply by hand, with three days, two assets, A at +1.0, −0.2, +0.4 and B at +0.5, +0.1, −0.3 percent. Centre them and the deviations are (+0.6, −0.6, 0.0) and (+0.4, 0.0, −0.4). The off-diagonal entry of the product is 0.6×0.4 + (−0.6)×0 + 0×(−0.4) = 0.24, and dividing by n−1 = 2 gives 0.12. That is the dot product from the centering step, landing in the cell with its name on it.

The full toy is Σ̂ = [0.36, 0.12 ; 0.12, 0.16], with standard deviations 0.6 and 0.4, so the correlation is 0.12/(0.6×0.4) = 0.5. So the covariance matrix is the Gram matrix of the asset arrows, which is the complete table of their lengths and the angles between them.

08What the matrix is for

Now give the container a job. A portfolio is a set of weights, so its return is Rₚ = w₁R₁ + w₂R₂ + … + w_k R_k. Its variance is the same expansion we did at the top of this chapter, run over every pair instead of one.

Do the two-asset case completely, because the general case adds nothing but indices. Expanding the square and using bilinearity to pull the weights out of each covariance gives this.

Var(w₁R₁ + w₂R₂) = w₁²Var(R₁) + w₂²Var(R₂) + 2w₁w₂Cov(R₁,R₂)

Notice the shape of it before moving on. The weights arrive squared on the variances and multiplied on the cross term. Expectations were not like this, since E[Rₚ] = Σwᵢμᵢ is linear in the weights and risk plainly is not. Return and risk do not scale the same way, and that asymmetry is roughly the whole of portfolio theory.

For k assets the identical expansion produces one term for every ordered pair, which is Var(Rₚ) = ΣᵢΣⱼ wᵢwⱼ Cov(Rᵢ,Rⱼ). That double sum is exactly what the matrix sandwich wᵀΣw computes. Multiply Σw and each row of Σ gets weighted and summed, then wᵀ weights and sums those results.

Write the two-asset case both ways on adjacent lines and the four terms match one for one. That verification matters more than it looks, because Chapter 30 differentiates this expression to build every optimizer in the book, and you cannot differentiate an incantation with any confidence.

Run it on our toy matrix with weights (0.6, 0.4). The three pieces are 0.36×0.36 = 0.1296, 0.16×0.16 = 0.0256, and 2×0.6×0.4×0.12 = 0.0576. They total 0.2128, so the portfolio volatility is 0.4613%. Set the covariance to zero and the same weights give 0.1552 and 0.3940%. On two assets, the cross term is already 27% of the variance.

Now count what sits inside that sum for a real book, because the count is the point. For 100 assets there are 100 variance terms on the diagonal and 100 × 99 = 9,900 covariance terms off it, which is 4,950 distinct pairs. Ninety-nine percent of the terms in your risk number are covariances.

Portfolio risk is not mostly about how risky your assets are. It is almost entirely about how they move together. That is why every risk system in the world is built around this matrix rather than around a list of volatilities.

09Why it cannot be any matrix you like

Read the last result backwards and collect the debt from the three-dial figure. wᵀΣw is the variance of a real portfolio, one you could actually hold with those weights. A variance is an average of squares. So it cannot be negative, ever, for any weights at all.

That sentence, wᵀΣw ≥ 0 for every w, is the definition of a positive-semidefinite matrix. Read the order carefully, because textbooks run it the other way and that is why the term feels like bureaucracy. PSD is not a technical condition someone imposes on covariance matrices, but the shadow cast on the matrix by the fact that variances are never negative.

Every consequence is now something you already own from Chapter 7. A symmetric matrix's eigenvectors are the directions it only stretches, so wᵀΣw along an eigenvector is just its eigenvalue times a squared length. Therefore every eigenvalue is ≥ 0. And here is the reading nobody hands you: each eigenvector is a portfolio, and its eigenvalue is that portfolio's variance.

The level set wᵀΣw = 1 is an ellipsoid whose axes are those eigenvectors. It is long where risk is small and short where risk is large, which makes it the shape of the portfolio's risk, drawn. Drag one eigenvalue down to zero and the ellipse flattens into a line. Push it below zero and the picture tears open into a hyperbola, at which point "the set of portfolios with risk 1" stops being a bounded thing at all.

Two consequences are worth cashing immediately. The first is the floor we felt in words earlier, now proved in one line. Three variables standardized to variance 1, all pairwise correlations ρ, give Var(X+Y+Z) = 3 + 6ρ ≥ 0, so ρ ≥ −½. Run the same line for k variables and 1 + (k−1)ρ ≥ 0, so the floor is ρ ≥ −1/(k−1). The more assets you try to make mutually hedging, the less mutual hedging geometry will allow.

The second is the difference between semidefinite and definite, which is never explained and matters constantly. An eigenvalue of exactly zero means some non-zero portfolio has variance exactly zero, so there is a combination of your assets that never moves. That is a genuinely redundant asset. It is also Chapter 6's singular matrix wearing a financial hat.

And Chapter 7's Cholesky decomposition needs positive-definiteness to exist, which quietly makes it two useful things at once. Run it as a test and a refusal tells you your matrix is not a covariance matrix. Run it forwards and it turns independent random numbers into correlated ones, which is how every Monte Carlo in Part 4 generates a market.

Try it on the three-asset matrix with every correlation at −0.6. Its eigenvalues are 1 + 2ρ = −0.2 and 1 − ρ = 1.6 twice, so Cholesky refuses to run. Being told a matrix is invalid is forgettable. Watching it promise a portfolio with negative risk is not.

10How much can you diversify away?

Cash the matrix out on the question every investor has already asked. Take k assets, each with volatility σ, every pair correlated ρ, held in equal weights of 1/k.

Before the algebra, commit to an answer. Every stock has 20% annual volatility and every pair is correlated 0.3. What is your portfolio volatility with 10 names, then 100, then all 5,000?

The answers are 12.17%, 11.08% and 10.96%. Almost everyone predicts something falling toward zero, or at least toward a few percent, because Chapter 16's SE = σ/√n showed exactly that shape. The whole benefit was gone by about thirty names, where the volatility is already 11.37%.

Now look at why, because it is visible in the formula itself. The double sum has k diagonal terms and k(k−1) off-diagonal ones, all carrying weight 1/k². So Var = (1/k²)[kσ² + k(k−1)ρσ²] = σ²[1/k + (1−1/k)ρ].

Read the two pieces separately, because they behave nothing alike. The first, σ²/k, is the part that dies as you add names. That is idiosyncratic risk, the wiggle each asset does alone, and it is the σ/√n world you came from. The second term marches straight to ρσ² and stops. It does not contain k anywhere.

So push k to infinity and the variance does not go to zero. It goes to ρσ², which means the volatility floor is σ√ρ. With σ = 20% and ρ = 0.3 that is 10.95%. Owning every stock in the market, equally weighted, removes 45.2% of your volatility and not one basis point more.

Half the risk in a diversified equity book is not diversifiable at all. It is not the assets' risk but the covariance they share, and there is nothing left to average that against.

The only thing that actually removes shared risk is a negative cross term, and we just proved how little of that is available. With k assets the average pairwise correlation cannot go below −1/(k−1), which for a hundred names is −0.0101. A large, mutually hedging book is not difficult to build. It is geometrically forbidden.

11The Σ you actually have

Everything so far assumed you have Σ. You do not. You have Σ̂, an estimate, and the gap between the two is where real money is lost. Keep the hat visible from here on, because the population object and the sample estimate share one symbol in most of the literature and switch meaning mid-sentence.

Start by counting the parameters. A k-asset covariance matrix holds k(k+1)/2 distinct numbers, which for 500 stocks is 125,250. Estimate that from one year of daily data and you have 252 rows to do it with. There is not enough data in the file.

The failure is not gentle degradation. It is structural, and Chapter 6 already gave us the tool to see it. Xᶜ has at most 252 rows, and centering spends one, so its rank is at most 251. Therefore Σ̂ = XᶜᵀXᶜ/(n−1) is a 500 × 500 matrix of rank at most 251, and at least 249 of its eigenvalues are exactly zero.

Read that through the dictionary we built, in which an eigenvalue of zero is a portfolio with estimated variance zero. So this matrix is asserting the existence of 249 riskless portfolios built out of ordinary stocks, and they are not riskless. They are artefacts of having fewer days than names.

An optimizer that minimizes wᵀΣ̂w will walk straight into them with enthusiasm, because to a machine that believes its inputs they look like free money. Nothing errors and nothing warns, and you get a beautiful low-risk portfolio whose reported risk is fiction. That is the honest reason Chapter 22's shrinkage and Chapter 24's PCA exist.

Two more field failures belong here. Correlation matrices assembled by hand, or stitched from different sources, routinely come out non-PSD — this pair from five years of data, that one from a trader's judgement, a third interpolated. Cholesky refusing to run is the alarm. The standard repair is to eigendecompose, clip the negative eigenvalues to zero, and rebuild, which is Chapter 7's spectral view earning its keep.

And now the deepest one, which is that Σ is not a constant of nature. Correlations rise toward 1 in crashes, when you need them not to. The diversification floor we just computed uses a ρ measured in calm markets, and the ρ you actually need is the one that will hold on the day everything falls. That number is higher, exactly when you were relying on it being low.

One more thing on this page is an estimate, and it is the one everybody quotes as though it were a fact, which is r̂ itself. Somebody says gold and equities are correlated −0.2. Over what window?

A correlation computed from n observations is a statistic, so Chapter 16 says it has a sampling distribution and Chapter 17 says the way to judge it is against the cloud a boring world would produce. Before you look, guess. Take two genuinely unrelated series over 60 days, and ask how big r̂ can get by pure luck 5% of the time.

Most people guess 0.05, maybe 0.10. The true answer is 0.254, which is squarely inside the range where a practitioner would say there is a relationship there. The spread of r̂ under ρ = 0 is about 1/√(n−1), which at n = 60 is 0.130.

The width shrinks like 1/√n, and that is the whole practical lesson. The 5% cut is 0.361 at 30 days, 0.254 at 60, 0.124 at 250 and 0.039 over ten years. The same measured 0.25 is weather in one column and a real finding in another. So a correlation quoted without its sample size is not a fact.

Now notice the door this opens. If one honest correlation can be off by 0.13 through luck alone, what happens when you screen a thousand candidate pairs looking for the biggest one? That question is Chapter 19.

12The causation trap

Everyone can recite the slogan and almost nobody can defend it. So here is the reason, and it is not that statisticians are cautious.

You proved it yourself in the rectangle picture, where Cov(X,Y) = Cov(Y,X). Correlation is symmetric in its two arguments, and causation is not, because "the rain wet the street" and "the street wet the rain" are different claims. A symmetric quantity cannot encode an asymmetric relation.

Read what that rules out. It is not that correlation is weak evidence of direction. Direction is structurally not among the things it can carry, no matter how large it is or how much data produced it.

So any observed r̂ leaves four live explanations, and the discipline is naming which one you are claiming.

It could be chance, and we just watched luck manufacture 0.25 on sixty days. It could be reverse causation, where Y drove X. It could be a confounder, a third variable Z driving both, which is easy to see geometrically now: two arrows both pulled close to a third are automatically close to each other. Summer heat drives both ice-cream sales and drownings, and Chapter 15's Simpson's paradox was this same effect strong enough to flip a sign. Or it could be genuine causation.

There is a fifth trap that belongs to markets specifically, and it is a false positive of a different kind.

Correlate prices instead of returns and you are correlating two random walks. Chapter 14 built those as running sums of independent steps, and their sample correlation does not settle down to zero at all. It wanders across the whole range, so two entirely unrelated trending series routinely score 0.9. Correlate returns, not levels. The proper treatment of the genuine exception, where two prices really are tied together, is cointegration in Chapter 35.

Now the constructive half, because "correlation is not causation" gets over-learned into "correlation is useless", and that is badly wrong. Correlation is exactly what a portfolio is built on, and you need no causal story at all to hedge. This whole chapter was a demonstration of how much work a bare association can do.

What licenses a causal claim is not a bigger r̂. It is a different kind of data, which is an intervention. When you assign the treatment yourself at random, you have severed every arrow into it, because no confounder can be feeding the assignment when a coin flip did. The difference that remains has one explanation left.

You already built that machine in Chapter 17, and the A/B test is not merely a hypothesis test. Randomizing is the operation that converts an association into a cause.

13The leverage effect, unresolved

Close on a live case from your own field, where every piece of this chapter is in play at once and the honest answer is not a clean one.

Take the daily returns of an equity index and the daily changes in its volatility, and correlate them. The number comes back near −0.7. It is not a marginal effect and it is not a small-sample artefact, since it clears the test we just built with enormous room to spare over decades of data. Volatility goes up when the market goes down, hard and reliably.

This is the leverage effect, and it is the single most important covariance in derivatives. So run the four-explanation checklist on it, and watch chance die on the first line. What remains is two mechanisms that predict the same correlation with the arrow pointing in opposite directions.

Black's leverage story runs price → volatility. A firm's equity is the residual claim sitting on top of its debt. When equity falls, the debt-to-equity ratio mechanically rises, so the remaining equity is a more levered claim on the same assets, and a more levered claim is more volatile.

The volatility-feedback story runs volatility → price, because when expected volatility rises investors demand a higher return to hold the asset. The only way to deliver a higher future return on unchanged cash flows is for the price to fall right now.

Same sign, same data, opposite causal arrows. The evidence is genuinely mixed, and the measured effect for indices is generally too large to be explained by leverage ratios alone. We do not fully know.

What we do know is the sign, and the sign is what the desk trades. A market that gets more violent as it falls is a market where a crash is worth more than a symmetric model says. That single asymmetry is what bends the flat Black–Scholes volatility surface into the downward skew you will invert in Chapter 29.

Notice what separated the actionable part from the open part. Pricing a skew needs the sign and its size, while forecasting whether the effect will hold through a regime change would need the mechanism. Knowing which half of a causal question you actually need answered before you can act is a skill nobody teaches.

Look back at what you are holding, because it is one picture. Two centered columns are two arrows. The angle between them is the correlation, which is why it cannot exceed 1 and why a right angle says nothing about dependence. The table of all their lengths and angles is the covariance matrix, so wᵀΣw is a squared length and cannot be negative. Three arrows cannot spread past 120°, which is the −½ floor. And the angle between two arrows is the same whichever one you name first, which is why none of this can ever tell you what caused what.

The chapter opened with a cross term nobody had noticed. It closes with that same cross term setting the price of every index put in the world, still carrying a mechanism the field has not settled. Chapter 19 picks up the loose end. If a single honest correlation can wander by 0.13 through luck alone, then screening a thousand pairs for the largest one is not research. It is manufacturing, and something has to pay for the extra looks.