16How Variables Move Together

Chapter 15 handed you something and refused to name it. Here is the name, and the whole chapter in one line. That leftover piece is covariance: the average, over the cloud, of a single signed product — how far X sits from its mean, times how far Y sits from its. Now the story behind it. You expanded the spread of a sum and out fell a third piece, E[(X−μ_x)(Y−μ_y)], that would not go away. You even saw what it was for. Two worlds, one with independent dice and one with Y = 7 − X welded to the first, had the exact same mean for their sum. Yet one sum sprawled and the other never moved. The mean was blind to the difference between them. The cross term was the thing that could see it. The signed product carries the direction in its sign. When X and Y rise together the product is positive. When one rises as the other falls it is negative. Its size, though, is drowned in whatever units you happened to measure in. So we divide the units out and get correlation, a clean number pinned between −1 and +1. One honest warning rides the whole way. Covariance and correlation see only the straight-line thread between two variables, and a curve can slip past both of them completely.

01The number we don't have yet

You already have a mean and a spread for every variable, one number per axis, and it is natural to think those two tell the whole story. They don't. Let me convince you there is a gap here, because it does not feel like one. The cleanest way to feel it is to look at data where the per-axis summaries are identical and the relationships are opposite. Same mean, same spread, and yet the two variables are doing completely different things.

Picture three scatterplots. In the first, taller people tend to weigh more, so the cloud leans up to the right. In the second, more hours gaming go with lower grades, so the cloud leans down. In the third, two unrelated measurements make a round blob that leans nowhere. Now here is the trap. I can arrange all three so their x-histogram is the same and their y-histogram is the same. Same mean on each axis. Same spread on each axis. Three different worlds, and the per-axis numbers cannot tell you which one you are looking at.

Watch the two histograms sit still while the cloud swings from up-tilt to blob to down-tilt. The marginals — Chapter 14's word for the shadow each axis casts — cannot tell the three apart. So whatever distinguishes the three is not in either shadow. It lives in the pairing: which y got attached to which x. A marginal throws that pairing away. The pairing is exactly what we now need one number to capture.

So the goal is honest and small. We want a single number that scores how two variables move together. positive when they rise as one. negative when they trade off. Near zero when the cloud leans nowhere. That is the whole assignment. The rest of this chapter is us earning that number instead of being handed it.

02One point, one signed rectangle

Forget the whole cloud for a moment and zoom in on a single point. Draw two lines through the picture. A vertical one at μ_x, the mean of X. A horizontal one at μ_y, the mean of Y. Call the pair of lines the mean-cross. They split the plane into four quadrants, and the quadrant a point lands in says something precise about that point.

A point in the top-right has x above its mean and y above its mean. Both deviations — the signed distances (x − μ_x) and (y − μ_y) — come out positive. A point in the bottom-left has both of them negative. Those two are the "together" quadrants: the point sits on the same side of both means. The other two are the "trade-off" quadrants, where one deviation is positive and the other is negative.

Now the question the whole definition turns on. We want an operation that reads "same side of both means" as a positive number and "opposite sides" as a negative one. Same sign should give plus. Opposite sign should give minus. One ordinary operation already does exactly that. Multiplication. Two positives multiply to a positive, two negatives also to a positive, and a mixed pair to a negative. So the product (x − μ_x)(y − μ_y) is not a guess someone made. It is forced by what we asked for.

Drag the point around and watch the signed rectangle. Its two sides are the deviations, and its area is their product. In the two together-quadrants the rectangle fills green, a positive contribution. In the two trade-off quadrants it fills red, a negative one. Slide the point back toward the cross and the rectangle shrinks to nothing, because a point sitting near either mean has almost no deviation to contribute. The sign of the co-movement is now something you can see, not a rule to memorize.

03Averaging the rectangles

One point gives one signed rectangle. A whole cloud gives a whole pile of them, some green, some red. The move now is the one that turned a squared deviation into variance last chapter: take the average over the distribution. That average is the number we have been chasing.

Here it is. Cov(X,Y) = E[(X − μ_x)(Y − μ_y)], the covariance of X and Y — the expected value of the signed product of their deviations. It says exactly what the picture showed. Average the green-and-red rectangles over the cloud, letting each point contribute in proportion to its probability. A point 2 above the mean in x and 3 above the mean in y contributes 2 × 3 = +6. Swap it to 3 below in y and it contributes 2 × (−3) = −6. If the cloud tilts up to the right, most of its mass lands in the two green quadrants, the positives win, and covariance comes out positive. Tilt the cloud down and the red quadrants fill, so covariance goes negative. A round blob spreads evenly across all four, the rectangles cancel, and covariance sits near zero.

The slider summed the rectangles for you. Now do it once by hand, so the number stops being magic. Here are five points with their mean-cross drawn in. For each point, read off the two deviations, multiply them, and note the sign. Add the five products, divide by five, and you have built Cov yourself.

Two things are worth doing with your own hands here, and they land on one verdict: only the shape of the cloud matters, never where it sits. First, before you drag the tilt, predict the sign. The cloud leans up, so which colour wins? Then slide the tilt down through flat, and watch the running sum pass through exactly zero at the level cloud and go negative past it. The tilt is the sign — you are watching one become the other. Second, grab the whole cloud and slide it sideways, keeping its shape. Covariance does not move. That is not an accident. Every rectangle is measured from the mean-cross, and the cross rides along with the cloud, so no deviation changes. So only two things drive covariance. The tilt sets its sign. The tightness sets its size. Where the cloud sits is irrelevant.

Let me lock the "tilt is the sign" claim with a cleaner control than a hand-dragged cloud. Take a perfectly axis-aligned ellipse of points, wide one way and narrow the other, leaning nowhere. Its covariance is zero, because the ellipse is symmetric across both lines of the cross. Now rotate it.

The lean of the cloud and the covariance number are the same thing wearing two costumes. Watch it happen. The moment the ellipse turns, two things arrive at the same instant: the cloud visibly leans, and the covariance readout lifts off zero. Rotate the other way and both flip negative together. They are not two facts you have to connect. They are one fact seen twice. That is the intuition to carry out of this section: covariance is the cloud's tilt, written as a number.

04Covariance is variance, reborn

Here is the quiet gift that makes covariance feel earned instead of invented. We just built a machine that takes two variables and averages the product of their deviations. Watch what happens when you feed that machine the same variable twice. Put Y = X into the definition and turn the crank.

Here is the punchline before the algebra: variance was always a covariance. The second factor becomes the first, so Cov(X,X) = E[(X − μ_x)(X − μ_x)] = E[(X − μ_x)²]. That is Var(X), exactly the definition from last chapter. Variance is a variable's co-movement with itself. And covariance is the honest generalization: the identical machine, with the second slot freed to hold a different variable. Do not undervalue this as a relabel. It runs the other way round. Variance is the special case, the diagonal where the two variables coincide, and covariance is the general object that contains it.

Now let's cash this on the two worlds Chapter 15 left you holding. World A is two independent dice. World B is one die X with Y = 7 − X welded to it. Both worlds have identical shadows, so the mean of the sum is 7 in world A and 7 in world B. The mean genuinely cannot distinguish them. So ask covariance instead.

In the independent world the cloud is a square blob, and its covariance reads 0. In the welded world every point lies on the falling line y = 7 − x, a perfect down-tilt, and its covariance reads −35/12 ≈ −2.917 — the same 35/12 that was one die's variance, now carrying a minus sign. That number is the residue Chapter 15 produced and would not name. It was never mysterious. It is the covariance, and you can read its sign straight off the tilt of the cloud. One caution about its size. That −2.917 is measured in pips squared, the units of the dice. When we divide those units out shortly, the welded world will turn out to sit at exactly −1, the most negative a correlation can be.

05The shortcut, and the one-way arrow

The definition is the meaning, but there is a second face you will actually compute with. It is the twin of last chapter's Var(X) = E[X²] − μ², and it comes out of the same move. Expand the product, then let linearity walk through it.

Expand (X − μ_x)(Y − μ_y) into XY − μ_yX − μ_xY + μ_xμ_y and take the expectation of each piece. Here μ_x and μ_y are numbers, not random, so they slide out of the bracket just as μ did last chapter. The middle terms collapse and you are left with Cov(X,Y) = E[XY] − E[X]E[Y]. Check it on the welded dice: E[XY] = 9.333 and E[X]E[Y] = 3.5 × 3.5 = 12.25, so Cov = 9.333 − 12.25 = −2.917, the −35/12 from before. Now bring back one fact from Chapter 15. When X and Y are independent, E[XY] = E[X]E[Y]. Subtract, and the two pieces are equal, so Cov = 0. Independence forces the covariance to vanish.

Read that arrow carefully, because your gut is about to reverse it and it must not. What we proved is independent ⇒ Cov = 0. That arrow points one way only. The mirror claim, Cov = 0 ⇒ independent, is a different statement. And it is false. I am flagging it here, before you have seen why, because the false version installs itself the instant you look away. The next section is built to demolish it in your hands.

06What covariance is blind to

Here is the reversal the whole chapter exists to break, and it is the deepest counterintuition in the subject. Take Y = X², with X spread symmetrically around zero. Y is completely determined by X. Tell me X and I tell you Y exactly, with no slack at all. There is no stronger dependence than that. Before you read on, look at the U-shaped scatter and answer one question. Is the correlation positive, negative, or zero?

Almost everyone says positive, or at least "strong". The answer is exactly zero. Now watch why. Overlay the mean-cross. The scatter is a U, mirror-symmetric across the vertical line. Take the point at x = +1 and its twin at x = −1. Squaring wipes out the sign, so both sit at the same height. They have opposite x-deviations but the same y-deviation. Put numbers on it: let X be −1, 0 or +1 with equal chance, so X averages 0 and Y averages 2/3. The right-hand twin contributes (+1) × (1 − 2/3) = +1/3, the left-hand twin (−1) × (1 − 2/3) = −1/3. Their signed rectangles are equal and opposite — one green, one red — and they cancel. Every point cancels with its mirror. The rectangles sum to zero.

So covariance did not measure "no relationship". It measured "no straight-line trend". And a U genuinely has none — it rises on the right exactly as much as it rises on the left, with no net tilt. That is the load-bearing limitation of this whole chapter, and now it is a fact you watched happen rather than a caution you were handed. Covariance is blind to any co-movement that is not a straight line.

And the U is not special. Any shape with a symmetric, tilt-free structure fools covariance in exactly the same way.

Step through a ring, an X, a symmetric arch. Each one is strong dependence. Each one reads a covariance of zero, because for every green rectangle there is a mirror red one. One tool, one blind spot. Covariance and correlation see the linear thread of a relationship and nothing else. Keep that flag planted. It is the reason the last section of this chapter has to exist.

07The units lie — so divide them out

Covariance has the right sign, but its size lies, and it is worth breaking the number yourself to feel it. Measure a group of people's height in feet and compute the covariance of height with weight. Now measure the exact same people in inches. Not one person changed height. But every height number, and therefore every x-deviation, just got multiplied by 12.

Toggle the units and watch the cloud sit perfectly still while the covariance readout jumps twelvefold. Say it read 40 in foot-pounds. The same people in inch-pounds give 40 × 12 = 480. A number that is supposed to say "how strongly" just changed by a factor of 12 for a pure relabel. That settles it. Covariance carries the units of X times the units of Y, so its raw magnitude means nothing on its own. You cannot say a covariance of 40 is "big" without asking, big in what units.

The fix is the oldest trick in physics: cancel the units by dividing. Covariance is measured in X-units times Y-units. The standard deviation σ_x is in X-units, σ_y in Y-units. So divide covariance by σ_xσ_y and the units cancel exactly. That pure number is the correlation, written ρ = Cov(X,Y) / (σ_xσ_y). Put some numbers in — invented ones, chosen to keep the arithmetic clean: covariance 40, height spread 0.5 feet, weight spread 100 pounds, so ρ = 40 / (0.5 × 100) = 0.8.

Press the feet-to-inches toggle again with correlation on screen. Covariance balloons; ρ sits pinned. Here is why. Rescale X by 12 and covariance scales by 12, but σ_x also scales by 12, so the two twelves cancel in the ratio. In inches the spread is 6 and the covariance is 480, so 480 / (6 × 100) = 0.8, the very number the feet data gave. Correlation is scale-invariant. It does not care what units you chose, which is exactly the property covariance was missing.

There is a second way to read ρ that makes it click, and it reaches back to Chapter 13. Dividing each deviation by its own σ is exactly standardizing: turning X into Z_x = (X − μ_x)/σ_x, the affine map that lands every variable at mean 0 and spread 1. Do that to both variables, and correlation is just the covariance of the standardized variables: ρ = E[Z_xZ_y].

Watch both axes get rescaled and recentered until the cloud sits on unit-free footing, the same footing for every problem. On those axes there are no feet, no inches and no pounds — just the bare shape of the relationship. Correlation is the covariance you read off after you have put both variables on that common footing. The mysterious denominator was never a fudge factor. It is the price of standardizing.

Now that ρ is a stable number, it is worth training your eye to read it off a picture. The strength of a correlation is the tightness of the cloud around its line. The sign of a correlation is the direction of the lean.

You just learned to eyeball ρ from the whole cloud. But one stray point can forge that number. Start from a round blob correlating at nearly zero, then drag a single far-off point and watch ρ climb on its own. Nothing else in the cloud moved.

Drag ρ from −1 through 0 to +1 and watch the cloud change under it. At the bottom end it is a tight falling line. In the middle it is a shapeless blob. At the top end it is a tight rising line. Then flip it around: a fresh scatter appears, you guess its correlation, and the true value is revealed. After a few rounds you will feel 0.3 versus 0.8 in your gut. That intuition is the payoff — the number exists so you can compare relationships without redrawing them.

08The algebra of the cross term

One rung is missing before we can trust correlation's range, and it also pays a debt from last chapter. When you add two variables, do their variances simply add? Chapter 15 said "only when independent" and left the reason parked. Here is the reason. Apply the variance definition to the sum. Note that (X + Y) deviates from its mean by (X − μ_x) + (Y − μ_y), then square that.

The algebra says the spread of a sum carries a 2Cov term. Now feel it. Slide the correlation from −1 to +1 and watch the pile of X + Y breathe. The pile is widest at the +1 end, where the two variables move together. It collapses to a single spike at the −1 end, where they perfectly cancel.

The square (a + b)² = a² + 2ab + b² gives three pieces. Take expectations term by term and you get Var(X + Y) = Var(X) + Var(Y) + 2·Cov(X,Y). There it is: the variances do not just add — a cross-term rides along, and it is twice the covariance. So Chapter 15 was right for one reason only. When X and Y are independent, the covariance is zero, the cross-term vanishes, and the variances add cleanly. When they move together, covariance is the exact correction that makes spreads combine. Run it on the welded dice: 35/12 + 35/12 + 2 × (−35/12) = 0, which is why that sum never moved off 7. That is the debt paid in full.

Now correlation's range falls out honestly, from a fact nobody argues with: a variance can never be negative. Standardize both variables, so each one has variance 1 and their covariance is ρ. Then look at Var(Z_x − Z_y), which must be ≥ 0.

The proof said ρ = ±1 means the points lie on an exact line. Now test that fence with your hands. Drag these points until the meter reads 1.000. You will find it locks only when every point sits on one straight line, and knocking a single point off drops ρ below the ceiling at once.

Turn the cross-term crank on it: Var(Z_x − Z_y) = 1 + 1 − 2ρ = 2 − 2ρ. That has to be at least zero, so ρ ≤ 1. Do the same on the sum, Var(Z_x + Z_y) = 2 + 2ρ ≥ 0, and you get ρ ≥ −1. So −1 ≤ ρ ≤ 1, always, proved in two lines. Nothing went in but the fact that a spread can never be negative. And the edges of that range mean something exact. If ρ = 1, then Var(Z_x − Z_y) = 0. Zero spread means no variation at all, so Z_x − Z_y is a constant. The standardized points then lie exactly on a line. That is what ±1 means: Y is a perfect linear function of X. It also closes the loop on Chapter 15's welded world, where Y = 7 − X is exactly linear. The covariance there was −2.917, a number in pips squared. The correlation there is −1, right on the floor, and that one carries no units at all. Correlation is co-movement measured as a fraction of the most a straight line allows: ±1 on the line, 0 for the tilt-free blob.

09The one place zero is safe

The whole chapter has drilled one warning: zero correlation is not independence. Now I owe you the single regime where it is. I want to be careful here, because after all that drilling this can feel like a betrayal. It isn't. It is a property of one special shape, not a reversal of the rule.

Take a jointly Gaussian pair — the tilted-ellipse bell from Chapter 14. Write out its density and look at where x and y actually touch each other. They are coupled in exactly one place: a single ρ·x·y cross-term buried in the exponent. Everything else in the formula is an x-part or a y-part, sitting apart. So watch what happens when you slide ρ to zero.

As ρ → 0, that lone cross-term disappears, and the exponent splits into an x-part plus a y-part. And a sum in an exponent is a product of exponentials — e^a+b = e^a·e^b. Try it with a = 1 and b = 2: e to the 1 is 2.718, e to the 2 is 7.389, and their product 20.09 is e to the 3. So the joint density factorizes into f_X(x)·f_Y(y), which is Chapter 14's exact definition of independence. You can watch the contours relax from a tilted ellipse to a perfect circle as it happens. For a Gaussian, and only in a case like this, zero correlation really does mean independent.

And there is a clean reason why the Gaussian is the exception. It has no curve for covariance to be blind to. The U-shape fooled covariance precisely because its dependence was nonlinear — a trend the straight-line ruler could not see. A Gaussian is a pure elliptical blob: all of its dependence is the linear tilt, with nothing hidden underneath. So once you kill the tilt, there is genuinely nothing left to miss. The exception and the rule are the same idea, read from two ends.

10What you carry out

Before the map, one honest check. Everything so far has been the average over a whole distribution. Real data is a finite pile of points, and the covariance you compute from it is a sample covariance — the same signed-product average, taken over the darts you actually have.

This one runs real code, with no formula smuggled in. Scatter a few hundred darts from an oval generator, compute the sample covariance, and it lands near zero. Now rotate the generator so the cloud leans, recompute, and the number climbs off zero — with its sign matching the lean, exactly as the tilt picture promised. Sample and population tell the same story, which is the whole reason the theory is worth carrying.

So here is what you leave with, and it is smaller and stronger than a page of formulas. Covariance is the expected signed product of deviations. Its sign is the cloud's tilt. And it is literally variance with the second variable swapped in — Cov(X,X) = Var(X). Correlation is that same number with the units divided out, pinned into [−1, 1], hitting the edges exactly when the relationship is a perfect line. The welded dice show the pair side by side: a covariance of −2.917 in pips squared, and a correlation of −1 in no units at all. And the warning you will never drop: both of them read only the straight-line thread, so r = 0 does not mean independent unless the cloud is Gaussian.

Look where that leaves the course. Chapter 15 built two numbers for one variable — where it balances and how far it sprawls. This chapter named the cross-term that appears the moment two variables are added, and gave it a clean, unit-free scale. That cross-term is not a curiosity. Chapter 17 bounds the tails of any distribution using nothing but a mean and a variance, and it proves the Law of Large Numbers the earlier chapters kept promising. Chapter 18's Central Limit Theorem asks about the spread of a sum, which you now know is unanswerable without the term you were handed here. And covariance at scale — a whole grid of them between many variables — is exactly the machine behind linear regression and principal component analysis. You are holding the seed of both.