18The MGF and the Central Limit Theorem

Add up almost any independent random variable — coin flips, wait times, a two-humped monster nobody has named — then standardize the sum, and one particular bell appears. Every time. That is the most astonishing fact in the subject, and it is not a coincidence. It is forced. Chapter 17 climbed to the Law of Large Numbers, and the law delivered exactly one thing: the sample average lands on μ. That is a statement about a point. This chapter asks about the wobble left around that point. The parent's shape washes out on the way, and only its mean and variance survive the trip. Proving that needs one genuinely new tool. The moment-generating function packs a whole distribution into a single curve, and holds every moment in its Taylor coefficients. Best of all, it turns sums of variables into products of curves, so the impossible convolution becomes multiplication a human can push to the limit. At the center of the proof sits one beautiful accident of size. After the √n standardization, the variance term is exactly the right size to survive the limit, while every higher term carries a bigger power of n underneath and dies. That is why the bell is universal, and by the end you will be able to re-derive it from nothing.

01The question the law left open

The Law of Large Numbers names a center and stops there. It never says what the wobble around that center looks like, and that gap is this whole chapter. So let's stand on the rung we just built. Chebyshev handed us P(|X̄ₙ − μ| ≥ a) ≤ σ²/(na²), so the odds of the average straying die as n grows, and the sample mean converges in probability to μ. Now read that promise closely and notice how little it pins down. It names a center, one number. It says the wobble around that number shrinks. That is all it says. The scatter of a real average around μ has a shape, and the Law of Large Numbers is silent about it.

The LLN and the CLT are the same scene under two different magnifications, and the divisor you pick is the zoom knob. Take the centered sum Sₙ − nμ and choose what to divide by. Divide by n and you get the average's deviation, which collapses to a point — that collapse is Chapter 17. Divide by √n instead and something better happens. At n = 100 that divisor is √100 = 10 rather than 100, ten times gentler, so the wobble neither dies nor explodes. It holds steady, and now it has a visible shape. Try all three lenses below and watch which one keeps a picture in the frame.

The middle lens is the whole game, and it is why the two theorems never contradict each other. Dividing by n zooms out until the scatter is a dot. That proves the average settles, but it shows you nothing of its texture. Dividing by √n is the exact zoom that holds the scatter at unit size, so its shape fills the view at every n. The LLN says the wobble vanishes on the coarse scale. The CLT magnifies that vanishing wobble and asks what it looks like inside. The answer is the shock this chapter is built around.

Commit to a prediction before you touch the machine, and be honest about it. Take the exponential from Chapter 12: all its mass piled near zero, a long lopsided tail running right. Now add up fifty independent copies and standardize the sum. Surely the result leans right like its parent? A skewed thing summed should stay skewed — that is what the gut says. Sketch your answer in your head, then slide n and watch what actually happens, on every parent in the menu.

The bell. The same bell, whether you fed it the exponential, a fair coin, a flat uniform, or the two-humped monster. Four parents with nothing in common produce one identical standardized shape, and that is universality. It should feel wrong. Each parent's skew, its lumps, its whole personality get washed out, and only its mean and variance make it into the limit. Write the object down, because it is the hero of the chapter: Zₙ = (Sₙ − nμ)/(σ√n), the standardized sum, the z-map from Chapter 12 aimed at Sₙ. The rest of the chapter is one long answer to the only reasonable question left. Why?

02Sums are convolutions — and that road closes

The direct road to that bell is a wall, and feeling the wall is what makes the new tool worth building. We want the distribution of a sum of independent variables. Your gut may whisper a shortcut here: just add the curves. Each die from Chapter 1 has a flat PMF, every face carrying 1/6. If adding variables added their curves, two dice would give a flat line at 1/3, since 1/6 + 1/6 = 1/3. Run the test below and watch two things break at once.

Adding the curves fails before it starts. The heights now total 2, not 1, so the result is not even a legal distribution, and Chapter 3's axioms throw it out at the door. The true shape of a two-dice sum is not flat anyway. To find P(sum = 7) you must count every way to split 7 across the two dice: 1+6, 2+5, 3+4, 4+3, 5+2, 6+1. That is six splits out of 36 equally likely cells, which makes the answer 6/36 = 1/6, about 0.17. On the old 6×6 grid those splits are exactly one diagonal, and stepping the target total sweeps that diagonal across the square.

Out comes a triangle, peaked at 7, and the move you just performed has a name. Summing over every split of the total is called convolution. It is the honest operation behind every sum of independent variables, discrete or continuous, dice or wait times. So adding the variables means convolving the distributions, never adding them. Now scale the job up. A third die means every total collects splits of splits, and by five dice each value of the sum gathers hundreds of terms: the total 18 alone collects 780 splits out of 7776 equally likely rolls. Stack the dice below and watch the arithmetic grow while the shape smooths.

Two things stare back at you. First, the workload: each extra die convolves the whole answer again, and chasing that shape all the way to n → ∞ by hand is hopeless. Second — hold this thought — the triangle is already bowing outward, and by five dice the outline looks suspiciously like a bell. So the phenomenon is begging for a proof the direct road cannot deliver. What we need is a transform: a different representation of a distribution in which convolution becomes something easy. That tool exists, and building it is the next three sections.

03The machine: M(t) = E[e^(tX)]

Here is the tool, first as a bare definition, and I want to be upfront about how strange it looks. The moment-generating function of a random variable X is M(t) = E[e^(tX)]. Read it slowly. Pick a number t, weight every outcome x by e^(tx), and take the expected value of those weights. Do that for each t and you trace out one curve. A whole distribution goes in, and a single function comes out. Why the exponential, of all weightings? I owe you that reason, and I will pay the debt two sections from here. For now let's just turn the crank once, on the fair coin, and confirm the machine produces something real.

For a coin worth 0 or 1, each with probability ½, the expectation has just two terms: M(t) = ½·e^(t·0) + ½·e^(t·1) = (1 + e^t)/2. No mystery and no infinities — an honest curve you can plot. Feed it t = 1 and it returns (1 + 2.718)/2 ≈ 1.86. Feed it t = 0 and it returns M(0) = 1, and that is no accident. At t = 0 the weight e^(0·x) equals 1 on every outcome, so M(0) = E[1] = 1 for every distribution on earth. So what does turning the dial actually do? Push t positive and e^(tx) inflates the weight on big outcomes. Pull it negative and the small outcomes dominate instead. Tilt the dial on a die below and watch the weights lean.

So M(t) is a panorama: the same distribution photographed from every tilt of the dial, one snapshot per t, all stitched into one curve. That is a promising amount of information for one function to carry. What we need next is evidence that it carries all of it — that nothing about the distribution escapes the curve. That evidence is written in the derivatives, and it is where the machine gets its name.

04Why "moment-generating" — the fingerprint

One recall first, so nobody stands on a rung we never laid. When we built the Poisson in Chapter 11, we met e through the limit (1 + a/n)ⁿ → eᵃ. The same number has a second identity, the power series: e^u = 1 + u + u²/2! + u³/3! + …, which converges for every u you feed it. Put u = 1 in and the first four terms already give 1 + 1 + 0.5 + 0.167 = 2.667, against a true e of 2.718. If the series is rusty, watch its partial sums hug the true curve below — each added term buys a better fit. This expansion is about to do the chapter's heavy lifting, twice.

Now substitute u = tX inside the expectation and let linearity, Chapter 15's workhorse, pull E through the series term by term: M(t) = 1 + t·E[X] + t²·E[X²]/2! + t³·E[X³]/3! + …. Look at what landed in the coefficients. The quantities E[X], E[X²], E[X³], … are called the moments of the distribution, and the MGF has them stacked in order, one per power of t. That stacking is the whole trick. Differentiating at zero peels them off one at a time: M′(0) = E[X], then M″(0) = E[X²], and so on up the ladder. Turn the crank on the coin below and watch each derivative strip off exactly one moment.

Do it on the coin and check the harvest against numbers you already own. Differentiate (1 + e^t)/2 and you get e^t/2. At t = 0 that is 1/2, because e^0 = 1, so the coin's M′(0) comes out to 1/2 — exactly its mean from Chapter 15. Differentiating again gives e^t/2 again, so M″(0) is also 1/2, which is E[X²]. Then E[X²] − (E[X])² = 1/2 − 1/4 = 1/4 is precisely the coin's variance. The MGF generates moments, and the name has earned itself. File the deeper reading too: M(t) is the distribution's Taylor series, the entire ladder of moments packed into one function.

Here is the claim everything rests on, stated before the argument for it: the MGF is a fingerprint. If the MGF stores every moment, does it pin down the whole distribution — could two different distributions ever share one MGF? The answer is no. Same MGF, same distribution. And here I owe you radical honesty, because this is the one claim in the chapter we will not prove. The Taylor picture makes it believable: pin every moment and you have pinned every coefficient, and with them the whole function. But the airtight proof needs later mathematics, Lebesgue integration and complex analysis, tools a few courses past this one. We take exactly this one plank on trust, and we say so out loud.

That plank converts our goal into something reachable, and it is the whole strategy of the proof. To prove the standardized sum becomes normal, we never have to touch its distribution directly. The convolution wall stays untouched. We only need its MGF to converge to the normal's MGF. Match the fingerprints, and uniqueness finishes the argument for us. The strategy stands ready the moment sums become tractable.

05The payoff: sums become products

Now the debt comes due: why e^(tX), of all possible weightings? Because the exponential turns a sum into a product, and that is the punchline the definition was built for. The exponent law from way back — e^(a+b) = e^a · e^b — turns e^(t(X+Y)) into the product e^(tX) · e^(tY). Then independence does the quiet, load-bearing thing. For independent X and Y, the expectation of a product of separate functions factors: E[f(X)·g(Y)] = E[f(X)] · E[g(Y)]. Name that rule and keep it. It is the clean cousin of Chapter 16's dying cross-terms, and most books apply it silently. Chain the two moves and the result falls out: M_{X+Y}(t) = M_X(t) · M_Y(t). Watch both moves land below.

Convolution just became multiplication. Hold that next to the dice grid and feel the trade you made. On the convolution side, every value of the sum demanded a sweep over every split of the total, and again for every extra variable. On the MGF side, two curves multiply pointwise, and n identical variables cost nothing more than an n-th power. That is exactly the trick logarithms pull on arithmetic — hard operation upstairs, easy operation downstairs — and it is the entire reason the MGF's weighting is an exponential. The strange definition was reverse-engineered from this payoff.

But the factoring step was purchased with independence. So before we trust it with a theorem, let's break it on purpose and watch the receipt bounce.

Dial the dependence up until Y is a full copy of X, and the two curves visibly part ways. The true MGF of the sum is now (1 + e^(2t))/2, because X + X doubles the coin. The product formula claims (1 + e^t)²/4 instead. At t = 1 the true value is 4.19 while the product's claim is 3.46 — not a rounding gap, a different curve. Dependence revives all the machinery that multiplication was hiding. So mark the hinge in ink: independence is not fine print in what follows. It is the exact clause that lets sums become products at all.

06The target: the normal's own MGF

A proof that ends "…and therefore the MGF converges to the normal's" needs a finish line to converge to. So let's compute the normal's own MGF once and pocket the answer. Put the weight e^(tx) against the standard normal density from Chapter 10, and the whole integrand's exponent is tx − x²/2. The computation is one algebra move, completing the square: tx − x²/2 = t²/2 − (x−t)²/2. I'll show it honestly, but don't let the algebra fog the point. The point is the answer. Step through the derivation below.

The normal's own MGF is M(t) = e^(t²/2) — at t = 1 that reads e^(1/2) ≈ 1.65 — and here is where it comes from. Pull the constant e^(t²/2) outside the integral and look at what remains: the same bell, slid sideways to sit at t instead of 0. Sliding a curve changes nothing about its area, and the bell's area is 1, since Chapter 10 built its normalizer for exactly that job. So the leftover integral collapses to 1, and the pulled-out constant is the entire answer. Memorize that curve. The entire summit is now one sentence long — show that the standardized sum's MGF walks to e^(t²/2), and uniqueness does the rest.

07The keystone: only the variance survives

Time to assemble, and the first move is the z-map doing what it has always done. Standardize each piece first: let Yᵢ = (Xᵢ − μ)/σ, so every Yᵢ has mean 0 and variance 1. Then the standardized sum is Zₙ = (Y₁ + ⋯ + Yₙ)/√n. And before anything else, honour the gut's standing objection: why divide by √n and not n, like the average does? Because the sum's own spread grows like √n·σ, which was Chapter 17's first stroke. At n = 100 the sum is spread over 10σ, so dividing by √100 = 10 hands back a width of exactly σ. Slide the exponent of the divisor below and watch each choice play out.

Divide by n and the wobble dies — you have re-proved the LLN, and the frame goes blank. Divide by nothing and the wobble spreads without limit. Divide by √n and the picture holds still forever, at width σ = 1 for our standardized pieces. So √n is the precision rate: the exact magnification under which the shape neither collapses nor escapes. It is the same rate that priced every poll in Chapter 17, and the CLT lives entirely at this zoom level.

One small rule before the main event, stated out loud so no step of the proof is silent. Scaling a variable re-dials its MGF: M_{aX}(t) = E[e^(taX)] = M_X(at). Nothing deep is happening — the constant a simply rides inside and lands on the dial. With a = 1/10, a dial setting of t becomes a dial setting of t/10. Verify it below, because the proof is about to feed t/√n into M_Y on the strength of this line.

Now watch closely, because the next three lines are the theorem. Sum→product plus the scaling rule give M_{Zₙ}(t) = [M_Y(t/√n)]ⁿ: n identical factors, one shrunken dial. Expand M_Y with the Taylor series and let the standardization make its two kills. E[Y] = 0 deletes the linear term outright. E[Y²] = 1 pins the quadratic coefficient at t²/2. What survives on the page is M_Y(t/√n) = 1 + (t²/2)/n + (skew term)/n^{3/2} + …. So the whole theorem is now one bracket: [1 + (t²/2)/n + smaller]ⁿ. Before you slide anything, commit to a prediction. As n → ∞, which term inside that bracket is the only one that neither blows up nor dies?

Only the variance. The quadratic term is (t²/2)/n, a constant over n — exactly the shape the old Poisson limit (1 + a/n)ⁿ → eᵃ eats. It fires one last time and leaves e^(t²/2). Every higher term carries n^{3/2} or worse underneath, and the ()ⁿ can only rescue terms of size 1/n. So the skewness contribution dies like 1/√n and the kurtosis like 1/n: at n = 10,000 the skew term is down to 1/100 and the kurtosis term to 1/10,000. Every trace of the parent's personality is literally divided away by the √n lens. Its variance alone was the right size to survive. The limit is e^(t²/2), the normal's fingerprint, and uniqueness converts fingerprint into distribution: Zₙ → N(0,1). That is the Central Limit Theorem, and that is why the bell is universal. The bell was never about the parent. It is what remains of any parent once √n has filtered out everything but the variance.

Step back and look at the whole chain as one object, because there is no borrowed link anywhere in it. Every arrow below is a section you already own.

The proof hands you a limit: as n runs to infinity, any parent's standardized sum becomes the bell. But a limit is a destination, not a starting line. A real problem has a real, finite n, so the honest question is how fast the bell arrives. You have probably heard the rule of thumb n ≥ 30. Set a lopsided parent below, guess the n where it finally looks Gaussian, then drive n and watch the lean fade.

08The fine print, and the summit

Now the fine print, spoken plainly, because a theorem this famous deserves exact wording. First: the CLT does not say Zₙ is normal at any finite n. It says Zₙ converges in distribution to N(0,1). The approach is the theorem, and at small n the parent's fingerprints still show. Second: the proof leaned once, quietly but completely, on E[Y²] = 1 being a finite number at all. Finite variance is load-bearing. There are distributions that refuse it, and the most honest way to respect that clause is to break it. Feed the blender a Cauchy parent below and wait for the bell.

A Cauchy breaks the bell everywhere, at every n you can reach. A well-behaved parent is kinder, yet it is not equally kind at every point. The fit in the middle and the fit out in the tails are not the same, and the weaker of the two is exactly where rare-event odds get read. Before you trust a bell on a one-in-a-million question, ask where its error is hiding. Guess first: is the fit worst in the centre, or out in the tails?

The bell never comes, and the reason is the clause we just named. The Cauchy's tails are so heavy that its variance integral diverges — no σ exists, at any price. So there is no variance term left to survive the limit, and the engine of the proof has no fuel. Worse, the average of n Cauchy draws has exactly the same Cauchy distribution again, as wide at n = 10,000 as at n = 1. More data narrows nothing. When a real dataset shows monstrous tails — crash returns, flood peaks — this is not a curiosity. It is a warning label on every bell-shaped assumption downstream.

One last honest note, about which object actually converges. A discrete parent's density is a row of spikes, and a row of spikes never pointwise-approaches a smooth curve, no matter how large n gets. Integration repairs it. The CDF turns each spike into a jump, and the resulting staircase genuinely does snake onto the normal's smooth CDF Φ. Compare the two views below.

So the theorem says P(Zₙ ≤ z) → Φ(z). That line is not only a proof's end. It is a recipe. To find the odds a total clears some value, subtract nμ, then divide by σ√n, then read Φ. Everything in this chapter was built so you can run those three steps by hand. Pick a parent, set a threshold, predict the odds, then check your number against the exact answer.

So the official statement tracks the CDF: P(Zₙ ≤ z) → Φ(z) at every point z. That is what "converges in distribution" means, and it is why careful books prove the CLT on CDFs. Integration smooths spikes into jumps, and the smoothed object is the one that behaves. You have now heard the theorem exactly: independent pieces, finite variance, standardize by √n, and the CDF of the sum approaches the bell's. Nothing more is claimed, and nothing less.

Stand at the top and look back down the ladder, because you have climbed the whole thing. Events, then random variables, then distributions, then their summaries, then the two limit theorems. The LLN names the center. The CLT names the shape. Two halves of one story, split by a single power of √n.

And the tool you built is bigger than this chapter. The MGF is the Laplace transform of a distribution wearing probability clothes, and its trick is the oldest one in engineering: move to a domain where the hard operation is easy. Logarithms turn multiplication into addition. Transfer functions turn cascaded systems into products. The MGF turns convolution into multiplication, and its cousin the characteristic function powers the rigorous proofs. That same transfer is why the bell owns experimental science. Every measurement error is a sum of small independent nudges — thermal, mechanical, human — and the CLT grinds their shapes away until only the bell is left. When a lab plots Gaussian error bars, it is quietly citing the theorem you can now re-derive from nothing. That is the summit, and the view is the whole map. One page left, to walk it.