Scan barcode
A review by lpm100
The Signal and the Noise: Why So Many Predictions Fail--But Some Don't by Nate Silver
informative
medium-paced
5.0
Book Review
The Signal And The Noise
Nate Silver
5/5 enthusiastic stars
*******
This book is kick-ass good: It feels like a more readable/ less self-aggrandizing write of "Fooled by Randomness"/ "The Black Swan" but with the writing skill of Levitt/Dubner (of "Freakonomics" fame).
Nate Silver is a rare professional/ applied statistician (he used to make a living playing poker, and repurposed those skills to making political / economic predictions) who can communicate in words.
Of the book:
-455 pages of prose over 14 chapters+ introduction.
-≈30pps/chapter
-Needn't be read in order
The book is essentially about discerning signal from noise, and especially in noisy systems. (Economics. Politics. Climate.) A running theme throughout is also: testing the data generated by a model (as a way to see how well the model ACTUALLY WORKS).
***A few salient points from each chapter:
Intro. Information now is much greater than it has been, but the signal to noise ratio is lower than it ever has been. The internet makes this worse.
1. Risk≠uncertainty. Simplified explanation of collateralized debt obligations. (CDOs). Resynopsis of the 2008 mortgage based financial crisis. "Out of sample." Precision ≠accuracy.
2. Poor predictive powers of experts, à la Philip Tetlock. "Foxes" and "hedgehogs" (applied both to models and pundits). Presidential, senatorial, and gubernatorial elections are easy to predict. The House is much harder.
3. A fairly short chapter about predictive methods of baseball statistics. (Much more appropriate as a cure for insomnia.)
4. Chaos theory. Laplace's demon. Nonlinear error compounding. Discussion of weather forecasting / meteorology. Explanation of why weather forecasts take billions and billions of calculations. (Forecasts get worse the further you go out, and past 8 days are nowhere near accurate.)
5. Gutenberg-Richter law. (Power law distributions.) Overfitting. Time dependent forecasting (where the probability of something is not assumed to be constant across time). Prediction ≠Forecast. Some things are inherently unpredictable, but forecastable. Such as earthquakes. If you know that an earthquake is a once every 3 century event, what incentive does a developing country have to prepare for something so far away? History of a lot of very stupid false earthquake predictions. Chaos theory≠complexity theory.
6. Extended thoughts on the weakness of economic predictions of "experts" as well as reasons for the inherent unpredictability of economic systems.
7. If you have seen one pandemic, you have seen one pandemic. Extrapolation on exponential increase is a fool's errand. Self-fulfilling prophecies and self-canceling prophecies. Increased media mention of some disease may cause increased self diagnosis. (Transgender hysteria today is the autism of yesterday. )
8. Introduction to Bayesian reasoning (especially as contrasted to frequentism). Larger numbers of observations also create larger amounts of noise. (Ionnidis: "Why Most Published Research Findings Are False.") Bayer Laboratories was not able to replicate 2/3 of positive findings claimed in medical journals.
9. Calculation (domain of computers)≠strategy (current domain of humans), demonstrated by trying to computerized chess. The best computers could be the best chess players, but for calculation limits.(10^10^50 possible games.)
10. Practical application of Bayes Theorem to professional Texas hold'em / poker. All poker tables rely on an incompetent poker player to subsidize the winnings of the rest. The best poker players can win a lot less money per hand than the worst poker players can lose.
11. Efficient market hypothesis as well as testing. No, efficient market hypothesis doesn't work perfectly in the Real World (WOW! What is the world coming to? Academics generating ideas that have no basis in reality) because if everybody realized that they could not beat the market, then no one has a reason to trade. But in reality, the great majority will make only average returns on the stock market.
12. Climate change. Clausius-Clapeyron relation. Initial condition uncertainty; structural uncertainty; scenario uncertainty. Intergovernmental Panel on Climate Change (IPCC) 20 different climate models, but they settle on one by consensus (and NOT by predictive validity of the models). No one disagrees on climate change, but the question is whether or not it is even predictable (economics has much better mathematicians, and a longer track record, and an unbroken record of failure). What your trend is depends on what year you use as baseline, which may be quite arbitrary.
13. Unfamiliar≠improbable. Mind-blindness. It's much easier to pick out signal from noise after something has already happened, and it's easier to see the signal that you want in a mess of data. (Governments tend to not be very good at distinguishing signal from noise. The September 11th attacks were shocking but not surprising.) Application of power law distributions to explain the September 11th attack as well as predict an even bigger attack somewhere down the road. (These big attacks are estimated to happen once every 80 years. [p.432]). All roads lead to Pakistan. (Islamic, nuclear capable, with most of the population illiterate/educated in a way that's more appropriate for the 11th century.)
Good information about how to live with attacks by destructive Arabs/Muslims is no further away than Israel--because they do just that. (I doubt that the United States will take advantage of this rich source of information. They're probably too busy litigating cases about men using women's bathrooms.)
*******
Take away messages:
1. Garbage in, garbage out. ("Data-driven"ness notwithstanding.)
2. Very few forecasters take the trouble to actually test their predictions. (And nothing wrong with updating your model if your predictions turn out to be empirically false!)
3. Unfamiliar≠improbable.
Good quote:
"Moody's 50% adjustment was like applying sunscreen and claiming it protected you from a nuclear meltdown."
"The statistician drowned crossing a river that was only 3 ft deep on average." (p.179).
"The market can stay irrational longer than you can stay solvent." (p.360)
Verdict: Strong recommendation.
The Signal And The Noise
Nate Silver
5/5 enthusiastic stars
*******
This book is kick-ass good: It feels like a more readable/ less self-aggrandizing write of "Fooled by Randomness"/ "The Black Swan" but with the writing skill of Levitt/Dubner (of "Freakonomics" fame).
Nate Silver is a rare professional/ applied statistician (he used to make a living playing poker, and repurposed those skills to making political / economic predictions) who can communicate in words.
Of the book:
-455 pages of prose over 14 chapters+ introduction.
-≈30pps/chapter
-Needn't be read in order
The book is essentially about discerning signal from noise, and especially in noisy systems. (Economics. Politics. Climate.) A running theme throughout is also: testing the data generated by a model (as a way to see how well the model ACTUALLY WORKS).
***A few salient points from each chapter:
Intro. Information now is much greater than it has been, but the signal to noise ratio is lower than it ever has been. The internet makes this worse.
1. Risk≠uncertainty. Simplified explanation of collateralized debt obligations. (CDOs). Resynopsis of the 2008 mortgage based financial crisis. "Out of sample." Precision ≠accuracy.
2. Poor predictive powers of experts, à la Philip Tetlock. "Foxes" and "hedgehogs" (applied both to models and pundits). Presidential, senatorial, and gubernatorial elections are easy to predict. The House is much harder.
3. A fairly short chapter about predictive methods of baseball statistics. (Much more appropriate as a cure for insomnia.)
4. Chaos theory. Laplace's demon. Nonlinear error compounding. Discussion of weather forecasting / meteorology. Explanation of why weather forecasts take billions and billions of calculations. (Forecasts get worse the further you go out, and past 8 days are nowhere near accurate.)
5. Gutenberg-Richter law. (Power law distributions.) Overfitting. Time dependent forecasting (where the probability of something is not assumed to be constant across time). Prediction ≠Forecast. Some things are inherently unpredictable, but forecastable. Such as earthquakes. If you know that an earthquake is a once every 3 century event, what incentive does a developing country have to prepare for something so far away? History of a lot of very stupid false earthquake predictions. Chaos theory≠complexity theory.
6. Extended thoughts on the weakness of economic predictions of "experts" as well as reasons for the inherent unpredictability of economic systems.
7. If you have seen one pandemic, you have seen one pandemic. Extrapolation on exponential increase is a fool's errand. Self-fulfilling prophecies and self-canceling prophecies. Increased media mention of some disease may cause increased self diagnosis. (Transgender hysteria today is the autism of yesterday. )
8. Introduction to Bayesian reasoning (especially as contrasted to frequentism). Larger numbers of observations also create larger amounts of noise. (Ionnidis: "Why Most Published Research Findings Are False.") Bayer Laboratories was not able to replicate 2/3 of positive findings claimed in medical journals.
9. Calculation (domain of computers)≠strategy (current domain of humans), demonstrated by trying to computerized chess. The best computers could be the best chess players, but for calculation limits.(10^10^50 possible games.)
10. Practical application of Bayes Theorem to professional Texas hold'em / poker. All poker tables rely on an incompetent poker player to subsidize the winnings of the rest. The best poker players can win a lot less money per hand than the worst poker players can lose.
11. Efficient market hypothesis as well as testing. No, efficient market hypothesis doesn't work perfectly in the Real World (WOW! What is the world coming to? Academics generating ideas that have no basis in reality) because if everybody realized that they could not beat the market, then no one has a reason to trade. But in reality, the great majority will make only average returns on the stock market.
12. Climate change. Clausius-Clapeyron relation. Initial condition uncertainty; structural uncertainty; scenario uncertainty. Intergovernmental Panel on Climate Change (IPCC) 20 different climate models, but they settle on one by consensus (and NOT by predictive validity of the models). No one disagrees on climate change, but the question is whether or not it is even predictable (economics has much better mathematicians, and a longer track record, and an unbroken record of failure). What your trend is depends on what year you use as baseline, which may be quite arbitrary.
13. Unfamiliar≠improbable. Mind-blindness. It's much easier to pick out signal from noise after something has already happened, and it's easier to see the signal that you want in a mess of data. (Governments tend to not be very good at distinguishing signal from noise. The September 11th attacks were shocking but not surprising.) Application of power law distributions to explain the September 11th attack as well as predict an even bigger attack somewhere down the road. (These big attacks are estimated to happen once every 80 years. [p.432]). All roads lead to Pakistan. (Islamic, nuclear capable, with most of the population illiterate/educated in a way that's more appropriate for the 11th century.)
Good information about how to live with attacks by destructive Arabs/Muslims is no further away than Israel--because they do just that. (I doubt that the United States will take advantage of this rich source of information. They're probably too busy litigating cases about men using women's bathrooms.)
*******
Take away messages:
1. Garbage in, garbage out. ("Data-driven"ness notwithstanding.)
2. Very few forecasters take the trouble to actually test their predictions. (And nothing wrong with updating your model if your predictions turn out to be empirically false!)
3. Unfamiliar≠improbable.
Good quote:
"Moody's 50% adjustment was like applying sunscreen and claiming it protected you from a nuclear meltdown."
"The statistician drowned crossing a river that was only 3 ft deep on average." (p.179).
"The market can stay irrational longer than you can stay solvent." (p.360)
Verdict: Strong recommendation.