The Flaw of Averages: Why We Underestimate Risk in the Face of Uncertainty

This is a book about how to manage in a world of uncertainty. The beginning section presents a number of statistical concepts in simplified language; more sophisticated techniques follow. The author discusses how these concepts apply in a wide variety of contexts including oil exploration, pharmaceutical R&D, the stock market, the housing bubble, weather, climate change, health care, gene pools, the war of terror, and supply chains. He also describes some illogical accounting rules required by the FASB and the SEC.DISTRIBUTIONS. At its core, the Flaw of Averages is an oversimplified view of reality by fixating on a single number. “If organizations are to conquer the Flaw of Averages, they must start dealing in probability distributions, not in single numbers.” A simple way to view a distribution is a histogram. Figure 9.8 shows the profit of “28 live-action films of a particular genre, which had been selected so as not to over-represent any single theme or star.” The average profit is $20 million, but the histogram reveals a much more insightful picture of the risk: 25% of the films lost more than $5 million.DIVERSIFICATION. If you invested in making one film, you’d have a 25% chance of losing your shirt, but the author ran some simulations to determine that investing in a portfolio of only four films would reduce the risk of loss to 1%, with the average profit remaining $20 million.MODELS. “The surest way I know to detect the Flaw of Averages is the build a small spreadsheet model of your business situation and simulate the uncertainties you face. A good model improves your intuition by connecting the seat of your intellect to the seat of your pants… The best models are those you no longer need because they have changed the way you think… Most large financial organizations spend millions of dollars on complex risk models… that may actually mask what’s going on… What they need are simple models that everyone understands.”UNCERTAINTY VERSUS RISK. “The terms ‘uncertainty’ and ‘risk’ are often used interchangeably, but they shouldn’t be… I consider uncertainty to be an objective feature of the universe, whereas risk is in the eye of the beholder.” Some people are more risk-averse than others when faced with the same uncertainty.INTERRELATED UNCERTAINTIES. “People also use the term Statistical Dependence when discussing interrelated uncertainties, but this phrase implies that one uncertainty depends on another, when in fact the relationship may be mutual. So I will stick to interrelated uncertainties instead… Scatter plots are my favorite way to grasp the interrelationships between uncertain numbers.”INFORMATION. “Information is the complement of uncertainty; that is, for every uncertainty, there is information that would reduce or remove it. Sometimes such information is cheap, and sometimes it is unavailable at any price… Information has no value at all unless it has the potential to change a decision.”EXTREMES. “Did I mention where one would find the smallest average earlobes? Small towns, of course. The sizes of towns and earlobes have nothing to do with each other; it’s just that averages with small samples have more variability than averages over large samples.”OPTIONS. “Recall the example of the gas well… that was worth $500,000 given the average price of gas but was worth $1 million on average. The increased value was due to the option not to pump in the event that the gas price was below the production cost. An opportunity such as this is known as a real option, and it is analogous to a call option on the gas, with a strike price equal to the pumping cost.”FUTURES. “Given how terrible individuals are at estimating probabilities, it is a profound result that marketplaces usually get them right… For example, a classic 1984 paper by UCLA finance professor Richard Roll shows that the futures market in frozen concentrated orange juice is a better predictor of Florida weather than the National Weather Service… Although the futures markets predict the average future values of assets, as we have seen, option prices indicate the degree of uncertainty.”ORGANIZATIONAL BOUNDARIES. David Cawlfield writes, “Of course the Flaw of Averages occurs everywhere, but often the flaws are small and go unnoticed, or the consequences are insignificant. At organizational boundaries the Flaw can cause real damage. This is where most of the cross-communication occurs between upper level managers, and involves decisions with large consequences. Lower in the organization, experts with the most knowledge about variability leave out the gory details about day-to-day ups and downs by communicating only the ‘big picture’ to the boss… An understanding of variability constitutes a large portion of the value of long work experience. The secret of successful Probability Management is learning to capture this understanding in a stochastic library.”THE DECISION FOREST. “A firm introducing a new product lineup should use a portfolio approach to avoid risks due to both cannibalization and potential competitive products. A portfolio of pharmaceutical R&D projects would be designed with both competitive drugs and potential governmental regulation changes in mind. Most people [don’t take] true portfolio effects into account… Think of each project as a decision tree reflecting uncertainties that pertain only to that project… This results in a bunch of decision trees or a decision forest… But modeling all of the individual projects as decision trees is not enough, because, as discussed earlier, portfolio decisions must reflect he interrelationships between constituent parts. For this, I model global uncertainties such as the price of oil or political upheaval as the winds of fortune, which blow through the entire forest, influencing all the trees at once.”MONTE CARLO. “A computational technique similar to the shaking of a ladder can test the stability of uncertain business plans, engineering designs, or military campaigns. Monte Carlo simulation, as it is known, bombards a model of the business, bridge, or battle with thousands of random inputs, while keeping track of the outputs. This allows you to estimate the chance that the business will go bust, the bridge will fall down, or the battle will be lost. The shaking forces you apply to the ladder are known as the input probability distribution and correspond to the uncertainty demand levels for your product, the magnitudes of potential earthquakes, or the sizes of the enemy forces you will encounter. The subsequent movements of the ladder are known as the output probability distribution and correspond to the profit of the business, the deflection of the bridge, or the number of casualties you suffer.”DISTRIBUTION STRING. “The sum of the simulations of the parts is not the simulation of the whole… The key is a new computer data type called the Distribution String (DIST), which encapsulated distributions in a manner that allows them to be added together like numbers… In terms of the spinner, think of the DIST as consisting of the outcomes of one thousand spins, stuffed, like a genie in a bottle, into a single cell in your spreadsheet.”SCENARIO LIBRARIES. In this example, risk is estimated across two banking divisions. “First, the chief probability officer (CPO) generates the distribution of housing market conditions. This is provided as a stochastic information packet (SIP)… One input SIP went in, and two output SIPs came out… They now form a scenario library unit with relationships preserved (SLURP), which maintains the dependence of each division on the housing market. The final step is just to add the output SIPs of the two divisions together to create the SIP of total profit… With SLURPs, the simulation of the sum does equal the sum of the simulations because it preserves the interrelationships… If you ignore the interrelationships, you would calculate the chance that both divisions lose money at the same time as 2/6 x 1/6 = 2/36 = 1/18.” With the consolidated method, “the chance of losing money is not 1 in 18 after all, but 1 in 3, six times greater!”“If a CPO has no idea of a particular distribution, they should pull several different ones from the seat of their pants and see how the results differ under each. Today’s interactive simulations are so fast that you can actually enter probabilities as variables and discover at what point you would make different decisions.”“But let’s not forget that risk is not inherently bad. In fact, risk, clearly presented and understood, is required for people to make investments. And if people don’t make investments, society pretty much grinds to a halt. Let’s also not forget that risk is in the eye of the beholder and that the risk attitude that a publicly traded firm should take on is the one anticipated by its shareholders. Unlike its employees, who may be more concerned about keeping their jobs than making profit, the shareholders, who are generally diversified across many other investments, want the firm to take the business risks that induced them to invest in the first place.”

Everyone in your company who makes decisions or supports decision-making needs to read The Flaw of Averages by Sam Savage. Why? Because together you are likely committing the flaw of averages, wasting time and exposing yourself to unnecessary risk. Savage shows you how to avoid this pitfall in a cleverly written, enlightening, and fun to read guide to making better decisions.Most likely you have never heard of Jensen's Inequality, and most likely you don't care. However, you should care, and The Flaw of Averages introduces why this concept poses profound implications to the way we tend to think about making decisions. The problem is that in thinking about the issues or opportunities we face and the decisions we exercise to address them, we often go through a kind of accounting process in which we consider the best case, most likely case, and worst case scenarios (or any number of scenarios) and the corresponding conditions that have to exist for each scenario to be realized. We let ourselves believe that those assumed conditions are averages that we can use as proxies for the full range of uncertainty we face. Our final conclusion is that the outcome of our analysis closely corresponds to the average real-world outcome and the extent of possible variation (if we get that far). Understanding that, we commit to action, often disastrously so.Savage reveals the flaw in the traditional way of thinking by explaining the implications of Jensen's Inequality. In short, Jensen's Inequality says that in situations where the output we care about varies in a non-linear way to inputs (which is much of life), the outcome as a function of average inputs (the flawed traditional analytic approach) is NOT equal to the average outcome as a function of the inputs treated as they naturally vary. [For those mathematically inclined, if E() is an operator that determines the average of a sample, and f(Xi) is a function of inputs, then E( f(Xi ) ) 'does not equal f( E(Xi) ). For those not so mathematically inclined, don't worry. The Flaw of Averages is not a math book; rather, it is a book about making decisions and how math can be used constructively to support that process.]The way around this failure is to use Monte Carlo simulation to consider simultaneously the effects of the range of the assumptions as they naturally vary on the outcome we care about. Instead of thinking about the outcome as a single point or a constellation of points representing exhaustive guesses about the future, we see the full range of potential outcomes and their likelihood as a distribution. We see the implications of our decisions and corresponding uncertainties as a picture and not a point. As a result, not only do we avoid never ending analysis paralysis, we gain a deeper appreciation for the effect of typically unconsidered outcomes, both good and bad, and are able to plan accordingly with contingencies and options.

Quite an interesting read. Explores how the general reliance on the "average number" in business, finance, technology can cause predictive problems. The author writes that one should use probability distributions instead of the average to make decisions and create realistic models.

People with some experience in life may grasp the key lessons to their benefit. Innocents will not.

Really interesting book on the use of statistics… appreciate the good condition it was delivered in!

I hope that many buy & read this book to start them on the road to a better understanding of the world we live in - one of uncertainty.Before some critical points I would like to make clear I thought this was a superb read and is, on average, excellent. I was no stranger to statistics when I read this book (mainly from my graduate education in physics and then post-graduate experience of Monte Carlo methods) or the world of finance (having worked as an IT supplier to that industry), but I learned a lot from reading this well written and easily consumed book.Critical Points: (which are clearly only my opinions so I do not use words like "I feel", "I think" etc in the following)- there are some technical points that are not well explained. One specific passage has two paragraphs that explain nothing of the content implied by the "header" of the subject. This was frustrating.- Oh my ! graphs, plots etc with axes not labelled ? very poor. A constant annoyance throughout the book- Simpson's paradox is not explained - not such a problem as it is a well documented elsewhere. (BTW - it's only sums.)- given the otherwise authoritative nature of the book the above are notable deficiencies.- there is more than a little element of "my dad did this", "my friends did that" - at first a bit annoying and then both charming (we can only hope our sons & friends are so appreciative) and then informing. The author was an "insider" to a lot of formative economics and statistical innovation as a child and then as a practitioner. (So not a criticism really - only in a time-series way)- there is an element of "evangelism" about the book and a lack of reflective self-criticism that detracts from the otherwise, albeit easy-going, analytical style.Overall - I recommend this book.

I work in IT - software development - and I've always had to deal with plans and estimates. This book explains with extreme clarity why the traditional way of using averages for such reasons is wrong in so many ways!I already knew the saying that you're not meant to use averages as they are wrong half the time, but this book still opened my eyes on the real reasons behind it.Some of my favourite quotes:- "when we use single numbers to estimate uncertain future outcomes we are not just usually wrong, but we are consistently wrong"- "Plans based on average assumptions are wrong on average"- "when managers ask for a forecast, they are really asking for a number, which involves the Flaw of Averages"- "risk is in the eye of the beholder"- "it is essential to replace numbers with distributions to cure the flaw of averages"- "weak form of the flaw of averages: a single number doesn’t give the whole picture"- "the strong form of the flaw of averages states that average or expected inputs don’t always result in average or expected outputs"- "information has no value at all unless it has the potential to change a decision"- "the simulation of the sum is not the sum of the simulation"By the way the book is well written. The author is funny and has an easy-to-read style - he manages to make difficult topics like finance and economics easy to understand even for me that I know very little about them.The bottom line for me is: whenever faced with uncertainties (which is the norm in software development) use distributions and simulations, not averages.