The Pareto Distribution

The Pareto distribution shows up in a remarkably wide array of complex systems. Together, “the” and “of” account for 10 percent of all words used in English. The most volatile day in the history of a stock market will typically be twice as volatile as that of the second-most volatile and ten times the tenth-most. Tag frequency on Flickr photos obeys a Pareto distribution, as does the magnitude of earthquakes, the popularity of books, the size of asteroids, and the social connectedness of your friends. The Pareto principle is so basic to the sciences that special graph paper showing Pareto distributions as straight lines rather than as steep curves is manufactured by the ream.

And yet, despite a century of scientific familiarity, samples drawn from Pareto distributions are routinely presented to the public as anomalies, which prevents us from thinking clearly about the world. We should stop thinking that average family income and the income of the median family have anything to do with one another, or that enthusiastic and normal users of communications tools are doing similar things, or that extroverts should be only moderately more connected than normal people. We should stop thinking that the largest future earthquake or market panic will be as large as the largest historical one; the longer a system persists, the likelier it is that an event twice as large as all previous ones is coming.

Clay Shirky on what is also known as the "long-tail effect" or "90% of everything is crap" effect.