Epistemology and risk management

Nicholas Taleb and Avital Pilpel discuss the perils of using known unknowns to predict the consequences of catastrophic events.

The need for epistemology

Risk management is a serious business. Accordingly, the production of a risk ‘measure’ must be subjected to the question ‘how do you know what you claim to know’ – in other words, epistemology. Claims regarding risk cannot be made without any rigorously established supervision of their validity. There is a need for skeptical inquiries concerning how a risk measure was obtained and how an opinion was formed. The fields of economics, finance, and insurance (in spite of their reliance on mathematics) have so far produced unreliable risk measures – particularly with the highly quantitative Modern Portfolio Theory (A). Very little check has been made on the theoretical and practical fitness of the assertions by the researchers and practitioners. Further, the discipline of statistics, with its confirmatory orientation, falls severely prey to the problem of induction – where proof of one level of probability is assumed to be proof of another.

Now, if the field of risk studies and quantitative risk management lacks adequate supervision, the field of mainstream epistemology itself provides no help for a decision maker under uncertainty (we tried!). Firstly, it is too theoretical, focusing on paradoxes of no practical use for decision makers – published material appears to focus on complications of what constitutes ‘Justified True Belief’. These are largely ‘rigorous’ but inconsequential for us. Secondly, traditionally the field deals more with whether some claim is true and justified rather than whether its disproval has some impact and consequence: how hurt am if I am wrong? We might be able to tolerate a 1 per cent error rate in some circumstances but not when the rare event, as we will see, dominates the statistical properties. The problem of induction is: how can we logically make claims about the unseen based on the seen? This is illustrated in philosophy with the exception of the ‘black swan’; which surprises those who (on the basis of past experience) thought that all swans were white. This might be inconsequential in logic: that is, the colour of a bird may not change much of our lives. But in risk management we need to deal with ‘black swans’ that have consequences. Further, a search of the literature in the philosophy and history of probability shows the depressing fact that the large impact event is absent from the discussions – the focus is on casino-style games that do not apply to real life situations (B).

This paper will discuss two epistemological problems with risk management – and present a possible simple solution to them, all linked to minimize reliance on inductive claims about rare events.

The First Epistemological Problem: induction and small probability

In the summer of 1982, US banks had a bad month. They lost more dollars than they ever made and would have been bankrupt had their portfolios been marked to market, or left without assistance from the United States Federal Reserve. Losses came from loans to ‘growing’ international markets. Because its lending was domestic, the industry called Savings and Loan was spared ... until about a decade later, when the business disappeared. This required a government sponsored bailout in the hundreds of billions of dollars (with the formation of the Resolution Trust Corporation). One single downturn in the early 1990s cost more than every penny previously made in the history of real estate lending in the US. Let’s consider an English example: after years of comfortable insurance fees, many of the ‘Names’ of Lloyd of London became suddenly insolvent after what appeared to be a great business (investment in asbestos manufacture) was simply the equivalent of sitting on a stochastic time bomb.

Thus, probabilities by themselves do not matter. They can be very small, but their results are not. What matters in life is the equation probability × consequence. This point might appear to be simple, but its consequences are not.

Suppose that you are deriving probabilities of future occurrences from the data, assuming that the past is representative of the future (C). An event can be an earthquake, a market crash, a spurt in inflation, hurricane damage in an area, a flood, crops destroyed by a disease, people affected in an epidemic, destruction caused by terrorism, etc. Note the following: the severity of the event, will be in almost all cases inversely proportional to its frequency: the ten-year flood will be more frequent than the 100 year flood – and the 100 year flood will be more devastating (D).

Now, say that you estimate that an event happens every 1,000 days. You will need a lot more data than 1,000 days to ascertain its frequency, say 3,000 days. Now, what if the event happens once every 5,000 days? The estimation of this probability requires some larger number, 15,000 or more. The smaller the probability, the more observations you need, and the greater the estimation error for a set number of observations. Therefore, to estimate a rare event you need a sample that is larger and larger in inverse proportion to the occurrence of the event.

We summarize: If small probability events carry large impacts, and (at the same time) these small probability events are more difficult to compute from past data itself, then: our empirical knowledge about the potential contribution – or role – of rare events (probability × consequence) is inversely proportional to their impact.

We understand so little about catastrophic events, yet these are the events that we talk about the most casually (E). In risk management terms, the bigger the event, the less we have a clue.

Probability distributions

This problem has been seemingly dealt with using the notion of ‘off-the-shelf’ probability distribution. A probability distribution is a model that assigns probabilities to the unseen based on some a priori representation – something pre-prepared for us and conveniently taught in a statistics class (F).

In other words, you can now confidently extrapolate from the seen to the unseen: you observe a variety of events and make inferences to those you haven’t seen, under some mathematical structure. But distributions have problems. They are self-referential.

The Second Epistemological Problem: the problem of self-reference

People using these probability distributions tend to forget that the distributions are not directly observable, which makes any risk calculation suspicious since it hinges on some presupposed knowledge. How do we know if we have enough data? If the data distribution is, say, the traditional bell curve then we may be able to say that we have sufficient data – for instance the bell curve itself tells us how much data we need. However, if the distribution is not from such a well-bred family, then we may not have enough data. How do we know which distribution we have on our hands? Well, from the data itself.

We can state the problem of self-reference in the following way: If one needs data to obtain a probability distribution to gauge knowledge about the future behavior of the distribution from its past results, and if, at the same time, one needs a probability distribution to gauge data sufficiency and whether or not it is predictive of the future, then we are facing a severe regress loop. This is a problem of self reference akin to that of Epimenides the Cretan stating whether the Cretans are liars or not liars. Indeed, it is too uncomfortably close to the Epimenides situation, since a probability distribution is used to assess the degree of truth – but cannot reflect on its own degree of truth and validity. And, unlike many problems of self reference, ones related to risk assessment have severe consequences.

Conclusion

We should not stop businesses from taking risks – just know that those which do not expose themselves to rare events are more robust, from an epistemological standpoint.

We can separate businesses using this epistemological robustness. In a business virtuous to the rare event, what the past did not reveal is almost certainly going to be good for you. For example, when you look at past biotech revenues, you do not see the emerging superblockbuster in them, and owing to the potential for a cure for a disease, there is a small probability that the sales in that industry may turn out to be monstrous, far larger than expected. On the other hand, consider businesses negatively exposed to rare events. The track record you see is likely to overestimate the probabilities. Recall the 1982 blowup of banks: the banks appeared to the naïve observer to be more profitable than they were. Look at reinsurance companies: according to the data, reinsurers have lost money on underwriting over the past couple of decades, but it actually could have been far worse because the past twenty years did not have a big catastrophe, and all you need is one catastrophic event per century to kiss the business good-bye.

The solution is to take the risks you know better more aggressively than others; to use skepticism to rank knowledge about risks. Epistemology can easily allow us to rank situations based on their robustness to consequential estimation error.

An active solution is also easy: we can either avoid taking a certain category of risks, because we do not understand them, or we can use financial contracts to cap our exposure to large losses from the rare event, some form of ‘black swan insurance’ – if available.

Footnotes:
(A) For the details, see Mandelbrot, 1997; Mandelbrot and Taleb, 2007; Taleb 2007.
(B) Taleb, 2007
(C) It does not concern learning from history. More formally, this can be generalized to any form of estimation of the unseen from a finite sample of seen.
(D) More technically, this comes from unimodal distributions in which  the probabilities decline monotonically with the absolute value of their impact.
(E) Taleb (2007) presents another problem with trying to understand catastrophes from past records: an application of the anthropic bias by which had a catastrophe occurred, we would not be here to discuss the point. This implies that a survivor is a poor assessor of the probabilities that led to his own survival.
(F) Mathematically, it is a function over a domain of “possible outcomes”, X, which assigns values to (some) subsets of X. It is not that different, essentially, than describing quantitatively other properties of a system (such as describing its mass by assigning it a numerical value of two kilograms).

References:
Mandelbrot, Benoît,1997, Fractals and Scaling in Finance: Discontinuity, Concentration, Risk. New York: Springer-Verlag.
Mandelbrot, Benoît, and Nassim Nicholas Taleb, 2007, “Mild vs. Wild Randomness: Focusing on Risks that Matter.” Forthcoming in Frank Diebold, Neil Doherty, and Richard Herring, eds., The Known, the Unknown and the Unknowable in Financial Institutions, Princeton, N.J.: Princeton University Press.
Taleb, Nassim Nicholas, 2007, The Black Swan: The Impact of the Highly Improbable, London: Penguin.
Von Neumann, John and Morgenstern, Oskar (1944), Theory of Games and Economic Behavior. Princeton: Princeton University Press.

Nassim Nicholas Taleb, is an applied statistician, veteran derivatives trader, and quantitative risk manager. He specializes in rare events and their impact across disciplines. He is the author of The Black Swan: The Impact of the Highly Improbable (2007) and Fooled by Randomness (2001).

Avital Pilpel is a post-doctorate researcher and lecturer at the University of Haifa’s philosophy department, specializing in the philosophy of science. He received his PhD at Columbia University, his dissertation being on the role of (rational) belief change in scientific explanations. His work deals mostly with the application of decision theory to medical, economic, and scientific situations.

^