"as we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns -- the ones we don't know we don't know." US Defense Secretary Donald Rumsfeld, February 12, 2002

I’ve been a long time reader of the blog “Messy Matters” (which invokes terrible images now that I am potty training a toddler). The authors, Sharad Goel and Daniel Reeves are academics who work in the Microeconomics and Social Systems (get it, MESS?!?) lab funded by Yahoo!. (What does Strunk and White say about punctuation after a proper noun which includes punctuation as part of the proper noun?) Anyhow, the Messy Matters blog had a very interesting post recently about testing to see if you are overconfident. The gist is this: take a test and try to not answer each question exactly but give an upper and lower bound which you think represents a 90% confidence band around the right answer. If you haven’t seen this done, you should go take and look and then read the rest of this blog post.

I didn’t do worth a shit on their “overconfidence” test. I think I got 5 of the ranges right. The other 5 times the real answer fell outside my bounds. As I was answering the questions I had this strong feeling of not being confident at all. I was very tempted to answer HUGE ranges on some of the questions because I felt totally unable to make a good guess. But I took a swag and tried to put in big ranges, but not TOO big, if I didn’t know the answer. I’m not the only one who struggled with this test. In their summary of results I fall in the 76th percentile. Hey, I’m above average… or at least above the median. Clearly I didn’t know how wrong I was in many cases. But does this mean I am “overconfident”? I don’t think so. I think this means something a bit more subtle. This exercise reminded me of creating a forecasting model and trying to predict values far outside the training data.

Having read the book On Intelligence I am convinced that one of the main functions of the human brain (or at least the prefrontal  cortex)  is to be a pattern matching machine. We all build little mental models in our head all the time. And these models are trained, by definition, on the situations which we run into day in and day out. And these models are VERY accurate around the mean (i.e. around the experiences we are used to having). For example, how small of a piece of sand can you feel between your teeth? Our brains have a ‘model’ of what it normally feels like when our teeth close against each other. The slightest unexpected disruption in that pattern triggers our brain to notice. Ever miss a step when walking down stairs? When did you know you were in trouble? Probably when your foot was about 2 inches past where you expected the next step to be. You didn’t have to wait for your face to hit the railing before your mental model of step walking was throwing warning bells. Us humans are freaking amazing mental model makers!

Well we’re amazing… except when we suck. When we suck is when we are faced with trying to predict something that is orders of magnitude outside our experience. The question on the MESS test which I struggled the most was the question about how much an empty 747 weighs. I don’t ever deal with massive weights. Ever. I only had two reference points which I could think up: 1) my first car was a ‘69 Cadillac which I know weighed 5,040 lbs. We used to call it “Two and a half tons of fun.” and 2) a hopper bottom rail car carries ~3500 bushels of corn which is ~ 196,000 lbs. And I’ve never been up next to a 747. But they are HUGE. I’ve seen pictures of the space shuttle riding around on the back of one of those bad boys. But they have to be pretty light relative to their volume because they have a lot of cargo room. And then I did the math on how many 1969 Cadillacs = 1 rail car of corn… almost 39!?!? But rail cars on not that big. I’ve climbed up on rail cars of grain. Kinda seems like it should be about 10 Cadillacs big. At that point I was pretty perplexed and just guessed a range which turned out to be WAAAY too high. It turns out that a 747 weighs around 360,000 lbs, which is less than 2 rail cars of corn (not including the actual cars, just the weight of the corn!). My intuition, as trained by my two data points, didn’t do worth a tinker’s damn at guessing the weight of airplanes.

But here’s the whole point of that last paragraph: If a human has no reference points and no experience with a domain, we (or at least me) can’t make good guesses and, more importantly, we can’t know how bad our guesses are!  We CAN’T know how much we suck! If you think in terms of distributions, this exercise is akin to having a very small sample size and trying to guess the distribution’s second moment (the standard deviation). Well shit, we know in practice that if we have small samples the mean has a big error term but the standard deviation has an even BIGGER error term.

So simply put, providing confidence bands around a guess which is out of my area of experience is really hard and I’m not good at it. The biggest problem is knowing when I’m out of my domain. In both The Black Swan and Fooled by Randomness, Nassim Nicholas Taleb points out that the single strongest predictor for how bad someone is going to do at the confidence band game is if they hold a PhD. If anyone has a reference on the study he refers to, I’d love to see it. I’m resisting the temptation to throw stones at both actuaries and finance quants right here. And if I didn’t live in a glass house, I would!

My take away from all this is that confidence bands around a guess should not be expected to be statistically accurate. That’s the very nature of not knowing something at all. We don’t even know what we don’t know (thank you Donald Rumsfeld). The very definition of an expert might be someone who, if they don’t know the exact answer, can at least put confidence bands around their guess. In other words, you have to have some level of knowledge to put accurate confidence bands around a guess. And failing to be able to do that is not necessarily overconfidence. It might just be ignorance.

Plymouth Rock... or something

There’s an interesting article in the WSJ this week about poultry farmers who grow for Pilgrim’s Pride (PPC). PPC is in bankruptcy and have canceled grower contracts with 400+ poultry growers. In poultry farming the farmers own the chicken houses and provide the labor but the poultry company (Pilgrim’s Pride, Tyson, Perdue, etc) own the birds and provide the feed. The producers get paid for every pound they put on the flock and they get extra bonuses for efficiency. If there is only one poultry company in the area, or if there are multiple companies but they are not taking new growers, then a grower is truly beholden to the poultry processing company. If the processing company cancels a grower’s contract then that grower can end up with hundreds of thousand’s of dollars of capital and/or debt that is going fallow.

One of the jarring annicdotes in the WSJ story is this one:

Last May, a tornado whipped through Center Ridge, population 1,332, demolishing two of the Dixons’ chicken houses. Their $370,000 insurance payment wasn’t enough to rebuild; they grappled with whether to quit.

Mr. Dixon says he received a visit from a Pilgrim’s representative who said, “Build them back as quick as you can and get ‘em rolling again.”

The Pilgrim’s spokesman says, “At the time of the May tornado, the company was in need of square footage for housing…But no one could have foreseen the dramatic changes that occurred in the U.S. chicken industry last summer.”

The Dixons tapped their savings to rebuild. On Aug. 1, a fresh batch of Pilgrim’s chicks took up residence. Ten days later, Pilgrim’s called to say those chicks would be the last.

There are two big business risks mentioned above. Both of these risks are difficult to deal with, but critical. The first is the risk the growers face if the poultry processing company goes bankrupt. The second is the reputational/legal risk faced by the processing company if they encourage growers to expand and then cancel their contract as happened to the Dixon’s.

The risk producers face if the poultry processor goes bankrupt is exactly the type of risk that a credit default swap (CDS) can hedge. Interestingly enough I have not heard a single mention of a poultry farmer or rural bank buying a CDS against a poultry processing company. WTF? Seems like JP Morgan would put their jeans and boots on and head out to the fly-over states and make some money. Too late now. You can’t shut the barn door once it’s on fire.

The other risk, legal/reputational, is much harder to manage. The risk is qualitative not quantitative, like credit risk. The WSJ article mentions a group of farmers who are suing PPC because the farmers think PPC made verbal promises that were not lived up to. I’m not a lawyer, I just sleep with one every night, but if the farmers can make a strong case that PPC was saying ‘go’ with one hand then ’stop’ with the other they might just have a case at which point I guess they line up behind all the other creditors in bankruptcy court. This is a hard risk to avoid and this type of mixed message issue is increasingly more common as a firm gets larger. My guess would be that the probability of a firm employee saying/doing something stupid goes up exponentially as the firm gets larger. At least PPC employees were not blogging about it… we hope.

Taleb is a bit harder to like than Mandelbrot. He’s mouthy, arrogant, and demands that his publisher not edit his books. If he crosses your ideology you will find yourself wanting to write him a pissy letter starting with the greeting, “Look you arrogant ass…” But he also has some really great intuition about risk. Most folks get religion after they have a near death experience. Taleb got his religion after he got rich. NNT was holding a double butt load (that’s a technical term, go look it up) of calls when the financial world went to hell in 1987. NNT knew that the reason he got rich was not because he was smarter or had more foresight than anyone else. He did know one thing, there is no way to know what the probability of a huge price swing is. So he bought out of the money options in both directions since he suspected that the probability of these options paying off BIG was not properly accounted for in the price. It seems he may have been right. He’s also waged a war against value at risk or VAR which I don’t fully agree with, but C’est la vie.

Below is a video segment from PBS News Hour where they interview NNT and Dr. Mandelbrot (notice how I did not use his initials? gotta show him some respect). It’s a good interview, but it cracks me up that the host clearly has Mandelbrot’s ideas confused with Edward Lorenz, the man who coined the phrase ‘Butterfly Effect.’ See 4:55 in the video for the most blatant confusion. Mandelbrot does a good job of ignoring the confusion but eventually at 6:20ish he starts to redirect the host to his true thought contribution which is that big movements are under-anticipated. To the host’s credit Mandelbrot and Lorenz have co-authored papers so it’s not like they are totally disconnected.

I can’t mention Mandelbrot without posting this great tribute song/video made in honor of his namesake Mandelbrot Set.

Shameless commerce (yeah, I stole that from those Car Talk Guys) related to this post:

If you are an option trader or would like to learn how options markets really work, try NNT’s first book: