If you could go back and work more on one project from your past, what would it be and why?

This immediately struck me as a good question. Like all really good interview questions, there is no right answer, but any answer tells a LOT about the person answering it. I talked about a few projects I had really enjoyed from my past: fuel hedging dashboard for an international airline, data mining government program data, but said that the one thing I wish I could work more on was reinsurance ceding strategies for insurance companies. Naturally he responded, “Why so?” So I explained the challenge and how I felt that if I had a little more time and a little more data I could numerically optimize reinsurance strategies and when I last worked on the problem it was 2001 and now, four years later, the computing power was better and I thought I could really get it right.

I’m pretty sure I didn’t explain very well. Mark was obviously fishing around to see if I got a little OCD about analytical challenges and if I loved digging. I thought about Mark’s question a lot three years later when I left Genworth to go work in reinsurance, optimizing reinsurance strategies.

André-Louis Cholesky is my homeboy

When I did a brief post three days ago I had no plans on writing two more posts on correlated random number generation. But I’ve gotten a couple of emails, a few comments, and some Twitter feedback. In response to my first post, Gappy, calls me out and says, “the way mensches do multivariate (log)normal variates is via Cholesky. It’s simple, instructive, and fast.”  And I think we’re all smart enough to read through Mr. Gappy’s comment and see that he’s saying I’m a complicated, opaque, and slow, גוי‎. My wife called and said his list would be more accurate if he added ‘emotionally detached.’ I have no idea what she means.

At any rate, in response to Gappy’s comment, here is the third verse (same as the first). The crux of the change is the following lines:

# shift the mean of ourData to zero
ourData0 <- as.data.frame(sweep(ourData,2,colMeans(ourData),"-"))

#Cholesky Decomposition of the covariance matrix
C <- chol(nearPD(cov(ourData0))$mat)

#create a matrix of random standard normals
Z <- matrix(rnorm(n * ncol(ourData)), ncol(ourData))

#multiply the standard normals by the transpose of the Cholesky
X <- t(C) %*% Z

myDraws <- data.frame(as.matrix(t(X)))
names(myDraws) <- names(ourData)

# we still need to shift the means of the samples.

# shift the mean of the draws over to match the starting data
myDraws <- as.data.frame(sweep(myDraws,2,colMeans(ourData),"+"))

Edit: When I first publishes this example, I didn’t shift the means prior to taking the cov(). I’ve sense corrected that.  Also thanks to @fdaapproved on Twitter who pointed out that I can replace the loop above with myDraws <- as.data.frame(sweep(t(X),2,colMeans(ourData),”+”))

This method, which uses Cholesky decomposition, is how I initially learned to create correlated random draws. I think this method is comparable to the mvrnorm() method. mvrnorm() is handy because it wraps everything above in one single line of code. But the above method is reliant only on the Matrix package and that’s only for the nearPD() function. If you are familiar with the guts of the mvrnorm() function and the chol() function, I’d love for you to comment on any technical differences. I looked briefly at the code for both and quickly realized my matrix math was rusty enough that it was going to take a while for me to sort through the code.

If you want the whole script you can find it embedded below and on Github.

?mvrnorm

and, I’ll be damned, he’s right! The advantage of using a copula is the ability to simulate correlation structures where the correlation is different for different levels of values. So that gives the flexibility to make the tails of the distributions more correlated, for example. But my example yesterday was purposefully simple… so simple that a copula was not even needed.

After creating my sample data all I really needed to do was this:

myDraws <- mvrnorm(1e5, mu=mean(ourData), Sigma=cov(ourData))

So I  took my example from yesterday and updated it using the mvrnorm() code and, as is my custom, put up a Github gist. The code is embedded below as well. I added a little ggplot2 code at the end that will create a facet plot of the 4 distributions showing the shape of the distributions of both the starting data and the simulated data. The plot in the upper left of this post is the ggplot output.

EDIT: The email hipping me to this was sent by Dirk Eddelbuettel who’s been very helpful to me more times than I can count. I had omitted his name initially. However after confirming with Dirk, he told me it was OK to mention him by name in this post.

I had some code where I used the QRMlib package, but it was really messy and fairly specific to my use case. So I whipped up very simple example of how to create correlated random draws from a multivariate distribution. In this example I used normally distributed marginals and Gaussian correlation to keep things simple and easy to follow. Rather than blogging through the code, I added a shit load (metric ass ton, if you’re in Canada) of comments. The code is designed to be stepped through. So don’t just run the whole blob and wonder what happened.

Walk through the code and if you find any errors be sure and let me know.

The code is embedded in a Github gist below, but if you are reading this in an aggregator (shout out to R-Bloggers) you’ll need to manually go to the gist.

"as we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns -- the ones we don't know we don't know." US Defense Secretary Donald Rumsfeld, February 12, 2002

I’ve been a long time reader of the blog “Messy Matters” (which invokes terrible images now that I am potty training a toddler). The authors, Sharad Goel and Daniel Reeves are academics who work in the Microeconomics and Social Systems (get it, MESS?!?) lab funded by Yahoo!. (What does Strunk and White say about punctuation after a proper noun which includes punctuation as part of the proper noun?) Anyhow, the Messy Matters blog had a very interesting post recently about testing to see if you are overconfident. The gist is this: take a test and try to not answer each question exactly but give an upper and lower bound which you think represents a 90% confidence band around the right answer. If you haven’t seen this done, you should go take and look and then read the rest of this blog post.

I didn’t do worth a shit on their “overconfidence” test. I think I got 5 of the ranges right. The other 5 times the real answer fell outside my bounds. As I was answering the questions I had this strong feeling of not being confident at all. I was very tempted to answer HUGE ranges on some of the questions because I felt totally unable to make a good guess. But I took a swag and tried to put in big ranges, but not TOO big, if I didn’t know the answer. I’m not the only one who struggled with this test. In their summary of results I fall in the 76th percentile. Hey, I’m above average… or at least above the median. Clearly I didn’t know how wrong I was in many cases. But does this mean I am “overconfident”? I don’t think so. I think this means something a bit more subtle. This exercise reminded me of creating a forecasting model and trying to predict values far outside the training data.

Having read the book On Intelligence I am convinced that one of the main functions of the human brain (or at least the prefrontal  cortex)  is to be a pattern matching machine. We all build little mental models in our head all the time. And these models are trained, by definition, on the situations which we run into day in and day out. And these models are VERY accurate around the mean (i.e. around the experiences we are used to having). For example, how small of a piece of sand can you feel between your teeth? Our brains have a ‘model’ of what it normally feels like when our teeth close against each other. The slightest unexpected disruption in that pattern triggers our brain to notice. Ever miss a step when walking down stairs? When did you know you were in trouble? Probably when your foot was about 2 inches past where you expected the next step to be. You didn’t have to wait for your face to hit the railing before your mental model of step walking was throwing warning bells. Us humans are freaking amazing mental model makers!

Well we’re amazing… except when we suck. When we suck is when we are faced with trying to predict something that is orders of magnitude outside our experience. The question on the MESS test which I struggled the most was the question about how much an empty 747 weighs. I don’t ever deal with massive weights. Ever. I only had two reference points which I could think up: 1) my first car was a ‘69 Cadillac which I know weighed 5,040 lbs. We used to call it “Two and a half tons of fun.” and 2) a hopper bottom rail car carries ~3500 bushels of corn which is ~ 196,000 lbs. And I’ve never been up next to a 747. But they are HUGE. I’ve seen pictures of the space shuttle riding around on the back of one of those bad boys. But they have to be pretty light relative to their volume because they have a lot of cargo room. And then I did the math on how many 1969 Cadillacs = 1 rail car of corn… almost 39!?!? But rail cars on not that big. I’ve climbed up on rail cars of grain. Kinda seems like it should be about 10 Cadillacs big. At that point I was pretty perplexed and just guessed a range which turned out to be WAAAY too high. It turns out that a 747 weighs around 360,000 lbs, which is less than 2 rail cars of corn (not including the actual cars, just the weight of the corn!). My intuition, as trained by my two data points, didn’t do worth a tinker’s damn at guessing the weight of airplanes.

But here’s the whole point of that last paragraph: If a human has no reference points and no experience with a domain, we (or at least me) can’t make good guesses and, more importantly, we can’t know how bad our guesses are!  We CAN’T know how much we suck! If you think in terms of distributions, this exercise is akin to having a very small sample size and trying to guess the distribution’s second moment (the standard deviation). Well shit, we know in practice that if we have small samples the mean has a big error term but the standard deviation has an even BIGGER error term.

So simply put, providing confidence bands around a guess which is out of my area of experience is really hard and I’m not good at it. The biggest problem is knowing when I’m out of my domain. In both The Black Swan and Fooled by Randomness, Nassim Nicholas Taleb points out that the single strongest predictor for how bad someone is going to do at the confidence band game is if they hold a PhD. If anyone has a reference on the study he refers to, I’d love to see it. I’m resisting the temptation to throw stones at both actuaries and finance quants right here. And if I didn’t live in a glass house, I would!

My take away from all this is that confidence bands around a guess should not be expected to be statistically accurate. That’s the very nature of not knowing something at all. We don’t even know what we don’t know (thank you Donald Rumsfeld). The very definition of an expert might be someone who, if they don’t know the exact answer, can at least put confidence bands around their guess. In other words, you have to have some level of knowledge to put accurate confidence bands around a guess. And failing to be able to do that is not necessarily overconfidence. It might just be ignorance.

Plymouth Rock... or something

There’s an interesting article in the WSJ this week about poultry farmers who grow for Pilgrim’s Pride (PPC). PPC is in bankruptcy and have canceled grower contracts with 400+ poultry growers. In poultry farming the farmers own the chicken houses and provide the labor but the poultry company (Pilgrim’s Pride, Tyson, Perdue, etc) own the birds and provide the feed. The producers get paid for every pound they put on the flock and they get extra bonuses for efficiency. If there is only one poultry company in the area, or if there are multiple companies but they are not taking new growers, then a grower is truly beholden to the poultry processing company. If the processing company cancels a grower’s contract then that grower can end up with hundreds of thousand’s of dollars of capital and/or debt that is going fallow.

One of the jarring annicdotes in the WSJ story is this one:

Last May, a tornado whipped through Center Ridge, population 1,332, demolishing two of the Dixons’ chicken houses. Their $370,000 insurance payment wasn’t enough to rebuild; they grappled with whether to quit.

Mr. Dixon says he received a visit from a Pilgrim’s representative who said, “Build them back as quick as you can and get ‘em rolling again.”

The Pilgrim’s spokesman says, “At the time of the May tornado, the company was in need of square footage for housing…But no one could have foreseen the dramatic changes that occurred in the U.S. chicken industry last summer.”

The Dixons tapped their savings to rebuild. On Aug. 1, a fresh batch of Pilgrim’s chicks took up residence. Ten days later, Pilgrim’s called to say those chicks would be the last.

There are two big business risks mentioned above. Both of these risks are difficult to deal with, but critical. The first is the risk the growers face if the poultry processing company goes bankrupt. The second is the reputational/legal risk faced by the processing company if they encourage growers to expand and then cancel their contract as happened to the Dixon’s.

The risk producers face if the poultry processor goes bankrupt is exactly the type of risk that a credit default swap (CDS) can hedge. Interestingly enough I have not heard a single mention of a poultry farmer or rural bank buying a CDS against a poultry processing company. WTF? Seems like JP Morgan would put their jeans and boots on and head out to the fly-over states and make some money. Too late now. You can’t shut the barn door once it’s on fire.

The other risk, legal/reputational, is much harder to manage. The risk is qualitative not quantitative, like credit risk. The WSJ article mentions a group of farmers who are suing PPC because the farmers think PPC made verbal promises that were not lived up to. I’m not a lawyer, I just sleep with one every night, but if the farmers can make a strong case that PPC was saying ‘go’ with one hand then ’stop’ with the other they might just have a case at which point I guess they line up behind all the other creditors in bankruptcy court. This is a hard risk to avoid and this type of mixed message issue is increasingly more common as a firm gets larger. My guess would be that the probability of a firm employee saying/doing something stupid goes up exponentially as the firm gets larger. At least PPC employees were not blogging about it… we hope.

Taleb is a bit harder to like than Mandelbrot. He’s mouthy, arrogant, and demands that his publisher not edit his books. If he crosses your ideology you will find yourself wanting to write him a pissy letter starting with the greeting, “Look you arrogant ass…” But he also has some really great intuition about risk. Most folks get religion after they have a near death experience. Taleb got his religion after he got rich. NNT was holding a double butt load (that’s a technical term, go look it up) of calls when the financial world went to hell in 1987. NNT knew that the reason he got rich was not because he was smarter or had more foresight than anyone else. He did know one thing, there is no way to know what the probability of a huge price swing is. So he bought out of the money options in both directions since he suspected that the probability of these options paying off BIG was not properly accounted for in the price. It seems he may have been right. He’s also waged a war against value at risk or VAR which I don’t fully agree with, but C’est la vie.

Below is a video segment from PBS News Hour where they interview NNT and Dr. Mandelbrot (notice how I did not use his initials? gotta show him some respect). It’s a good interview, but it cracks me up that the host clearly has Mandelbrot’s ideas confused with Edward Lorenz, the man who coined the phrase ‘Butterfly Effect.’ See 4:55 in the video for the most blatant confusion. Mandelbrot does a good job of ignoring the confusion but eventually at 6:20ish he starts to redirect the host to his true thought contribution which is that big movements are under-anticipated. To the host’s credit Mandelbrot and Lorenz have co-authored papers so it’s not like they are totally disconnected.

I can’t mention Mandelbrot without posting this great tribute song/video made in honor of his namesake Mandelbrot Set.

Shameless commerce (yeah, I stole that from those Car Talk Guys) related to this post:

If you are an option trader or would like to learn how options markets really work, try NNT’s first book: