This blog's name in Chinese!

I just came back from the future and let me be the first to tell you this: Learn some Chinese. And more than just cào nǐ niáng (肏你娘) which your friend in grad school told you means “Live happy with many blessings”. Trust me, I’ve been hanging with Madam Wu and she told me it doesn’t mean that.

So how did I travel to the future to visit with Madam Wu, you ask? Well the short answer is Hadoop. Yeah, the cute little elephant. As I have told you before, multicore makes your R code run fast by using worm holes to shoot your results back from the future. Well Hadoop actually takes you to the future on the back of an elephant and you can bring your own results back! I couldn’t make this up if I tried, so you know it’s true! And what’s fantastic about all of this is Hadoop works with R! And Amazon will let you rent a time traveling elephant through their Elastic MapReduce service! I think Amazon coined the term “Time Travel as a Service” or TTaaS  generally pronounced as “ta-tas” in the industry. If you are a CTO be sure and use this in your next “vision statement” pitch so everyone will know you’re hip to all this cloud stuff.

So you use R and you want to travel into the future on the back of an elephant to visit Madam Wu and get your model results back, don’t you? Well it’s a damn good thing you read this blog because I’m going to give you the keys to the Wu dynasty and a little 福寿 while we’re at it.

I’ve never had an original thought in my life so I started with this discussion over at the AMZN E M/R discussion forum. Peter Skomoroch from Data Wrangling gives a very good example with all the data and code provided so you can run it yourself.  Pete’s example really shakes the yáng guǐzi, as we say in the future. In addition I read the documentation for David Rosenberg’s HadoopStreaming package which was good for insight, but I didn’t use the package as it’s really focused on the ‘big data’ problem.

That elephant is so freaking cute!

Prior to my foray into time travel, I knew that Hadoop could be used to process big text files and do something like rip out all the links and count them. But I thought that Hadoop was all about processing big data. I never paid attention to the big Hadoop elephant in the room because I don’t have big data. I have big CPU hogging models (mostly slow because I don’t code worth a shit). What got me reconsidering my world view was John Myles White’s comment on my multicore post earlier. John encouraged me to look into running my simulations on AMZN’s E M/R service using Hadoop streaming. So instead of giving Hadoop  a big fat text file to parse, I just gave it a text file with 10,000 rows each containing an integer from 1:10,000. Then I refactored my R code to read a line from stdin, trim it down to just the integer, and then go run the simulation with that number. When done I had it serialize the resulting model output and return that to stdout. Hadoop takes care of chopping up the input and pulling together the output.

I learned a few “gotchas” or, as we say in the future: 臭婊子(I think that should be plural). I’ll do a whole blog post on gotchas soon, but here’s the bullet points:

• AMZN is currently running the version of Debian Linux named Lenny which has version 2.7.1 of R installed. No matter what the documentation says, don’t let Lenny tend to the rabbits.
• Test all code by firing up an interactive Pig instance and logging in as ‘hadoop’. Instead of running Pig, run R and test your code. And as it says in the FAQ: “The Pig don’t care either way. ” Which, despite sounding like buggery, is the truth.
• If your code runs inside of R on a Hadoop instance, drop back to the command line on the Hadoop instance and run ‘cat infile.txt | yourMapper.R | sort | yourReducer.R > outfile.txt’. This pipes your input file into your mapper file which does it’s thing and then pipes the results to your reducer file which then “pumps up the jam” into an output file.  What you see in the outfile.txt is what Hadoop will produce. So it you don’t like what you see, you better do some more coding.
• You CAN load packages into R in a Hadoop instance running in AMZN E M/R. There are a few caveats, of course:

1. Your package has to work in R 2.7.1. (until AMZN upgrades to the next stable version of Debian.
2. As far as I can tell, all the output has to come out of stdout. So if you want to end up with R objects which you use for other things, you should get comfortable with the serialize() command and reading text files back into R. Which, as you can see from this question, I am not yet comfortable with.
3. There will be multiple instances of R running on every machine. So if they are all trying to download a package to the same directory, you are going to get file lock errors. One solution is to have each R instance create a directory for packages that includes the PID of the R instances. That way there’s no possibility for a conflict! Here’s an example of how I load the Hmisc package:

• You’ll probably want to provide some data to R. This is done by uploading your files to S3 and then passing the “-cacheFile” option to Hadoop. To get the plyr package to load in R 2.7.1 I had to edit the package. I then uploaded the altered package thusly:

-cacheFile s3n://rdata/plyr_0.1.9.tar.gz#plyr_0.1.9.tar.gz

More to come later. I’ve gotta get back to the future.

You hold the elephant and I'll plug this in.

When running processor intensive crap (that’s a stochastic modeling term) the single threaded nature of R is painful. In Linux or Mac (i.e. NOT Windows) the multicore package is a real godsend. I did a quick code review and, from what I can tell, multicore exploits worm holes to travel back in time and reports your results in a fraction of the time you would expect it to take. Seriously. I expect that as the code matures my computer will fill up with simulation results from simulations which I have not even coded yet. It’s almost like magic, except without the rabbit and hat.

The crux of the package is a parallel-ized version of lapply() called mclapply(). I believe the mc stands for ‘magic carpet’ and is an allusion to the worm hole technology. So how does one harness this package for nefarious self interest doing parallel operations in R? The ultra short answer is: write your R code so that the most processor intensive bit is done with an lapply() function. Then replace the lapply() with mclapply().  Of course you have to load the multicore package before you run it. But that’s basically it.

How I implement mcapply() is thusly: I build a table with all my random draws for my simulations. So if I have 20 variables and want to run 10,000 simulations then I’ll build a data frame with all 200,000 values (generally 10K rows and 21 columns for 20 variables + and index). The index keeps track of the draw number. Then I have code that performs the ‘valuation’ based on a single observation of the 20 variables. I wrap the valuation step in a function and then call the valuation process 10,000 times with mclapply(). So it might look something like this:

myOutput <- mclapply( drawList, function(x) valuationReturns(drawNumber=x))

The drawList object is simply a list of the possible indexes (i.e. 1:10000). When the code has iterated over each value from drawList the results will be in the myOutput object. Tada!

I recommend the htop program for tracking what’s going on with processor utilization in Linux (I presume Mac too if you ask Steve Jobs nicely). If everything is cranking well, and you have 8 cores, you might see an image that looks something like this:

I don’t understand time travel, but I’ve found that I have better luck if I set mc.preschedule=FALSE. Apparently prescheduled magic carpets are finicky. If I leave mc.preschedule to the default of TRUE then I find that often some of my cores go underutilized.

Let me know if you have other multicore tips and tricks.

If you want to give me shit for running my simulations as root, feel free. I’m impervious to your “best practices” mumbo jumbo. La la la la la la!! Not listening!

Special thanks to John Cavazos over at the University of Delaware from whom I stole the MC for Dummies image. John, your a gentleman and a humble scholar. Damn few of us left.

Pretty Normal

Dave, over at The Revolutions Blog, posted about the big ‘ol list of graphs created with R that are over at Wikimedia Commons. As I was scrolling through the list I recognized the standard normal distribution from the Wikipedia article on the same topic.

Below is the fairly simple source code with lots of comments. Here’s the source. Run it at home… for fun and profit.

# # External package to generate four shades of blue
# library(RColorBrewer)
# cols <- rev(brewer.pal(4, "Blues"))
cols <- c("#2171B5", "#6BAED6", "#BDD7E7", "#EFF3FF")

# Sequence between -4 and 4 with 0.1 steps
x <- seq(-4, 4, 0.1)

# Plot an empty chart with tight axis boundaries, and axis lines on bottom and left
plot(x, type="n", xaxs="i", yaxs="i", xlim=c(-4, 4), ylim=c(0, 0.4),
     bty="l", xaxt="n", xlab="x-value", ylab="probability density")

# Function to plot each coloured portion of the curve, between "a" and "b" as a
# polygon; the function "dnorm" is the normal probability density function
polysection <- function(a, b, col, n=11){
    dx <- seq(a, b, length.out=n)
    polygon(c(a, dx, b), c(0, dnorm(dx), 0), col=col, border=NA)
    # draw a white vertical line on "inside" side to separate each section
    segments(a, 0, a, dnorm(a), col="white")
}

# Build the four left and right portions of this bell curve
for(i in 0:3){
    polysection(   i, i+1,  col=cols[i+1]) # Right side of 0
    polysection(-i-1,  -i,  col=cols[i+1]) # Left right of 0
}

# Black outline of bell curve
lines(x, dnorm(x))

# Bottom axis values, where sigma represents standard deviation and mu is the mean
axis(1, at=-3:3, labels=expression(-3*sigma, -2*sigma, -1*sigma, mu,
                                    1*sigma,  2*sigma,  3*sigma))

# Add percent densities to each division, between x and x+1
pd <- sprintf("%.1f%%", 100*(pnorm(1:4) - pnorm(0:3)))
text(c((0:3)+0.5,(0:-3)-0.5), c(0.16, 0.05, 0.04, 0.02), pd, col=c("white","white","black","black"))
segments(c(-2.5, -3.5, 2.5, 3.5), dnorm(c(2.5, 3.5)), c(-2.5, -3.5, 2.5, 3.5), c(0.03, 0.01))

There’s some correlation as well as causation. I could be wrong, but I suspect that the reason Gelman does not like the XKCD cartoon is because he’s very literal, as geeks can be. Trust me, my wife is married to a geek. It probably also has something to do with how much Gelman hates box plots. He hates them so much that he is holding a contest to see if anyone can prove to him that a box plot is an appropriate way to show something. I don’t think I can persuade him that box plots are ‘appropriate’ as that sounds like a matter of taste… like when I break wind at the breakfast table and my wife says, ‘that’s not appropriate.’ However I can demonstrate the ease at which one can make box plots, and my preferred violin plots using R.

So stick with me and I’ll give you some free code to take home and try!

Here’s the type of thing you see a lot with box plots:

x <- rnorm(200)
y <- rlnorm(200)
plot(x, y, xlim=c(-5,5), ylim=c(-2,8))
boxplot(x, col=”gold”, horizontal=TRUE, at=-1, add=TRUE,lty=2, rectCol=”gray”)
boxplot(y, col=”blue”, horizontal=FALSE, at=-4, add=TRUE,lty=2)

That produces output like this:

Box Plot

That’s kinda cute. You can see the log normal shape of y and the normal shape of x.  An alternative would be to use a violin chart using the following syntax:

library(vioplot)
plot(x, y, xlim=c(-5,5), ylim=c(-2,8))
vioplot(x, col=”gold”, horizontal=TRUE, at=-1, add=TRUE,lty=2, rectCol=”gray”)
vioplot(y, col=”blue”, horizontal=FALSE, at=-4, add=TRUE,lty=2)

I kept the values for X and Y the same, but the new plot looks like this:

Violin Plot

I like that a little better. The violin plot captures the shape of the density mass function (PDF). But in both of these examples we would probably be just as well off if we simply plotted the PDF instead of either the violin plot or the box plot. So they aren’t really adding anything. So is Gelman right, the box/violin plot is useless? Here’s what I think it IS good for:

These are plots of state crop yields in terms of deviation from an expected trend. So 0 on the Y axis means no deviation from trend and 2 is 200% better than trend and you just can’t do any worse than -100% below trend. I look at this type of stuff all the time, and a box or violin chart is really nice because I can lay out a bunch of states along the X axis and look at how they compare. It’s easy to compare all the moments of the distributions visually. We can easily see that most states have a mean around 0, but Kansas has MUCH more dispersion as well as a lot of skew. And since you are wondering, no, I didn’t force the symmetry you see in the graph, it just turned out that way. Luck put Kansas in the middle and luck gave MO and IL the same relative tail. *shrug* sometimes this stuff just looks good. Kinda like me!

Here’s the same info as above but in a box plot:

I think the violin plot is more elegant, but the box plot is still OK, albeit rather noisy and less elegant.

So that’s all the free code you get from me today. Try not to spend it all on candy this time, OK?