set.seed(3)
x <- rgamma(1e5, 2, .2)
plot(density(x))
 
# normalize the gamma so it's between 0 & 1
# .0001 added because having exactly 1 causes fail
xt <- x / ( max( x ) + .0001 )
 
# fit a beta distribution to xt
library( MASS )
fit.beta <- fitdistr( xt, "beta", start = list( shape1=2, shape2=5 ) )
 
x.beta <- rbeta(1e5,fit.beta$estimate[[1]],fit.beta$estimate[[2]])
 
## plot the pdfs on top of each other
plot(density(xt))
lines(density(x.beta), col="red" )
 
## plot the qqplots
qqplot(xt, x.beta)

Created by Pretty R at inside-R.org

It’s not illustrated above, but it’s probably useful to transform the simulated data (x.beta) back into pre normalized space by multiplying by max( x ) + .0001 . (I swore I’d never say this but I lied) I’ll leave that as an exercise for the reader.

Another very useful tool in building a mental road map of distributions is the graphical chart of distribution relationships that John Cook introduced me to.

So here’s a short shell script to spin up an instance, wait 30 seconds, then connect:

If you’re reading this through an RSS reader, you can see the script over at github.

Obviously you’ll need to change the parameters at the top of the script to suit your needs. But since this was a bit of a pain in the donkey hole for me to figure out, I thought I would share.

If you want to help out, I’d love you to enlighten me on how to have the script figure out if an instance has finished booting so I could eliminate the sleep step.

If you could go back and work more on one project from your past, what would it be and why?

This immediately struck me as a good question. Like all really good interview questions, there is no right answer, but any answer tells a LOT about the person answering it. I talked about a few projects I had really enjoyed from my past: fuel hedging dashboard for an international airline, data mining government program data, but said that the one thing I wish I could work more on was reinsurance ceding strategies for insurance companies. Naturally he responded, “Why so?” So I explained the challenge and how I felt that if I had a little more time and a little more data I could numerically optimize reinsurance strategies and when I last worked on the problem it was 2001 and now, four years later, the computing power was better and I thought I could really get it right.

I’m pretty sure I didn’t explain very well. Mark was obviously fishing around to see if I got a little OCD about analytical challenges and if I loved digging. I thought about Mark’s question a lot three years later when I left Genworth to go work in reinsurance, optimizing reinsurance strategies.

I initially tried running Unison on BOTH my laptop and the server and had the server Unison set to sync with my laptop back through an SSH reverse proxy. After testing this for a while I discovered this is totally the wrong way to do it. The problem is that the Unison process makes temp directories and files in the file system of the target. So my Unison job on the laptop would be trying to syn files and, in the process, create temp files which would kick off a Unison sync on the sever which would make temp files on the laptop… I think you can see how convoluted this gets.

So a much better solution is to only run Unison from one machine (I chose my laptop) and have the other machine (server in my case) send an SSH command (over the aforementioned reverse proxy) to the laptop asking the laptop to kick off a Unison sync. This way all of the syncs happen from the laptop.

So, in short, both machines run lsyncd which monitors files for changes. I keep up an SSH tunnel with reverse port forwarding which forwards a remote machine port back to my laptop’s port 22 (SSH). Unison need be installed ONLY on my laptop. When a change happens on my laptop, lsyncd fires off a Unison sync from my laptop that syncs it with the server. When a file changes on the server, the lsyncd job on the server makes a connection to my laptop via ssh and fires off a Unsion sync between my laptop and the server.

Here’s an example of my lsyncd config scripts:

Laptop:

settings = {
logfile = “/home/jal/lsyncd/laptop/lsyncd.log”,
statusFile = “/home/jal/lsyncd/laptop/lsyncd.status”,
maxDelays = 15,
–nodaemon = true,
}

runUnison2 = {
maxProcesses = 1,
delay = 15,
onAttrib = “/usr/bin/unison -batch /home/jal/Documents ssh://12.34.56.78//home/jal/Documents”,
onCreate = “/usr/bin/unison -batch /home/jal/Documents ssh://12.34.56.78//home/jal/Documents”,
onDelete = “/usr/bin/unison -batch /home/jal/Documents ssh://12.34.56.78//home/jal/Documents”,
onModify = “/usr/bin/unison -batch /home/jal/Documents ssh://12.34.56.78//home/jal/Documents”,
onMove = “/usr/bin/unison -batch /home/jal/Documents ssh://12.34.56.78//home/jal/Documents”,
}

sync{runUnison2, source=”/home/jal/Documents”}

Server:

settings = {
logfile = “/home/jal/lsyncd/server/lsyncd.log”,
statusFile = “/home/jal/lsyncd/server/lsyncd.status”,
maxDelays = 15,
–nodaemon = true,
}

runUnison2 = {
maxProcesses = 1,
delay = 15,
onAttrib = “ssh localhost -p 5432 unison -batch /home/jal/Documents ssh://12.34.56.78//home/jal/Documents”,
onCreate = “ssh localhost -p 5432 unison -batch /home/jal/Documents ssh://12.34.56.78//home/jal/Documents”,
onDelete = “ssh localhost -p 5432 unison -batch /home/jal/Documents ssh://12.34.56.78//home/jal/Documents”,
onModify = “ssh localhost -p 5432 unison -batch /home/jal/Documents ssh://12.34.56.78//home/jal/Documents”,
onMove = “ssh localhost -p 5432 unison -batch /home/jal/Documents ssh://12.34.56.78//home/jal/Documents”,
}

sync{runUnison2, source=”/home/jal/Documents”}

Keep in mind that I am using version 2 of lsyncd which can be downloaded here: http://code.google.com/p/lsyncd/

The version of lsyncd available in the Ubuntu repo is version 1.x which does not use the same config format as I illustrate above. However, if you run into dependency issues with v2, the easiest thing to do is install the repo version which will install dependencies and then manually download and install v2 from the above URL.

My reverse port forwarding set up looks like this:

autossh -2 -4 -X -R 5432:localhost:22 12.34.56.78

the -R bit forwards remote port 5432 to my laptop’s port 22 which is the ssh. So on my server if I run ssh localhost -p 5432 what actually happens is I am sshing from the remote machine to my laptop.

Notes:

• The IP address of my server in this example is 12.34.56.78.
• Don’t try and sync the directories where the lsyncd logs are kept. That will results in an endless sync cycle as each machine keeps noticing changes endlessly. Don’t ask me how I know this.
• The command to start the sync on the laptop is “lsyncd /home/jal/lsyncd/laptop/configfile” where configfile is the above lsyncd configuration file.
• lsyncd could, conceivably, tell Unison to sync only the part of the directory tree that changed. I have not been able to make that feature work right, however. And it only takes Unison a few seconds to sync, so I’ve not worried about it.

This has greatly sped up my RStudio based workflow when doing analysis with R. Now when I change files on my server using RStudio they are immediately (well it waits 15 seconds) replicated to my local machine and vice versa!

Good luck and if you have any suggestions please post a comment!

I get around this conundrum by running much of my analytics on either my work server or on an EC2 machine (I’m going to call these collectively “my servers” for the rest of this post). The nagging problem with this has been keeping files in sync. RStudio Server has been a great help to my workflow because it lets me edit files in my browser and they run on my servers. But when a long running R job blows out files I want those IMMEDIATELY synced with my laptop. That way I know when I undock my laptop to run to the train station that all my files will be there for me to spill Old Style beer on as I ride the Metra North line.

I experimented with Dropbox and I gotta say, it’s great. It really is well engineered, fast, and drop dead simple. I love that with Dropbox I could pull up most any file from my Dropbox on my iPad or iPhone. That’s a very handy feature. And it’s fast. If I created a small text file on my server, it would be synced with my laptop in a few seconds. Perfect! Wel… almost. Dropbox has a huge limitation: encryption. Dropbox encrypts for transmission and may even store files encrypted on their end. However, Dropbox controls the key. So if a rogue employee, a crafty Russian hacker, or a law enforcement officer with a subpoena gained access to Dropbox, they could get access to my files without my knowledge. As a risk manager I can’t help but see Dropbox’s security as a huge, targeted, single point of failure. It’s hard to say which would be a bigger payday: cracking GMail, or cracking Dropbox. But I’m suspicious it’s Dropbox. There are some workarounds to try and shoehorn file encryption into Dropbox, and they all suck.

So Dropbox can’t really give me what I want (what I really really want). But I stumbled into Spideroak who are like the smarter, but lesser known cousins of Dropbox. Their software does everything Dropbox does (including tracking all revisions!) but they have a “trust no one” model which encrypts all files before leaving my computer using, and this is critical, MY key which they don’t store. Pretty cool, eh? Spideroak also has a iPad/iPhone app and offers a neat feature that allows emailing any file in my Spideroak “bucket” to anyone using my iPhone without having to upload the file to my iPhone first. They do this by sending a special link to the email recipient that allows them to open only the file you wanted them to have. This could be a huge bacon saver on the road.

So Spideroak’s the panacea then? Well… um… no. They have two critical flaws: 1) They depend on time stamps on files to determine most recent file. 2) Syncs are slow, sometimes taking more than 5 minutes for very small files. The time stamp issue is an engineering failure, plain and simple. I’ve talked to their tech support and been assured that they are going to change this and index using server time, not system time in the future. But as of April 6, 2011, Spideroak uses local system time. For most users this is no big deal. For my use case this is painful. My server and my laptop were 6 seconds different and that time difference was enough for me to get Spideroak confused about which files were the freshest. This is a big deal when syncing two file systems with fast changing files. The other issue, slow sync, was actually more painful but probably the result of their attempt to be nice with CPU time and also encryption. When jobs on my server finished, I expected those files to start syncing within seconds and the only delay I expected was bandwidth constraints. With Spideroak syncs might take 5 minutes to start and then it would go out for coffee, come back jittery and then finally complete. Even if SPideroak fixed the time sync issue (or I forced my laptop to set its time based on my server), it still would not work for my sync because of the huge lags.

So looking at Dropbox and Spideroak I realized that I liked everything about Spideroak except its sync. It’s a great cloud backup tool that seems to properly do encryption, it’s multiplatform (win, linux, mac), has an iPad/iPhone app for viewing/sending files, it’s smart about backups and won’t upload the same file twice (even if the file is on two different computers). For my business use, I just can’t use Dropbox. The lack of “trust no one” encryption is a deal killer. So what I really need is a sync solution to use along side Spideroak.

There are some neat projects out there for sync. Projects like Sparkleshare look really promising but they are trying to do all sorts of things, not just sync. I’ve already settled on letting Spideroak do backup and version tracking so I don’t really need all those features… OK, OK, I can hear you muttering, “just use rsync and be done with it already.” Yeah, that’s a good idea. But rsync is single directional and does a lot of things well, but can also be a bit of an asshole if you don’t set all the flags right and rub its belly the right way. If you google for “bidirectional sync” you’re going to see this problem has plagued a lot of folks. This blog post has already gone on long enough so I’ll cut to the chase. Here’s the stack of tools I settled on for cobbling together my own secure, real-time, bidirectional sync between two Ubuntu boxes (one of which changes IP address and is often behind a NAT router):

1) Unison – Fast sync using rsync-esque algos and really fast caching/scanning

2) lsyncd – Live (real-time) sync daemon

3) autossh – ssh client with a nifty wrapper that keeps the connection alive and respawns the connection if dropped

I’ll do another post with the nitty-gritty of how I set this up, but the short version is that I installed Unison and lsyncd on both the laptop and the server. Single direction sync from my laptop to the server is pretty straight forward: lsyncd watches files, if one changes it calls unison which syncs the files with the server. The tricky bit was getting my server to be able to sync with my laptop which is often behind a NAT router. The solution was to open an ssh connection from my laptop to my server using autossh and reverse port forward port 5555 from the server back to my laptop’s port 22. That way an lsyncd process on the server can monitor the file system and when it sees a change can kick off a unison job that syncs the server to ssh://localhost:5555//some/path which is forwarded to my laptop! Autossh makes sure that connection does not get dropped and respawns if it does get dropped. So with a little shell scripting to start the lsyncd daemon on both machines, some config of lsyncd, and a local shell script to fire off the autossh connection, I’ve got real-time bidirectional sync!

In a follow up post I’ll put of the details of this configuration. Stay tuned. (EDIT: Update posted!)

If you’ve solved sync a different way and you like your solution, please comment. I’ve not settled that this is my long-term solution. It’s just a solution that works. Which is more than I had yesterday.

And you may have heard that the R Cookbook by Chicago’s own Paul Teeter has been printed! Way to go Paul! And for a limited time you can get the book 50% off direct from O’Reilly.

And let it be known: I’ve double dog dared you to find a stats or programming book with any better back cover quotes:

So I was looking for more simple/elegant solutions. After thinking the Boto library for Python might be helpful, I realized that the easiest way to use that would be with rJython which meant having to interact with R, Python, AND Java. Considering I don’t program in Python or Java, that seemed like a fair bit of complexity. Then I realized that the canonical implementation of the AWS API was the AWS Java SDK. The rJava package makes interacting with Java from R a viable option.

Since I’ve never written a single line of Java code in my pathetic life, this was somewhat harder than it could have been. But with some help from Romain Francois I was able to cobble together “something that works.” The code below gives a simple example of interfacing with S3. The example will look to see if a given bucket exists on S3, if not it will create the bucket. Then it will upload a single file from your PC into that bucket. You will have to download the SDK, unzip it in the location of your choice, and then change the script to reflect your configuration.

If you are running R in Ubuntu, you should install rJava using apt-get instead of using install.packages() from inside of R:

sudo apt-get install r-cran-rjava

Here’s the codez. And a direct link for you guys reading this through an RSS reader:

I realize that Duncan Temple Lang has created the RAmazonS3 package which can easily do what the above code sample does. The advantage of using rJava and the AWS Java SDK is the ability to apply the same approach to ALL the AWS services. And since Amazon maintains the SDK this guarantees that future AWS services and features will be supported as well.

I’m a huge O’Reilly Media fan boy. I can’t hide it. I hear Tim O’Reilly speak at conferences and I think to myself, “Screw being president, I want to be Tim O’Reilly.” I’ve been a subscriber to their online book services called Safari Books Online for years. Every month I see the bill for $43 come through and I think to myself, “Self, that’s the best $43 you spent all month.” But the real downside of Safari Books Online is that it is, as the name implies, an online service. I spend 90 minutes each day on a train and I would LOVE to spend a huge chunk of that time reading O’Reilly books. My iPad is not the 3g model so reading Safari Books Online is not an option for me. Then earlier this week I read that they had released the O’Reilly Safari to Go app for the iPad. I was stoked and excited! I got so breathless that I even tweeted my excitement and then was re-tweeted by @OreillyMedia as you can see from the image in the upper left corner.

I immediately downloaded the app and started playing with it. The fit and finish was not too good, but this is a first release product so I was cutting it some slack. It was a little slow and the screens visibly flashed when I changed screens. Typing was so sluggish that the cursor would lag behind my typing for 5-6 letters. This was all annoying but I was so excited to have these books on my train ride. After struggling a little to figure out how to get books into my offline book-bag I loaded 6 books into the bag and then left the app up and iPad running so they could download while I worked. When I got on the train I was dismayed to discover that I had no books at all in my off line book back. Odd. I know I put 6 in there. After tucking my daughter into bed I spent 3 hours fighting with the app. My final conclusion is that the app is complete and utter shit. It’s poorly designed, poorly executed, and horrible to use. And the UI is nothing like an iPad app. It has zero redeeming value. The offline book bag is so buggy that it takes me ~8 tries to get a single book in the book bag. Often this after waiting for > 5 minutes for the book to download only to have it fail and I have to start over. For online book reading on the iPad the mobile version of the Safari Books website is far superior to the iPad app. Most of my time with the app was reading error messages like the one to the left. What I found bemusing was that I really did feel mad at O’Reilly for this app. It wasn’t the mad that I feel when I get ripped off, it was the mad that I feel when my 3 year old dumps her plate out on the table like a baby. It was a feeling of being let down by someone who I know can do better. And it appears I’m not the only one. The pissed off comments on the Safari Books Online official blog are down right angry. So I did a little soul searching and asked myself why I felt so angry about my experience with the app. What I uncovered I tried to capture in a response post I made to CJ Rayhill, SVP Product Management & Technology. You can see my response here. And here’s the same text for your easy reading enjoyment:

CJ, I know you and your team have to be in pain over this app. It’s terrible. You know that. And now you have a sunk cost problem, a vendor issue, and a “pissed off geeks with pitchforks” problem. Many of us have been there. There are bound to be multiple come-to-Jesus meetings over this. I’ve sat in meetings like that. I’ve led meetings like that. It sucks for every single person at the table.

I’m not sure if the vitriol in the tone of the comments above makes sense to you or your leadership team. Some folks reading this blog might think that the responses are a little over the top. Let me take a shot at helping this make sense through a personal anecdote.

I love O’Reilly Publishing. Recently I was invited to be a tech reviewer for _R Cookbook_ and I was over the moon to be asked by O’Reilly to be a reviewer because I love O’Reilly and I have a ton of positive feelings about those fantastic animal clad book covers. So, it’s an understatement to say I’m a fan. And I have this very personal device, my iPad, which I also love. This device is so intimate that I bring it to bed with me and my wife sometimes feels jealousy toward the time and attention I give to this device. So I invited O’Reilly, who I love and trust, to come join me for a shared experience on this very personal device. And when O’Reilly came over, in the form of the Safari to Go app, it was like having a trusted friend over who then decides to rub their muddy shoes on my suede couch while yelling “F*ck your couch! F*ck your couch!” The app is shockingly bad and totally inconsistent with the rest of my experience with O’Reilly. Hours which I could have spent kicking ass were spent being mocked by this poorly coded and dysfunctional app now hogging the resources of my most intimate personal companion.

You can see this level of hurt and frustration in the blog comments above. The relationship O’Reilly has with its customers is special. You help us kick ass each and every day. When we want to learn something we go to you and you teach us through your books, your blogs, and your magazines. We’re the ones who download IT Conversations podcast and scan through the playlist deleting Dr. Moira Gunn in order to move Tim O’Reilly higher up in the playlist. When we daydream about being rock stars, we don’t think about which model of Fender we’ll play, we think about which animal the editors will pick to go on the cover of our book. And we hope to god they don’t pick some overly cuddly critter or a 3 toed sloth. We want to be like Randal Schwartz and have our book known simply by the animal on the cover.

CJ, you’re an ass kicker too. You graduated from the Navel [sic] Academy, for crying out loud. You’re a trail blazer and the Safari to Go app is a trailblazer. But I (and many others) think this project has lost its way. It seems the trail you tried to blaze was creating a multi-platform reader. Please allow me to be so bold as to suggest this is not the right goal. A better goal is to thrill your rock star fans with the best possible mobile off line Safari reading experience that helps them kick serious ass. You’ve got some hard choices to make about your vendor, your technology stack, and your implementation strategy. They are hard choices. But hard choices are the cost of being a trailblazer. If it was easy, someone else would have already done it.

I believe that the Safari mobile initiative could revolutionize not only technical books, but also text books. But the core of the platform has to be solid. Not only is the current core not solid, it’s unusable. But I know you can fix it. I’m glad Safari Books Online has you at the helm of their ship. Fix this thing, CJ, so we can all be rock stars with you. We’re mad because were disappointed. But we want so much to be thrilled.

-JD Long
@CMastication

If the couch reference is not entirely obvious, then you should brush up on your Dave Chappelle:

Joshua Ulrich was kind enough to help me out by pointing me to RJDBC which scared me a little (I’m easily spooked) because it involves Java. The only thing I know about Java is every time I touch it I spend days trying to get environment variables loaded just exactly the way it wants them. But Josh assured me that it was really not that hard. Here’s the short version:

Download the RJDBC driver from Microsoft. There’s Win and *nix versions, so grab which ever you need. Unpack the driver in a known location (I used /etc/sqljdbc_2.0/). Then access the driver from R like so:

require(RJDBC)
drv <- JDBC("com.microsoft.sqlserver.jdbc.SQLServerDriver",
  "/etc/sqljdbc_2.0/sqljdbc4.jar") 
  conn <- dbConnect(drv, "jdbc:sqlserver://serverName", "userID", "password")
#then build a query and run it
sqlText <- paste("
   SELECT * FROM myTable
  ", sep="")
queryResults <- dbGetQuery(conn, sqlText)

I have a few scripts that I want to run on both my Ubuntu laptop and my Windows Server. To accommodate that I made my scripts compatible with both by doing the following to my drv line:

if (.Platform$OS.type == "unix"){
         drv <- JDBC("com.microsoft.sqlserver.jdbc.SQLServerDriver",
         "/etc/sqljdbc_2.0/sqljdbc4.jar")
} else {
         drv <- JDBC("com.microsoft.sqlserver.jdbc.SQLServerDriver",
        "C:/Program Files/Microsoft SQL Server JDBC Driver 3.0/sqljdbc_3.0
         /enu/sqljdbc4.jar")
 }

Obviously if you unpacked your drivers in different locations you’ll need to molest the code to fit your life situation.

EDIT: A MUCH better place to put the JDBC drivers in Ubuntu would be the /opt/ path as opposed to /etc/ which I used above. In Ubuntu the /opt/ directory is where one should put user executables and /etc/ should be reserved for packages installed by apt. I’m not familiar with all the conventions in Ubuntu (or even Linux in general) so I didn’t realize this until I got some reader feedback.

Be forewarned, RJDBC is pretty damn slow and it appears to no longer be in active development. For my use case, RODBC was clearly faster. But RJDBC works for me in Ubuntu and that was my biggest need.

I felt like this a number of times in my life. Many of my classes were focused on the technical implementations they kinda missed the section titled “Why I give a shit.” A perfect example was my Mathematics Principles of Economics class which taught me how to manually calculate a bordered Hessian but, for the life of me, I have no idea why I would ever want to calculate such a monster. OK, that’s a lie. Later in life I learned that bordered Hessian matrices are a second derivative test used in some optimizations. Not that I would EVER do that shit by hand. I’d use some R package and blindly trust that it was coded properly.

So back to PCA: as I was reading the aforementioned stats question I was reminded of a recent presentation that Paul Teetor gave at a August Chicago R User Group. In his presentation on spread trading with R he showed a graphic that illustrated the difference between OLS and PCA. I took some notes and went home and made sure I could recreate the same thing. If you have wondered what makes OLS and PCA different, open up an R session and play along.

Your Independent Variable Matters:

The first observation to make is that regressing x ~ y is not the same as y ~ x even in a simple univariate regression. You can illustrate this by doing the following:

set.seed(2)
x <- 1:100

y <- 20 + 3 * x
e <- rnorm(100, 0, 60)
y <- 20 + 3 * x + e

plot(x,y)
yx.lm <- lm(y ~ x)
lines(x, predict(yx.lm), col=”red”)

xy.lm <- lm(x ~ y)
lines(predict(xy.lm), y, col=”blue”)

You should get something that looks like this:

So it’s obvious they give different lines. But why? Well, OLS minimizes the error between the dependent and the model. Two of these errors are illustrated for the y ~ x case in the following picture:

But when we flip the model around and regress x ~ y then OLS minimizes these errors:

Ok, so what about PCA?

Well let’s draw the first principal component the old school way:

#normalize means and cbind together
xyNorm <- cbind(x=x-mean(x), y=y-mean(y))
plot(xyNorm)

#covariance
xyCov <- cov(xyNorm)
eigenValues <- eigen(xyCov)$values
eigenVectors <- eigen(xyCov)$vectors

plot(xyNorm, ylim=c(-200,200), xlim=c(-200,200))
lines(xyNorm[x], eigenVectors[2,1]/eigenVectors[1,1] * xyNorm[x])
lines(xyNorm[x], eigenVectors[2,2]/eigenVectors[1,2] * xyNorm[x])

# the largest eigenValue is the first one
# so that’s our principal component.
# but the principal component is in normalized terms (mean=0)
# and we want it back in real terms like our starting data
# so let’s denormalize it
plot(xy)
lines(x, (eigenVectors[2,1]/eigenVectors[1,1] * xyNorm[x]) + mean(y))
# that looks right. line through the middle as expected

# what if we bring back our other two regressions?
lines(x, predict(yx.lm), col=”red”)
lines(predict(xy.lm), y, col=”blue”)

PCA minimizes the error orthogonal (perpendicular) to the model line. So first principal component looks like this:

The two yellow lines, as in the previous images, examples of two of the errors which the routine minimizes.

So if we plot all three lines on the same scatter plot we can see the differences:

The x ~ y OLS and the first principal component are pretty close, but click on the image to get a better view and you will see they are not exactly the same.

All the code from the above examples can be found in a gist over at GitHub.com. It’s best to copy and past from the github as sometimes Wordpress molests my quotes and breaks the codez.

The best introduction to PCA which I have read is the one I link to on Stats.StackExchange.com. It’s titled “A Tutorial on Principal Components Analysis” by Lindsay I Smith.