Alternatives: I’ve been running Jungle Disk at home and really like it. I could use that at work except I have not set up an Amazon or RackSpace account with my work credit card. But I am in Chicago and my database server/ file server is in Dallas TX. So I decided to just create a mirror on my laptop onto a shared drive on my server. There’s lots of ways to do this, but the path I chose was to use RoboCopy, a command line copy tool from Microsoft that is part of the Windows Server 2003 Resource Kit. I’m running XP and I wanted the mirroring of my machine to be invisible, silent, and scheduled. To do this I found I needed to take the following steps:

1. Install RoboCopy
2. Create a batch file to mirror the directory I wanted
3. Create a windows script to call the batch silently
4. Schedule the windows script to run automagically

Install RoboCopy: Download the Windows Server 2003 Resource Kit and install it. Very easy.

Create a batch file to run RoboCopy: I named mine c:/backup.bat and it looks something like this:

Set Source=”C:\Documents and Settings\jdlong”

Set Dest=”\\myDallasServer\backup\jdlong”

Robocopy %Source% %Dest% /MIR /Z /R:0  >nul

This simply sets the source and destination and then runs RoboCopy with the /MIR (mirror) and /Z (restartable) switches invoked

Create a windows script: The problem with the batch file is that it is noisy when it runs. Even piping the output to nul it still produces a CMD window that stays up until it finishes running. That’s where the Windows Script file comes into play. It calls the batch file but hides the CMD window. I created a file called c:\runBackup.vbs that has this in it:

Set WshShell = CreateObject(“WScript.Shell”)
WshShell.Run chr(34) & “C:\backup.bat” & Chr(34), 0
Set WshShell = Nothing

The alternative title of this blog post is “How to get your sorry ass fired by violating your internal IT policies.” So keep that in mind as you read this.

I say lots of silly crap. Twitter allows me the pleasure of sharing this blather with the world. I was a little surprised that of all the things I have said over the last few months the above Tweet received the most discussion. Apparently this tweet captured the imagination and consternation of some fellow Tweeters. I had people follow up with me and basically ask, “what do you mean?” Twitter is good for a sound bite, but less so for an elaborate answer. Which brings us to this:

What are the top ways Amazon EC2 can allow a business user to escape the manipulative and counterproductive grip of corporate IT? Well I’m glad you asked!

1) Over-restrictive web filtering policies:  When I worked as a risk manager for a Fortune 500 insurance firm I was shocked on the first day when I could not search Google Groups. At the time Google Groups was one of my favorite resources for figuring out everything from SQL syntax to Excel formulas. The firm, like most firms, outsourced the filtering of web content. Apparently they signed up for “Super Freaking Restrictive” filtering. I could not even search the web for “Ubuntu” as all sites with the word Ubuntu in the title or with the world “Ubuntu” passed as a form submission were blocked. Apparently Ubuntu is not just a Linux distro, but also a militant organization of African computer programmers, or something. So how did I get around this with EC2? I would fire up an EC2 Ubuntu instance running Squid proxy before I left home, then ssh into the cloud from work and use a little SSH port forwarding to route my web traffic through the ssh connection and out via Squid. I set up my EC2 instance to listen for ssh on port 443 and my firm’s firewall would let the connection pass as it assumed it was simply ssl traffic into Amazon. Brilliant!

2) Under powered database servers: At another point I was responsible for data analytics on a portfolio of insurance policies. I had to join together data from multiple systems (underwriting, admin, claims, etc.). The firm was an Oracle shop and none of the Oracle machines had enough user space for me to make the big ass join that had to be made in order to cobble together my analytics. For a while I hobbled along using PROC SQL in SAS to bring all the data together inside of SAS running on a PC. Finally I just gave up and built my own data mart in the cloud. And I could totally cut my internal IT politics out of the system. Whew, once the politics and begging for resources was over I could kick ass at analytics without having to beg borrow and plead for permissions and space.

3) Failure to backup desktop machines / inadequate shared drive space: Another experience I had was with a firm that decided it was a good policy to NOT back up desktop PCs at all. Each department was given shared drive space on a central server where “business critical” files were supposed to be kept (whatever the hell that means). Only the files on the central server were backed up. I was in the risk management department (ironically) and we had a whopping 100 MB allocated to us. Yes, this was 2004 and 100 MB was not enough to hold 2 years of risk reviews. Not to mention any ad hoc analysis and all the supporting documents. So everyone had their desktop drives, at least one USB drive, and no off site backup. It was during this period that I discovered Jungle Disk which allows client side encrypted data to be backed up to Amazon! Off site backup problem solved! And, once again, corp IT cut out of the system. (yes, this is a use of S3, not EC2) By the way, I paid for backups out of my own pocket because I felt it was very important. Well, I did have the firm buy me books which I happily kept when I left. We’ll call it even.

Let me reiterate that all three of the above uses may have put me in direct violation of my corporate IT policies. And let me also state that ultimately I found a job at a firm where internal IT sees their job as helping the business units get crap done. If you are an IT professional and you find your self thinking, “damn, I have to make sure I restrict my users from all of these crafty uses of EC2″ then, jackass,you are the problem with your firm’s IT department. If you see your job as stopping users then you are a useless burden on your firm and you should be not only fired, but spat upon. The way to prevent users from doing these, and other “shadow IT” behaviors is to provide the IT services that help your users be awesome! If you do that then you don’t have to worry about what your users are up to. They’ll be too damn busy being awesome to have time to mess with Amazon EC2.

All the examples above took place at previous places of employment. I currently use Amazon EC2 in order to scale some of my analytics, but it is done with the knowledge and support of my internal IT team. They fully understand what I am doing and they want to help me be awesome at analysis. It’s amazing how much less time I am wasting these days now that I don’t have to be so creative about avoiding the manipulative and counterproductive intervention of my internal IT team.

I don’t want this blog to become an echo chamber of the same crap they blog about over at Revolutions Blog, but they pointed out a really good article from Wired magazine on the topic of the Gaussian Copula and its roll in the current market meltdown. I was pretty impressed that the author, Felix Salmon was able to boil the issues down well and even give really good examples of how correlations can change over time. He also presented one of the best illustrations of conditional probability by using an example of two school kids. If I keep reading articles like this maybe I will learn to describe what I do for a living better. As it stands I just tell folks at cocktail parties that I play piano in a whorehouse. It’s much easier for them to understand and also more socially acceptable.

One of my favorite lines from the article was the following:

…CDOs were invariably sold on the premise that correlation was more of a constant than a variable.

And I am glad to see Taleb getting more publicity even if it is for salacious sound bites like:

Anything that relies on correlation is charlatanism.

That’s a bit overstating it. But the qualifiers need to make that statement correct are just a bit too long for good publishing.

The article is simultaneously about the Gaussian Copula and also David X. Li who derived the formula for using the Gaussian Copula in bond pricing. His original published article on the topic is still available from Credit Metrics on their web site. I’ve never read the article so I printed it out to read on my flight today. The Wired article does a good job of separating Mr. Li’s derivation of a formula from the true problem, which was the application of this formula. Some stupid idiots who have opined on the topic can’t make that distinction and end up saying fantastically moronic things such as:

So now here we stand. The world’s most dynamic economic engine brought to it’s knees by one guy.

That’s a bit like blaming the Wright Brothers for the September 11 terrorist attacks. I mean if they had never invented the plane then that attack clearly would not have happened!

At the bottom of the above article I noticed that Mr. Salmon blogs over at Portfolio.com, the web site for Conde Nast(y) Portfolio magazine. I clearly need to start reading Portfolio magazine as that was the same magazine where Michael Lewis’ now infamous article on the death of Wall Street was published. You remember that article, right? It had this fantistic photoshopped image of the Wall Street Bull post copula:

Time for a smoke

Ahhh. that’s art!

I was reminded of this recently when reading about someone from the Object Management Group giving a conference presentation. They were listed as Whitey McCracker, OMG, giving a speech on “managing your objects in a functional object post internet paradigm (2.0).” To which I simply thought, “WTF does that mean? OMG that is hard to decipher. ”

I guess the bottom line is this, if you are thinking up a public acronym, be sure and run it by your 14 year old son to make sure it does not have an alternative meaning.

To battle the proliferation of TLAs I’ve taken to just saying “Whiskey Tango Foxtrot” because I think it makes me sound cool, like that gay guy in TopGun.

-JD

Staples Wants More Entropy

This blows my mind. My password is 24 characters long, contains numbers, mixed case,  and punctuation yet they think it is not strong? Damn. So I try again with a new routine adding more numbers more punctuation. Same error. Finally I used MiniSafe to produce a 30 character random password which I appended with “_staples”. Here’s the password:

r!-MzZDsXhczm&m#$@L%25HXw66cnb_staples

Looks friggin’ strong to me! But not to staples.com. I got the same error. I am left to conclude that Staples.com requires a password that does not contain the string “staples”. There is no warning of this. But since I entered a password with more entropy than Lyle Lovet’s hair I can only assume that the presence of the string “staples” causes the site to reject my password.

So after trial and error led me to this conclusion I chose a password without the forbidden string in it. Which password did I choose? The following:

password

Yep, and staples.com took it without so much as a warning. Nice ey? And their backup question about the name of my first dog? Yeah, his name was Rex. So my password is ‘password’ and my first dog is Rex. But my password with over 138 bits of entropy was rejected. I’m going to Office Max.

Technical Detail:

If you want to know how many bits of strength or entropy a password has, the Wikipedia article on password strength is a great place to start. My 30 character password was created with a generator that used all alphabet characters upper and lower case (26*2), all numbers (10) and 10 punctuation marks for a total of 72 possibilities for each of the 30 characters. To calculate the entropy I used the formula from Wikipedia and got:

30 * Log2(72) = 185.1