Joshua Ulrich was kind enough to help me out by pointing me to RJDBC which scared me a little (I’m easily spooked) because it involves Java. The only thing I know about Java is every time I touch it I spend days trying to get environment variables loaded just exactly the way it wants them. But Josh assured me that it was really not that hard. Here’s the short version:

Download the RJDBC driver from Microsoft. There’s Win and *nix versions, so grab which ever you need. Unpack the driver in a known location (I used /etc/sqljdbc_2.0/). Then access the driver from R like so:

require(RJDBC)
drv <- JDBC("com.microsoft.sqlserver.jdbc.SQLServerDriver",
  "/etc/sqljdbc_2.0/sqljdbc4.jar") 
  conn <- dbConnect(drv, "jdbc:sqlserver://serverName", "userID", "password")
#then build a query and run it
sqlText <- paste("
   SELECT * FROM myTable
  ", sep="")
queryResults <- dbGetQuery(conn, sqlText)

I have a few scripts that I want to run on both my Ubuntu laptop and my Windows Server. To accommodate that I made my scripts compatible with both by doing the following to my drv line:

if (.Platform$OS.type == "unix"){
         drv <- JDBC("com.microsoft.sqlserver.jdbc.SQLServerDriver",
         "/etc/sqljdbc_2.0/sqljdbc4.jar")
} else {
         drv <- JDBC("com.microsoft.sqlserver.jdbc.SQLServerDriver",
        "C:/Program Files/Microsoft SQL Server JDBC Driver 3.0/sqljdbc_3.0
         /enu/sqljdbc4.jar")
 }

Obviously if you unpacked your drivers in different locations you’ll need to molest the code to fit your life situation.

EDIT: A MUCH better place to put the JDBC drivers in Ubuntu would be the /opt/ path as opposed to /etc/ which I used above. In Ubuntu the /opt/ directory is where one should put user executables and /etc/ should be reserved for packages installed by apt. I’m not familiar with all the conventions in Ubuntu (or even Linux in general) so I didn’t realize this until I got some reader feedback.

Be forewarned, RJDBC is pretty damn slow and it appears to no longer be in active development. For my use case, RODBC was clearly faster. But RJDBC works for me in Ubuntu and that was my biggest need.

Yesterday I was listening to the IT Conversations Podcast and heard a presentation given by Michael Stonebraker from the Money:Tech conference. Stonebraker (I keep wanting to call him Ballbuster) talks about how the elephant database engines (SQL Server, Oracle, DB2, etc) are never going to make it in financial trading engines because their approach to databasing (like freebasing only with data) is antiquated. It hit me when I was listening to the presentation that I don’t really know about the different types of databases. I’ve always wondered why SAS can be so damn fast at certain database functions and Oracle or SQL Server so slow at the same thing. I understand a little better now. I’ve been trying to get SQL Server to act like some other kind of DB, when in reality I just need a different type of DB… like a column oriented DB, maybe.

Since I don’t need the type of super high performance DB that Ballbuster talks about, I am wondering if I can efficiently use some type of open source Column Oriented DB. But I have a lot of investment in learning SQL which I would like to not lose. Looks like I need to research my options.

Below are some sources for database technology information beyond the big three elephants:

• John Sichi’s blog. He is the co-founder of LucidDB
• Julian Hyde’s Blog - Founder of the Mondrian open-source OLAP engine, and chief architect of the SQLstream streaming SQL engine
• The Database Column - A multi-author blog on database technology and innovation.

More to come on this topic…

• 2 GB file size limit, at least historically
• Versions change with each edition of MS Office
• Sort of tough to write SQL scripts
• Very little automation, ie compression, backup, etc.
• Windows only

I used Oracle for a few years as a result of my previous employer being an Oracle shop. I then switched to SQL Server when I changed jobs. A full blown client/server DB really does not make a lot of sense for much of what I do. I don’t run a transactional data store. I don’t need to have dozens of users hooked to the DB. And I do sometimes need access to my data when I am not hooked to the mother-ship. So I could run the free version of SQL Server on my laptop or run MySQL on my laptop, but both of these options rub me the wrong way. Why? I do a lot of data analysis in R which is RAM intensive. Running a DB server on my laptop means that some fraction of my RAM is going to be taken up by the db server software which is hanging out waiting for me to throw requests at it. I could manually hack around this by starting the server before I load data and then killing it after the data is loaded. That’s just too big of a pain in my rectum. Oh yeah, one more design requirement: I want to be able to push the whole DB out to a storage blob at Amazon and pound on it using EC2 machines, running Linux. Plus I am cheap and don’t want to pay a lot.

I’ll probably end up with a model where I keep some master data sets on a client/server DB and then I will replicate chunks of that to my laptop into my serverless db. I’ll probably also put output from my desktop db back into the server after analytic work is  done.

I knew about SQLite because of an interview with its author, Richard Hipp on FLOSS Weekly. There’s also a video of Hipp talking at the Googleplex. I wish that guy was my neighbor. He seems like the type of guy who would shovel your walk for you then apologize for not doing it perfectly by sending over homemade cookies. Unrelated to the cookies, I really like that SQLite is weakly typed.  I’m a free spirit like that.

I did some digging for SQLite alternatives and came up with some stuff at StackOverflow. You can read the post but it reminded me of Firebird. I’m immediately drawn to FireBird since their logo looks so dang much like the Ruger logo:

But is Firebird able to be run severless?  If I have to install a server then I would just as well run MySQL.

Berkeley DB seems like another option worth investigating, although I am not sure if I can use it without really embedding it in another program the way that I can with SQLite.

SQLite gets bonus points for having native R drivers meaning that I don’t have to go through a connector technology like ODBC. This is important enough that I should probably make that a requirement. I think Berekley DB has support in R as well. I know for a fact that writing back to SQL Server through the R ODBC package (RODBC) is like pushing a car with a rope, but only slower. Plus I don’t know how to make ODBC work on Linux. Not rocket science, I am sure, but still one more thing I would have to learn before I do that which I am paid to do.

I’m going to do some testing, but it looks like I should test real life performance of SQLlite and Firebird with my data.  More to come on this, I am sure.