Comments on: Loading Big (ish) Data into R http://www.cerebralmastication.com/2009/11/loading-big-data-into-r/ Something to Chew On Mon, 19 Jul 2010 21:30:07 -0400 http://wordpress.org/?v=2.9 hourly 1 By: Jay http://www.cerebralmastication.com/2009/11/loading-big-data-into-r/comment-page-1/#comment-52 Jay Fri, 27 Nov 2009 06:01:19 +0000 http://www.cerebralmastication.com/?p=416#comment-52 The link from Dirk's post covers a few of th eoptions nicely: 1.Set nrows=the number of records in your data (nmax in scan). 2.Make sure that comment.char="" to turn off interpretation of comments. 3.Explicitly define the classes of each column using colClasses in read.table. 4.Setting multi.line=FALSE may also improve performance in scan. The link from Dirk’s post covers a few of th eoptions nicely:
1.Set nrows=the number of records in your data (nmax in scan).

2.Make sure that comment.char=”" to turn off interpretation of comments.

3.Explicitly define the classes of each column using colClasses in read.table.

4.Setting multi.line=FALSE may also improve performance in scan.

]]> By: Jay http://www.cerebralmastication.com/2009/11/loading-big-data-into-r/comment-page-1/#comment-51 Jay Fri, 27 Nov 2009 05:54:46 +0000 http://www.cerebralmastication.com/?p=416#comment-51 I second Greg's message. colClasses should help a great deal. In addition setting strings as factors to FALSE should also be helpful in terms of speed. I second Greg’s message. colClasses should help a great deal. In addition setting strings as factors to FALSE should also be helpful in terms of speed.

]]> By: Greg http://www.cerebralmastication.com/2009/11/loading-big-data-into-r/comment-page-1/#comment-50 Greg Wed, 25 Nov 2009 20:02:49 +0000 http://www.cerebralmastication.com/?p=416#comment-50 Did you try specifying colClasses in read.csv? If I remember, that will speed things up significantly (see ?read.csv Notes). This is on 1 million lines of observations in a csv file. I ran the following: df = data.frame(x = rnorm(1e6), y = rnorm(1e6)) write.csv(df, file="df.csv") rm(list=ls()) gc() sytem.time(read.csv("df.csv", row.names = 1)) gc() system.time(read.csv("df.csv", colClasses = c("character", "numeric", "numeric"), row.names=1)) Also, I ran your sqldf code: gc() library(sqldf) f <- file("df.csv") system.time(bigdf <- sqldf("select * from f", dbname = tempfile(), file.format = list(header = T, row.names = F))) Results: No colClasses: elapsed = 31.345s with colClasses: elapsed = 11.801s sqldf: elapsed = 29.565s I'm trying to remember the mechanism for read.csv, but I think it may import the data as "character", then tries to cast to another type with as.*. The Note section of ?read.csv explains that "character" is much slower than "integer" (and "numeric", in my example). Nice post. Greg sessionInfo() R version 2.9.0 (2009-04-17) i386-apple-darwin8.11.1 locale: en_US.UTF-8/en_US.UTF-8/C/C/en_US.UTF-8/en_US.UTF-8 attached base packages: [1] tcltk stats graphics grDevices datasets utils methods base other attached packages: [1] sqldf_0-1.5 gsubfn_0.5-0 proto_0.3-8 RSQLite_0.7-1 DBI_0.2-4 [6] cimis_0.1-2 RLastFM_0.1-4 RCurl_0.98-1 bitops_1.0-4.1 XML_2.5-3 [11] lattice_0.17-22 loaded via a namespace (and not attached): [1] grid_2.9.0 Did you try specifying colClasses in read.csv? If I remember, that will speed things up significantly (see ?read.csv Notes). This is on 1 million lines of observations in a csv file.

I ran the following:
df = data.frame(x = rnorm(1e6), y = rnorm(1e6))
write.csv(df, file=”df.csv”)
rm(list=ls())
gc()
sytem.time(read.csv(“df.csv”, row.names = 1))
gc()
system.time(read.csv(“df.csv”,
colClasses = c(“character”, “numeric”, “numeric”), row.names=1))

Also, I ran your sqldf code:
gc()
library(sqldf)
f <- file("df.csv")
system.time(bigdf <- sqldf("select * from f", dbname = tempfile(), file.format = list(header = T, row.names = F)))

Results:
No colClasses: elapsed = 31.345s
with colClasses: elapsed = 11.801s
sqldf: elapsed = 29.565s

I'm trying to remember the mechanism for read.csv, but I think it may import the data as "character", then tries to cast to another type with as.*. The Note section of ?read.csv explains that "character" is much slower than "integer" (and "numeric", in my example).

Nice post.

Greg
sessionInfo()
R version 2.9.0 (2009-04-17)
i386-apple-darwin8.11.1

locale:
en_US.UTF-8/en_US.UTF-8/C/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] tcltk stats graphics grDevices datasets utils methods base

other attached packages:
[1] sqldf_0-1.5 gsubfn_0.5-0 proto_0.3-8 RSQLite_0.7-1 DBI_0.2-4
[6] cimis_0.1-2 RLastFM_0.1-4 RCurl_0.98-1 bitops_1.0-4.1 XML_2.5-3
[11] lattice_0.17-22

loaded via a namespace (and not attached):
[1] grid_2.9.0

]]>