Comments for Cerebral Mastication

Comment on Loading Big (ish) Data into R by JD Long

JD Long — Wed, 02 Jan 2013 17:07:00 +0000

Thanks Dieter. I really appreciate the link refresh.

Comment on Loading Big (ish) Data into R by Dieter Menne

Dieter Menne — Wed, 02 Jan 2013 16:35:56 +0000

Since the link to “this discussion” is dead, here is it again:

http://r-project.markmail.org/thread/iatn4krvfbbykfmk

Comment on Loading Big (ish) Data into R by JD Long

JD Long — Sun, 18 Nov 2012 19:32:19 +0000

That’s odd, I agree.

Comment on Loading Big (ish) Data into R by Hui Cheng

Hui Cheng — Fri, 16 Nov 2012 22:24:58 +0000

Hello,

Thanks for the post. I successfully read a 18660916 x 4 table using sqldf. However, when I tried it on a 4000 x 4500 table. Only header was read in. I couldn’t figure out why.

Hui

Comment on Loading Big (ish) Data into R by JD Long

JD Long — Fri, 19 Oct 2012 13:16:07 +0000

Sandy,

What platform are you on and how much memory do you have. i.e. are you on 64 bit Linux with 16gb ram?

Comment on Details of two-way sync between two Ubuntu machines by Wavemaker

Wavemaker — Thu, 18 Oct 2012 23:31:50 +0000

This tip is still valuable. Since long I’ve used this syncing setup with reasonable succes. Over time I’ve developed the sysv init script as below. It allows automatic starting of the sync at boot and works reasonably well for me. I hope it can be of use to someone.

— /etc/init.d/lansync on client —

#!/bin/sh
#
### BEGIN INIT INFO
# Provides: lansync
# Required-Start: $ssh
# Short-Description: lansync init script
# Description: files synchronization using lsyncd and unison
### END INIT INFO
#

NAME=”lansync”

NICELEVEL=19
IONICECLASS=”idle”
export HOME=”/root”
DAEMON=”/usr/local/bin/lsyncd”
PIDFILE=”/var/run/$NAME.pid”

CONFIG_FILE1=”/etc/lansync/lsyncd.conf”
ROOT1=”/home/testuser/LanSync”
ROOT2=”ssh://serverip//home/testuser/LanSync”
LOG_FILE1=”/var/log/lansync.log”

if [ "$(id -u)" != "0" ]
then
log_failure_msg “You must be root to start, stop or restart $NAME.”
exit 1
fi

case “$1″ in
start)
echo -n “Starting $NAME daemon…”
echo -n “Starting $NAME daemon…” >> $LOG_FILE1
date >> $LOG_FILE1

start-stop-daemon –start –oknodo –nice $NICELEVEL –iosched $IONICECLASS \
–exec $DAEMON — $CONFIG_FILE1 -pidfile $PIDFILE
#debug alternative:
#$DAEMON $CONFIG_FILE1 -pidfile $PIDFILE

daemon — nice ionice -c 3 unison -batch -group -owner -ui text $ROOT1 $ROOT2 > $LOG_FILE1
#debug alternatives:
#unison -ui graphic -group -owner $ROOT1 $ROOT2
#unison -batch -group -owner -ui text $ROOT1 $ROOT2

echo “done”
exit 1
;;
stop)
echo -n “Stopping $NAME daemon…”
echo -n “Stopping $NAME daemon…” >> $LOG_FILE1
date >> $LOG_FILE1

start-stop-daemon –stop –oknodo –pidfile $PIDFILE –retry 5
rm -rf $PIDFILE

echo “done”
exit 1
;;
restart)

/etc/init.d/$NAME stop
/etc/init.d/$NAME start

exit 1
;;
*)
echo “Usage: /etc/init.d/$NAME {start|stop|restart}”
exit 1
esac

exit 0

— End —

— /etc/init.d/lansync on server —

#!/bin/sh
#
### BEGIN INIT INFO
# Provides: lansync
# Required-Start: $ssh
# Short-Description: lansync init script
# Description: files synchronization using lsyncd and unison
### END INIT INFO
#

NAME=”lansync”

NICELEVEL=19
IONICECLASS=”idle”
export HOME=”/root”
DAEMON=”/usr/local/bin/lsyncd”
PIDFILE=”/var/run/$NAME.pid”

CONFIG_FILE1=”/etc/lansync/lsyncd.conf”
ROOT1=”/home/testuser/LanSync”
ROOT2=”ssh://serverip//home/testuser/LanSync”
LOG_FILE1=”/var/log/lansync.log”

if [ "$(id -u)" != "0" ]
then
log_failure_msg “You must be root to start, stop or restart $NAME.”
exit 1
fi

case “$1″ in
start)
echo -n “Starting $NAME daemon…”
echo -n “Starting $NAME daemon…” >> $LOG_FILE1
date >> $LOG_FILE1

start-stop-daemon –start –oknodo –nice $NICELEVEL –iosched $IONICECLASS \
–exec $DAEMON — $CONFIG_FILE1 -pidfile $PIDFILE
#debug alternative:
#$DAEMON $CONFIG_FILE1 -pidfile $PIDFILE

— /etc/init.d/lansync on client —

#!/bin/sh
#
### BEGIN INIT INFO
# Provides: lansync
# Required-Start: $ssh
# Short-Description: lansync init script
# Description: files synchronization using lsyncd and unison
### END INIT INFO
#

NAME=”lansync”

NICELEVEL=19
IONICECLASS=”idle”
export HOME=”/root”
DAEMON=”/usr/local/bin/lsyncd”
PIDFILE=”/var/run/$NAME.pid”

CONFIG_FILE1=”/etc/lansync/lsyncd.conf”
ROOT1=”/home/testuser/LanSync”
ROOT2=”ssh://serverip//home/testuser/LanSync”
LOG_FILE1=”/var/log/lansync.log”

if [ "$(id -u)" != "0" ]
then
log_failure_msg “You must be root to start, stop or restart $NAME.”
exit 1
fi

case “$1″ in
start)
echo -n “Starting $NAME daemon…”
echo -n “Starting $NAME daemon…” >> $LOG_FILE1
date >> $LOG_FILE1

start-stop-daemon –start –oknodo –nice $NICELEVEL –iosched $IONICECLASS \
–exec $DAEMON — $CONFIG_FILE1 -pidfile $PIDFILE
#debug alternative:
#$DAEMON $CONFIG_FILE1 -pidfile $PIDFILE

daemon — ssh clientip nice ionice -c 3 \
unison -batch -group -owner -ui text $ROOT1 $ROOT2 > $LOG_FILE1

echo “done”
exit 1
;;
stop)
echo -n “Stopping $NAME daemon…”
echo -n “Stopping $NAME daemon…” >> $LOG_FILE1
date >> $LOG_FILE1

start-stop-daemon –stop –oknodo –pidfile $PIDFILE –retry 5
rm -rf $PIDFILE

echo “done”
exit 1
;;
restart)

/etc/init.d/$NAME stop
/etc/init.d/$NAME start

exit 1
;;
*)
echo “Usage: /etc/init.d/$NAME {start|stop|restart}”
exit 1
esac

exit 0

— End —

echo “done”
exit 1
;;
stop)
echo -n “Stopping $NAME daemon…”
echo -n “Stopping $NAME daemon…” >> $LOG_FILE1
date >> $LOG_FILE1

start-stop-daemon –stop –oknodo –pidfile $PIDFILE –retry 5
rm -rf $PIDFILE

echo “done”
exit 1
;;
restart)

/etc/init.d/$NAME stop
/etc/init.d/$NAME start

exit 1
;;
*)
echo “Usage: /etc/init.d/$NAME {start|stop|restart}”
exit 1
esac

exit 0

— End —

Comment on Details of two-way sync between two Ubuntu machines by Wavemaker

Wavemaker — Thu, 18 Oct 2012 23:25:12 +0000

I’ve since a long time experimented with this syncing setup and now use the following sysv init files on ubuntu, which makes automatic management of the syncing possible. I hope it can be of use to someone. lsyncd.conf is still identical and should be present on both systems.

—- Code starting for system 1 (client) here: —-

#!/bin/sh
#
### BEGIN INIT INFO
# Provides: lansync
# Required-Start: $ssh
# Short-Description: lansync init script
# Description: files synchronization using lsyncd and unison
### END INIT INFO
#

NAME=”lansync”

NICELEVEL=19
IONICECLASS=”idle”
export HOME=”/root”
DAEMON=”/usr/local/bin/lsyncd”
PIDFILE=”/var/run/$NAME.pid”

CONFIG_FILE1=”/etc/lansync/lsyncd.conf”
ROOT1=”/home/test/LanSync”
ROOT2=”ssh://serverip//home/test/LanSync”
LOG_FILE1=”/var/log/lansync.log”

if [ "$(id -u)" != "0" ]
then

log_failure_msg “You must be root to start, stop or restart $NAME.”
exit 1
fi

case “$1″ in
start)
echo -n “Starting $NAME daemon…”
echo -n “Starting $NAME daemon…” >> $LOG_FILE1
date >> $LOG_FILE1

start-stop-daemon –start –oknodo –nice $NICELEVEL –iosched $IONICECLASS \
–exec $DAEMON — $CONFIG_FILE1 -pidfile $PIDFILE
#debug alternative:
#$DAEMON $CONFIG_FILE1 -pidfile $PIDFILE

echo “done”
exit 1
;;
stop)
echo -n “Stopping $NAME daemon…”
echo -n “Stopping $NAME daemon…” >> $LOG_FILE1
date >> $LOG_FILE1

start-stop-daemon –stop –oknodo –pidfile $PIDFILE –retry 5
rm -rf $PIDFILE

echo “done”
exit 1
;;
restart)

/etc/init.d/$NAME stop
/etc/init.d/$NAME start

exit 1
;;
*)
echo “Usage: /etc/init.d/$NAME {start|stop|restart}”
exit 1
esac

exit 0

—- END —-

—- Code starting for system 2 (server) here: —-

#!/bin/sh
#
### BEGIN INIT INFO
# Provides: lansync
# Required-Start: $ssh
# Short-Description: lansync init script
# Description: files synchronization using lsyncd and unison
### END INIT INFO
#

NAME=”lansync”

NICELEVEL=19
IONICECLASS=”idle”
export HOME=”/root”
DAEMON=”/usr/local/bin/lsyncd”
PIDFILE=”/var/run/$NAME.pid”

CONFIG_FILE1=”/etc/lansync/lsyncd.conf”
ROOT1=”/home/test/LanSync”
ROOT2=”ssh://clientip//home/test/LanSync”
LOG_FILE1=”/var/log/lansync.log”

if [ "$(id -u)" != "0" ]
then

log_failure_msg “You must be root to start, stop or restart $NAME.”
exit 1
fi

case “$1″ in
start)
echo -n “Starting $NAME daemon…”
echo -n “Starting $NAME daemon…” >> $LOG_FILE1
date >> $LOG_FILE1

start-stop-daemon –start –oknodo –nice $NICELEVEL –iosched $IONICECLASS \
–exec $DAEMON — $CONFIG_FILE1 -pidfile $PIDFILE
#debug alternative:
#$DAEMON $CONFIG_FILE1 -pidfile $PIDFILE

daemon — ssh clientip nice ionice -c 3 \
unison -batch -group -owner -ui text $ROOT1 $ROOT2 > $LOG_FILE1

echo “done”
exit 1
;;
stop)
echo -n “Stopping $NAME daemon…”
echo -n “Stopping $NAME daemon…” >> $LOG_FILE1
date >> $LOG_FILE1

start-stop-daemon –stop –oknodo –pidfile $PIDFILE –retry 5
rm -rf $PIDFILE

echo “done”
exit 1
;;
restart)

/etc/init.d/$NAME stop
/etc/init.d/$NAME start

exit 1
;;
*)
echo “Usage: /etc/init.d/$NAME {start|stop|restart}”
exit 1
esac

exit 0

—– END —-

Comment on Loading Big (ish) Data into R by Sandy

Sandy — Thu, 18 Oct 2012 08:04:19 +0000

Hi,

I’ve tried this sqldf approach with a file of 611 MB (2099 columns, 96360 rows), but I get the error message that it cannot allocate a vector of 411 Kb.
My memory.limit is 3000 MB, so I guess it should be possible.
What am I doing wrong?

Thanks,
Sandy

Comment on A Fast Intro to PLYR for R by Bob Muenchen

Bob Muenchen — Fri, 21 Sep 2012 17:05:03 +0000

Oops, I missed Wabe’s comment about summarize!

Comment on A Fast Intro to PLYR for R by Bob Muenchen

Bob Muenchen — Fri, 21 Sep 2012 17:02:19 +0000

If you do this example using plyr’s summerize function, you can avoid typing df$ so many times and you also get to name your columns:

> ddply(dd, c(“dim1″,”dim2″), summarize,
+ v1mean = mean(v1),
+ v2mean = mean(v2),
+ v3mean = mean(v3),
+ v1sd = sd(v1),
+ v2sd = sd(v2),
+ v3sd = sd(v3))
dim1 dim2 v1mean v2mean v3mean v1sd v2sd v3sd
1 A J 0.04736 0.091364 -0.30501 1.07886 0.96619 1.13664
2 B J 0.30301 0.227677 -0.24818 1.22907 0.98196 1.29158
3 B K -0.28136 -0.264688 0.35963 0.77938 0.83391 1.11255
4 C K -0.24694 -0.163735 -0.24561 0.79591 0.91162 0.83031