I initially tried running Unison on BOTH my laptop and the server and had the server Unison set to sync with my laptop back through an SSH reverse proxy. After testing this for a while I discovered this is totally the wrong way to do it. The problem is that the Unison process makes temp directories and files in the file system of the target. So my Unison job on the laptop would be trying to syn files and, in the process, create temp files which would kick off a Unison sync on the sever which would make temp files on the laptop… I think you can see how convoluted this gets.

So a much better solution is to only run Unison from one machine (I chose my laptop) and have the other machine (server in my case) send an SSH command (over the aforementioned reverse proxy) to the laptop asking the laptop to kick off a Unison sync. This way all of the syncs happen from the laptop.

So, in short, both machines run lsyncd which monitors files for changes. I keep up an SSH tunnel with reverse port forwarding which forwards a remote machine port back to my laptop’s port 22 (SSH). Unison need be installed ONLY on my laptop. When a change happens on my laptop, lsyncd fires off a Unison sync from my laptop that syncs it with the server. When a file changes on the server, the lsyncd job on the server makes a connection to my laptop via ssh and fires off a Unsion sync between my laptop and the server.

Here’s an example of my lsyncd config scripts:

Laptop:

settings = {
logfile = “/home/jal/lsyncd/laptop/lsyncd.log”,
statusFile = “/home/jal/lsyncd/laptop/lsyncd.status”,
maxDelays = 15,
–nodaemon = true,
}

runUnison2 = {
maxProcesses = 1,
delay = 15,
onAttrib = “/usr/bin/unison -batch /home/jal/Documents ssh://12.34.56.78//home/jal/Documents”,
onCreate = “/usr/bin/unison -batch /home/jal/Documents ssh://12.34.56.78//home/jal/Documents”,
onDelete = “/usr/bin/unison -batch /home/jal/Documents ssh://12.34.56.78//home/jal/Documents”,
onModify = “/usr/bin/unison -batch /home/jal/Documents ssh://12.34.56.78//home/jal/Documents”,
onMove = “/usr/bin/unison -batch /home/jal/Documents ssh://12.34.56.78//home/jal/Documents”,
}

sync{runUnison2, source=”/home/jal/Documents”}

Server:

settings = {
logfile = “/home/jal/lsyncd/server/lsyncd.log”,
statusFile = “/home/jal/lsyncd/server/lsyncd.status”,
maxDelays = 15,
–nodaemon = true,
}

runUnison2 = {
maxProcesses = 1,
delay = 15,
onAttrib = “ssh localhost -p 5432 unison -batch /home/jal/Documents ssh://12.34.56.78//home/jal/Documents”,
onCreate = “ssh localhost -p 5432 unison -batch /home/jal/Documents ssh://12.34.56.78//home/jal/Documents”,
onDelete = “ssh localhost -p 5432 unison -batch /home/jal/Documents ssh://12.34.56.78//home/jal/Documents”,
onModify = “ssh localhost -p 5432 unison -batch /home/jal/Documents ssh://12.34.56.78//home/jal/Documents”,
onMove = “ssh localhost -p 5432 unison -batch /home/jal/Documents ssh://12.34.56.78//home/jal/Documents”,
}

sync{runUnison2, source=”/home/jal/Documents”}

Keep in mind that I am using version 2 of lsyncd which can be downloaded here: http://code.google.com/p/lsyncd/

The version of lsyncd available in the Ubuntu repo is version 1.x which does not use the same config format as I illustrate above. However, if you run into dependency issues with v2, the easiest thing to do is install the repo version which will install dependencies and then manually download and install v2 from the above URL.

My reverse port forwarding set up looks like this:

autossh -2 -4 -X -R 5432:localhost:22 12.34.56.78

the -R bit forwards remote port 5432 to my laptop’s port 22 which is the ssh. So on my server if I run ssh localhost -p 5432 what actually happens is I am sshing from the remote machine to my laptop.

Notes:

• The IP address of my server in this example is 12.34.56.78.
• Don’t try and sync the directories where the lsyncd logs are kept. That will results in an endless sync cycle as each machine keeps noticing changes endlessly. Don’t ask me how I know this.
• The command to start the sync on the laptop is “lsyncd /home/jal/lsyncd/laptop/configfile” where configfile is the above lsyncd configuration file.
• lsyncd could, conceivably, tell Unison to sync only the part of the directory tree that changed. I have not been able to make that feature work right, however. And it only takes Unison a few seconds to sync, so I’ve not worried about it.

This has greatly sped up my RStudio based workflow when doing analysis with R. Now when I change files on my server using RStudio they are immediately (well it waits 15 seconds) replicated to my local machine and vice versa!

Good luck and if you have any suggestions please post a comment!

I get around this conundrum by running much of my analytics on either my work server or on an EC2 machine (I’m going to call these collectively “my servers” for the rest of this post). The nagging problem with this has been keeping files in sync. RStudio Server has been a great help to my workflow because it lets me edit files in my browser and they run on my servers. But when a long running R job blows out files I want those IMMEDIATELY synced with my laptop. That way I know when I undock my laptop to run to the train station that all my files will be there for me to spill Old Style beer on as I ride the Metra North line.

I experimented with Dropbox and I gotta say, it’s great. It really is well engineered, fast, and drop dead simple. I love that with Dropbox I could pull up most any file from my Dropbox on my iPad or iPhone. That’s a very handy feature. And it’s fast. If I created a small text file on my server, it would be synced with my laptop in a few seconds. Perfect! Wel… almost. Dropbox has a huge limitation: encryption. Dropbox encrypts for transmission and may even store files encrypted on their end. However, Dropbox controls the key. So if a rogue employee, a crafty Russian hacker, or a law enforcement officer with a subpoena gained access to Dropbox, they could get access to my files without my knowledge. As a risk manager I can’t help but see Dropbox’s security as a huge, targeted, single point of failure. It’s hard to say which would be a bigger payday: cracking GMail, or cracking Dropbox. But I’m suspicious it’s Dropbox. There are some workarounds to try and shoehorn file encryption into Dropbox, and they all suck.

So Dropbox can’t really give me what I want (what I really really want). But I stumbled into Spideroak who are like the smarter, but lesser known cousins of Dropbox. Their software does everything Dropbox does (including tracking all revisions!) but they have a “trust no one” model which encrypts all files before leaving my computer using, and this is critical, MY key which they don’t store. Pretty cool, eh? Spideroak also has a iPad/iPhone app and offers a neat feature that allows emailing any file in my Spideroak “bucket” to anyone using my iPhone without having to upload the file to my iPhone first. They do this by sending a special link to the email recipient that allows them to open only the file you wanted them to have. This could be a huge bacon saver on the road.

So Spideroak’s the panacea then? Well… um… no. They have two critical flaws: 1) They depend on time stamps on files to determine most recent file. 2) Syncs are slow, sometimes taking more than 5 minutes for very small files. The time stamp issue is an engineering failure, plain and simple. I’ve talked to their tech support and been assured that they are going to change this and index using server time, not system time in the future. But as of April 6, 2011, Spideroak uses local system time. For most users this is no big deal. For my use case this is painful. My server and my laptop were 6 seconds different and that time difference was enough for me to get Spideroak confused about which files were the freshest. This is a big deal when syncing two file systems with fast changing files. The other issue, slow sync, was actually more painful but probably the result of their attempt to be nice with CPU time and also encryption. When jobs on my server finished, I expected those files to start syncing within seconds and the only delay I expected was bandwidth constraints. With Spideroak syncs might take 5 minutes to start and then it would go out for coffee, come back jittery and then finally complete. Even if SPideroak fixed the time sync issue (or I forced my laptop to set its time based on my server), it still would not work for my sync because of the huge lags.

So looking at Dropbox and Spideroak I realized that I liked everything about Spideroak except its sync. It’s a great cloud backup tool that seems to properly do encryption, it’s multiplatform (win, linux, mac), has an iPad/iPhone app for viewing/sending files, it’s smart about backups and won’t upload the same file twice (even if the file is on two different computers). For my business use, I just can’t use Dropbox. The lack of “trust no one” encryption is a deal killer. So what I really need is a sync solution to use along side Spideroak.

There are some neat projects out there for sync. Projects like Sparkleshare look really promising but they are trying to do all sorts of things, not just sync. I’ve already settled on letting Spideroak do backup and version tracking so I don’t really need all those features… OK, OK, I can hear you muttering, “just use rsync and be done with it already.” Yeah, that’s a good idea. But rsync is single directional and does a lot of things well, but can also be a bit of an asshole if you don’t set all the flags right and rub its belly the right way. If you google for “bidirectional sync” you’re going to see this problem has plagued a lot of folks. This blog post has already gone on long enough so I’ll cut to the chase. Here’s the stack of tools I settled on for cobbling together my own secure, real-time, bidirectional sync between two Ubuntu boxes (one of which changes IP address and is often behind a NAT router):

1) Unison – Fast sync using rsync-esque algos and really fast caching/scanning

2) lsyncd – Live (real-time) sync daemon

3) autossh – ssh client with a nifty wrapper that keeps the connection alive and respawns the connection if dropped

I’ll do another post with the nitty-gritty of how I set this up, but the short version is that I installed Unison and lsyncd on both the laptop and the server. Single direction sync from my laptop to the server is pretty straight forward: lsyncd watches files, if one changes it calls unison which syncs the files with the server. The tricky bit was getting my server to be able to sync with my laptop which is often behind a NAT router. The solution was to open an ssh connection from my laptop to my server using autossh and reverse port forward port 5555 from the server back to my laptop’s port 22. That way an lsyncd process on the server can monitor the file system and when it sees a change can kick off a unison job that syncs the server to ssh://localhost:5555//some/path which is forwarded to my laptop! Autossh makes sure that connection does not get dropped and respawns if it does get dropped. So with a little shell scripting to start the lsyncd daemon on both machines, some config of lsyncd, and a local shell script to fire off the autossh connection, I’ve got real-time bidirectional sync!

In a follow up post I’ll put of the details of this configuration. Stay tuned. (EDIT: Update posted!)

If you’ve solved sync a different way and you like your solution, please comment. I’ve not settled that this is my long-term solution. It’s just a solution that works. Which is more than I had yesterday.

1. Clear goals (expectations and rules are discernible and goals are attainable and align appropriately with one’s skill set and abilities).

2. Concentrating and focusing, a high degree of concentration on a limited field of attention (a person engaged in the activity will have the opportunity to focus and to delve deeply into it).

3. A loss of the feeling of self-consciousness, the merging of action and awareness.

4. Distorted sense of time, one’s subjective experience of time is altered.

5. Direct and immediate feedback (successes and failures in the course of the activity are apparent, so that behavior can be adjusted as needed).

6. Balance between ability level and challenge (the activity is neither too easy nor too difficult).

7. A sense of personal control over the situation or activity.

8. The activity is intrinsically rewarding, so there is an effortlessness of action.

9. People become absorbed in their activity, and focus of awareness is narrowed down to the activity itself, action awareness merging.

Wow, that’s just like me playing Call of Duty 4! I recall being an undergraduate and playing Castle Wolfenstein 3D for so long that when I laid down to sleep I felt motion sick because I could still see the game in my head. I would get my flow on and not be able to stop playing. It was really addictive.

That addictive nature of flow is really the nature of the downside of flow. I had a ‘downside of flow’ experience in my first job after grad school. I was a consultant in a small firm and we did analytical modeling. Once challenge we had was to fit distributions to unknown data. We didn’t have the underlying data but we did know a couple points on the distribution and we could infer some things about the general behavior of the distribution. This is a fairly odd thing to do, actually. So there were no canned SAS routines to call. Code had to be written. After I was sure I understood the problem and worked with one of the senior principles of the firm to test some ideas I ran off to bang out code. I got totally absorbed in this project. I laid in bed at night and visualized optimization routines and transposed matrices. I would get up in the morning excited to go to work and stay late to tweak my code. After about a week the principle of the firm asked me how it was going and I showed him my work. He nodded and said, “Well good. What else have you done this week?” All I could think was, “WTF do you mean by ‘what else?’” I stuttered a bit and said that this had taken  all week. He was noticeably chagrined. He rubbed his forehead and said, “this is not an all week project. You should never have thrown all week at this, we have other things to do.” It felt like he kicked me in the balls. The very thing I was enjoying and proud of he thought was a waste of time. Ouch.

That’s been a few years and I have had the pleasure supervising others who are learning about both sides of flow. I’ve also lost days working on things that in retrospect I should not have sunk so much time into. I’ve taken to calling this type of flow as “technician’s myopia“. As a technical person it is soooo easy to get enthralled with the technical challenge at hand and totally lose context. Yet that is what flow is all about; getting totally sucked in. I’ve read stories of an academic statistician in the 60’s  who got his first computer and completely stopped doing academic research because he got sucked into learning all he could about the computer.

Technician’s myopia  is a real productivity killer for those of us in technical fields. What makes it so hard is the lack of absolutes. If I am having a bit of a slow time and I want to spend two days researching systems for distributed regression using Amazon EC2, that might be a good use of time as long as I limit myself to two days. If, on the other hand, I am working against a deadline and I take a tangent and lose myself in that tangent, I can waste days at the worst possible time.  How can us techie types keep from having technician’s myopia? Here’s my recommendations:

1. Self Awareness - If you want to have flow all the time and work on the wrong things while in that flow, have at it. Don’t read the other points below. Keep your resume polished, however, as you will need it. However if you want to flow when it’s right yet keep that flow focused, read on.
2. Management - If you have a technical manager he/she is most likely well aware of technician’s myopia, although the term may be new. Talk to them about it. If you are doing weekly project reviews then you can’t have myopia for more than a week. If you struggle with this a lot, ask for 15 minutes every morning to ensure that you are focused properly.
3. Buddy Dive - You may not have the type of manager who can understand technician’s myopia. In that case take a tip from the Navy Seals and buddy dive. Get one of your technical coworkers to sit down with you once or twice a week to discuss your projects and how they are going. Specifically ask each other if you are struggling with spending time on the wrong parts. Remember, it’s ok to flow, just don’t get lost in it to the detriment of what really matters.
4. Set Clear Deadlines - Having clear deadlines and even intermediate deadlines is just good self management. The neat thing about it is having clear deadlines and focused work attention actually helps most folks flow and prevents technician’s myopia. Double bonus!
5. Free Flow Time - Set aside time every week to work on projects which you are drawn to but which are off focus from your main work. This may be personal time or work time depending on your situation. Outside of work I have to have periodic ‘garage therapy’ where I disappear into the garage to get my flow on while working on my project car. My wife knows and understands this. If you desire flow but don’t get enough of it, you will feel restless and cranky. So make sure you have an outlet. But put a time frame on it. You will hate your job even more if you stay up all night killing headcrabs.

PS: as of March 10, 2009 Google returns zero hits for “technician’s myopia” if you put it in quotes. Hmm…

Today I was reading over at Data Evolution about a presentation on how Google and Facebook use R. The following was a summary of what Bo Cowgill of Google said about his workflow:

The typical workflow that Bo thus described for using R was: (i) pulling data with some external tool, (ii) loading it into R, (iii) performing analysis and modeling within R, (iv) implementing a resulting model in Python or C++ for a production environment.

I found this interesting as I have been masticating on the idea of learning Python for some time. I have run into situations where R was slow, but generally I have solved those through rethinking my algorithm. I’m not really a good programmer in R (or any other language for that matter), but I do want/need/like the statistical functions and ease of plotting in R. If I do learn Python I’ll certainly use it to call R… but maybe I should just stick to R.

This has nothing to do with workflow, but the most thought provoking insights in the article above came from Itamar Rosenn at Facebook:

Itamar’s team used recursive partitioning (via the rpart package) to infer that just two data points are significantly predictive of whether a user remains on Facebook: (i) having more than one session as a new user, and (ii) entering basic profile information.

… [they also] found that activity at three months was predicted by variables related to three classes of behavior: (i) how often a user was reached out to by others, (ii) frequency of third party application use, and (iii) what Itamar termed “receptiveness” — related to how forthcoming a user was on the site.

So Facebook really wants new users to put more info into FB, use it more, and play with third party apps. I guess that logic is why LinkedIn is always telling me I am only 90% complete on my profile and I would be 95% if I would just, yada yada yada… The more info I put into their walled garden, the more I will play there. And the more ads I will see. Makes sense to me. I guess I follow the same model when I try to get my clients to use my services more and more… I want to be sticky too. But not in a bad way.