Comments on: Using the R multicore package in Linux with wild and passionate abandon http://www.cerebralmastication.com/2010/02/using-the-r-multicore-package-in-linux-with-wild-and-passionate-abandon/ Something to Chew On Wed, 07 Dec 2011 13:07:56 -0500 http://wordpress.org/?v=2.9 hourly 1 By: JD Long http://www.cerebralmastication.com/2010/02/using-the-r-multicore-package-in-linux-with-wild-and-passionate-abandon/comment-page-1/#comment-1585 JD Long Thu, 29 Apr 2010 18:12:45 +0000 http://www.cerebralmastication.com/?p=562#comment-1585 John, I think you want to use snow or RMPI. If you are already running MPI then RMPI is the way to go. If you just have a bunch of computers that are not currently running as a grid, I think snow is the way to go. I highly recommend also using the "for each" package. For each has backends for MC, snow, and RMPI so you only have to change one line of code to switch from one parallel method to another. John, I think you want to use snow or RMPI. If you are already running MPI then RMPI is the way to go. If you just have a bunch of computers that are not currently running as a grid, I think snow is the way to go. I highly recommend also using the “for each” package. For each has backends for MC, snow, and RMPI so you only have to change one line of code to switch from one parallel method to another.

]]> By: John Ramey http://www.cerebralmastication.com/2010/02/using-the-r-multicore-package-in-linux-with-wild-and-passionate-abandon/comment-page-1/#comment-1584 John Ramey Thu, 29 Apr 2010 17:00:23 +0000 http://www.cerebralmastication.com/?p=562#comment-1584 I have several networked computers that each have multiple cores. What R package would you recommend to take advantage of not only the multiple cores but the multiple computers? An even nicer feature that I'm seeking is a job queueing system for this kind of setup; if this is not available, that is okay. I'm really trying to figure out what my options are. Please note that I prefer an easier setup over an efficient setup because many of the people that will use this system are not proficient in R. I have several networked computers that each have multiple cores. What R package would you recommend to take advantage of not only the multiple cores but the multiple computers?

An even nicer feature that I’m seeking is a job queueing system for this kind of setup; if this is not available, that is okay. I’m really trying to figure out what my options are.

Please note that I prefer an easier setup over an efficient setup because many of the people that will use this system are not proficient in R.

]]> By: Aleks Clark http://www.cerebralmastication.com/2010/02/using-the-r-multicore-package-in-linux-with-wild-and-passionate-abandon/comment-page-1/#comment-1409 Aleks Clark Fri, 16 Apr 2010 08:06:46 +0000 http://www.cerebralmastication.com/?p=562#comment-1409 ever tried snowfall? sfClusterApplyLB is pretty efficient at maxing out my cores, not sure how the internals work, but it gets the job done without having to mess with what kind of jobs they are. ever tried snowfall? sfClusterApplyLB is pretty efficient at maxing out my cores, not sure how the internals work, but it gets the job done without having to mess with what kind of jobs they are.

]]> By: JD Long http://www.cerebralmastication.com/2010/02/using-the-r-multicore-package-in-linux-with-wild-and-passionate-abandon/comment-page-1/#comment-1236 JD Long Tue, 06 Apr 2010 20:16:30 +0000 http://www.cerebralmastication.com/?p=562#comment-1236 Ayman, thanks for posting that info! That is VERY helpful and I did not understand this before you posted it. Ayman, thanks for posting that info! That is VERY helpful and I did not understand this before you posted it.

]]> By: ayman http://www.cerebralmastication.com/2010/02/using-the-r-multicore-package-in-linux-with-wild-and-passionate-abandon/comment-page-1/#comment-1127 ayman Mon, 29 Mar 2010 23:14:10 +0000 http://www.cerebralmastication.com/?p=562#comment-1127 The way mc.preschedule works is this. Lets say you have 2 cores and a list of 10 things. If its set to TRUE like: mclapply(1:10, identity, mc.preschedule=TRUE) then R spawns two children (one for each core) and roughly takes all the odd numbers and sends it to the first core and the even numbers go to the second core. Lets say one of the even numbers (lets say #6) takes a long time to do. Then the odd numbers finish faster, then core #1 will appear to be idle while core #2 finishes. If its set to FALSE like: mclapply(1:10, identity, mc.preschedule=TRUE) the R will spawn 1 child for each number. As each child finishes, it launches the next. So in this example as #6 takes a long time holding up a core, it can still throttle the other # jobs through the remaining core. In the end, R will have launched as many children as was the length of the input sequence (1:10 in this case) - this takes some time to do as it has to launch a child and clean up afterwards for each thing. So generally, if you have a lot of little fast jobs, set it to TRUE to get better performance. If you have a lot of variance in the run time, set it to FALSE. The way mc.preschedule works is this. Lets say you have 2 cores and a list of 10 things.

If its set to TRUE like: mclapply(1:10, identity, mc.preschedule=TRUE) then R spawns two children (one for each core) and roughly takes all the odd numbers and sends it to the first core and the even numbers go to the second core. Lets say one of the even numbers (lets say #6) takes a long time to do. Then the odd numbers finish faster, then core #1 will appear to be idle while core #2 finishes.

If its set to FALSE like: mclapply(1:10, identity, mc.preschedule=TRUE) the R will spawn 1 child for each number. As each child finishes, it launches the next. So in this example as #6 takes a long time holding up a core, it can still throttle the other # jobs through the remaining core. In the end, R will have launched as many children as was the length of the input sequence (1:10 in this case) – this takes some time to do as it has to launch a child and clean up afterwards for each thing.

So generally, if you have a lot of little fast jobs, set it to TRUE to get better performance. If you have a lot of variance in the run time, set it to FALSE.

]]> By: userR http://www.cerebralmastication.com/2010/02/using-the-r-multicore-package-in-linux-with-wild-and-passionate-abandon/comment-page-1/#comment-1056 userR Tue, 23 Mar 2010 23:46:33 +0000 http://www.cerebralmastication.com/?p=562#comment-1056 Have you tried running Ubuntu's revolurion-r package in your VM? It includes a library with faster (multi-threaded) versions of many of the base functions. I've noticed that for certain operations, there are some substantial speedups (for my toy comparison on a 2-core Ubuntu VM, going from 8 seconds to 1.7) On Ubuntu 9.10, just run "sudo aptitude install revolution-r" My benchmark was: m <- matrix(rnorm(2 * 10^7), ncol = 10^4) system.time(crossprod(m)) Good luck! Have you tried running Ubuntu’s revolurion-r package in your VM? It includes a library with faster (multi-threaded) versions of many of the base functions.

I’ve noticed that for certain operations, there are some substantial speedups (for my toy comparison on a 2-core Ubuntu VM, going from 8 seconds to 1.7)

On Ubuntu 9.10, just run “sudo aptitude install revolution-r”

My benchmark was:

m <- matrix(rnorm(2 * 10^7), ncol = 10^4)
system.time(crossprod(m))

Good luck!

]]> By: dude http://www.cerebralmastication.com/2010/02/using-the-r-multicore-package-in-linux-with-wild-and-passionate-abandon/comment-page-1/#comment-584 dude Wed, 17 Feb 2010 04:40:55 +0000 http://www.cerebralmastication.com/?p=562#comment-584 How about some seamless integration of plyr with multicore - that would be awesome! How about some seamless integration of plyr with multicore – that would be awesome!

]]> By: Cerebral Mastication » Blog Archive » You can Hadoop it! It’s elastic! Boogie woogie woog-ie! http://www.cerebralmastication.com/2010/02/using-the-r-multicore-package-in-linux-with-wild-and-passionate-abandon/comment-page-1/#comment-575 Cerebral Mastication » Blog Archive » You can Hadoop it! It’s elastic! Boogie woogie woog-ie! Tue, 16 Feb 2010 18:31:27 +0000 http://www.cerebralmastication.com/?p=562#comment-575 [...] to visit with Madam Wu, you ask? Well the short answer is Hadoop. Yeah, the cute little elephant. As I have told you before, multicore makes your R code run fast by using worm holes to shoot your results back from the [...] [...] to visit with Madam Wu, you ask? Well the short answer is Hadoop. Yeah, the cute little elephant. As I have told you before, multicore makes your R code run fast by using worm holes to shoot your results back from the [...]

]]> By: JD Long http://www.cerebralmastication.com/2010/02/using-the-r-multicore-package-in-linux-with-wild-and-passionate-abandon/comment-page-1/#comment-532 JD Long Fri, 12 Feb 2010 16:36:52 +0000 http://www.cerebralmastication.com/?p=562#comment-532 Josh: Yeah I profiled and know exactly what's time consuming. Unfortunately it's the sauce ;) I do all my random draws prior to starting the runs, so the QRMlib process doesn't hold things up. In my sims I take my random draw and then do some curve fitting to resolve a number of relationships then it has to calculate gains/losses for 40K+ policy units. It's basically the valuation of the policies that takes the longest. Probably a reasonable analogy would be valuing 40K+ options in 40 markets. Josh:

Yeah I profiled and know exactly what’s time consuming. Unfortunately it’s the sauce ;)

I do all my random draws prior to starting the runs, so the QRMlib process doesn’t hold things up. In my sims I take my random draw and then do some curve fitting to resolve a number of relationships then it has to calculate gains/losses for 40K+ policy units. It’s basically the valuation of the policies that takes the longest.

Probably a reasonable analogy would be valuing 40K+ options in 40 markets.

]]> By: Joshua Ulrich http://www.cerebralmastication.com/2010/02/using-the-r-multicore-package-in-linux-with-wild-and-passionate-abandon/comment-page-1/#comment-528 Joshua Ulrich Fri, 12 Feb 2010 04:21:04 +0000 http://www.cerebralmastication.com/?p=562#comment-528 JD, Have you profiled your code to determine the bottleneck? Is it something in QRMlib or does your secret sauce need some spice? Best, Josh JD,

Have you profiled your code to determine the bottleneck? Is it something in QRMlib or does your secret sauce need some spice?

Best,
Josh

]]>