Comments on: Benchmark Ruby Code with R, rsruby and better-benchmark http://blog.smartlogicsolutions.com/2008/10/08/benchmark-ruby-code-with-r-rsruby-and-better-benchmark/ News and updates from the people at SmartLogic Solutions Sat, 17 Dec 2011 12:57:18 +0000 hourly 1 http://wordpress.org/?v=3.2.1 By: roger http://blog.smartlogicsolutions.com/2008/10/08/benchmark-ruby-code-with-r-rsruby-and-better-benchmark/comment-page-1/#comment-709 roger Wed, 24 Dec 2008 13:42:49 +0000 http://blog.smartlogicsolutions.com/?p=275#comment-709 should advertise package in ruby talk :) should advertise package in ruby talk :)

]]> By: roger http://blog.smartlogicsolutions.com/2008/10/08/benchmark-ruby-code-with-r-rsruby-and-better-benchmark/comment-page-1/#comment-708 roger Wed, 24 Dec 2008 13:41:11 +0000 http://blog.smartlogicsolutions.com/?p=275#comment-708 Fascinating use of ruby. Fascinating use of ruby.

]]> By: John Trupiano http://blog.smartlogicsolutions.com/2008/10/08/benchmark-ruby-code-with-r-rsruby-and-better-benchmark/comment-page-1/#comment-371 John Trupiano Thu, 09 Oct 2008 15:01:58 +0000 http://blog.smartlogicsolutions.com/?p=275#comment-371 Thanks for calling this out Pistos. It would be interesting to explore the mathematical basis for a lot of this a little further (it's been several years since I took a stats course, so I have some brushing up to do). I'm particularly excited about what else rsruby can make available to us via R (pretty graphs, charts, etc.). better-benchmark is a great start, but I think there's an opportunity to beef it up a bit more (just not sure how and with what yet....back to the drawing board). I've gone ahead and updated this post with results for performing only 25 iterations as per your suggestion. Note that the p-values this time around are still practically 0. Last but not least, thanks for creating better-benchmark in the first place! Thanks for calling this out Pistos. It would be interesting to explore the mathematical basis for a lot of this a little further (it’s been several years since I took a stats course, so I have some brushing up to do). I’m particularly excited about what else rsruby can make available to us via R (pretty graphs, charts, etc.). better-benchmark is a great start, but I think there’s an opportunity to beef it up a bit more (just not sure how and with what yet….back to the drawing board).

I’ve gone ahead and updated this post with results for performing only 25 iterations as per your suggestion. Note that the p-values this time around are still practically 0.

Last but not least, thanks for creating better-benchmark in the first place!

]]> By: Pistos http://blog.smartlogicsolutions.com/2008/10/08/benchmark-ruby-code-with-r-rsruby-and-better-benchmark/comment-page-1/#comment-370 Pistos Thu, 09 Oct 2008 13:00:09 +0000 http://blog.smartlogicsolutions.com/?p=275#comment-370 Nice comprehensive post. Thanks for using better-benchmark! There's one thing I wanted to point out, though: [Statistics experts caution us](http://en.wikipedia.org/wiki/Wilcoxon_signed-rank_test#Test_procedure) that using a high number of pairs in the Wilcoxon test causes the calculated probability(ies) to tend toward a normal approximation. In our case, the number of pairs is the number of better-benchmark :iterations, and "high" is a subjective number, ranging from 15 to 25. In your code, you used 100 and, as you can see, your p values were 0.0 and W were 10000. If I'm not mistaken, these are not usual numbers to get out of the Wilcoxon test. In contrast, see how I run no more than 20 outer iterations in [my uses of better-benchmark](http://blog.purepistos.net/index.php?s=better-benchmark), choosing instead to increase the time spent per iteration by doing inner iterations. My p values are always non-zero (though sometimes extremely close to zero), and my W values are in the hundreds. Having said all that, I don't expect that reducing the iteration count will change the conclusion (of which method is faster), or the statistical significance of the results, but it would raise my confidence in the conclusion. Perhaps a statistics expert could shed more light on this? Nice comprehensive post. Thanks for using better-benchmark! There’s one thing I wanted to point out, though: [Statistics experts caution us](http://en.wikipedia.org/wiki/Wilcoxon_signed-rank_test#Test_procedure) that using a high number of pairs in the Wilcoxon test causes the calculated probability(ies) to tend toward a normal approximation. In our case, the number of pairs is the number of better-benchmark :iterations, and “high” is a subjective number, ranging from 15 to 25. In your code, you used 100 and, as you can see, your p values were 0.0 and W were 10000. If I’m not mistaken, these are not usual numbers to get out of the Wilcoxon test. In contrast, see how I run no more than 20 outer iterations in [my uses of better-benchmark](http://blog.purepistos.net/index.php?s=better-benchmark), choosing instead to increase the time spent per iteration by doing inner iterations. My p values are always non-zero (though sometimes extremely close to zero), and my W values are in the hundreds.

Having said all that, I don’t expect that reducing the iteration count will change the conclusion (of which method is faster), or the statistical significance of the results, but it would raise my confidence in the conclusion. Perhaps a statistics expert could shed more light on this?

]]>