UPDATE

This post has been updated to include the awesome instructions from Leo regarding using the postgis-unstable ppa. Thanks Leo!

1. Install PostGIS from PPA

System -> Administration -> Software Sources

Click on the “Other Software” tab.

Click on “+Add”

Enter: “ppa:ubuntugis/ubuntugis-unstable”

More info: https://launchpad.net/~ubuntugis/+archive/ubuntugis-unstable

Now install the package “postgresql-8.4-postgis”

2. Setup your database

createdb my_db
createlang plpgsql my_db
psql -d my_db -f /usr/share/postgresql/8.4/contrib/postgis-1.5/postgis.sql
psql -d my_db -f /usr/share/postgresql/8.4/contrib/postgis-1.5/spatial_ref_sys.sql
psql -d my_db -f /usr/share/postgresql/8.4/contrib/postgis_comments.sql

Now you should be able to make a geometry table.

3. Setup your geometry table

At this point I’m showing how to make an example table. Your usage of the PostGIS library will probably differ.

create sequence points_id_seq;
create table points ( id integer primary key default nextval('points_id_seq') );
select AddGeometryColumn('points', 'location', 4326, 'POINT', 2);
create index points_location_idx on points using GIST ( location );

// Create a point
insert into points(location) values (ST_GeomFromText('POINT(-76.615657 39.327052)'));

// Returns the point
select * from points where ST_Distance(location, ST_GeomFromText('POINT(-76 39)')) < 1;

// Does not return the point
select * from points where ST_Distance(location, ST_GeomFromText('POINT(-76 39)')) < .1;

The post Installing PostGIS 1.5 on PostgreSQL 8.4 on Ubuntu appeared first on Web Development Advice and Tips.

But the answers are spread out and distributed randomly. Plus, I can only see “The program ‘______’ is currently not installed. You can install it by typing: sudo apt-get install ______” so many times before I lose interest and put off the task.

So, to prevent future headaches and help all of us out, I put together a how-to that can take a new Rails system from zero to code in a few minutes (depending on network speed) instead of half a day. With no further ado,

Setting up an Ubuntu box for Rails Development (including virtual machines)

Update everything

$ sudo aptitude update && sudo aptitude dist-upgrade

Install necessary packages

$ sudo aptitude install build-essential vim \
  vim-runtime git-core subversion libsqlite3-dev

We have projects in git (the newer ones) and svn, so we have to be able to switch fluidly between the two.

And I’m not trying to start a flame war, but Vim is the first choice for Rails development. You don’t have to take my word for it; Jamis Buck, Tim Pope, and Adam Lowe all say the same. I recommend the NERDTree, FuzzyFinderTextMate, BufExplorer, and rails.vim plugins if nothing else.

Install ruby

$ sudo aptitude install ruby1.8 ruby rdoc ruby1.8-dev \
  libpgsql-ruby1.8 libmysql-ruby1.8 irb libopenssl-ruby

Get rubygems from source. http://docs.rubygems.org/

$ wget http://rubyforge.org/frs/download.php/60718/rubygems-1.3.5.tgz

$ tar xzvf rubygems-1.3.5.tgz

$ cd rubygems-1.3.5

$ sudo ruby setup.rb

$ sudo ln -s /usr/bin/gem1.8 /usr/bin/gem

$ sudo gem update --system

Update ~/.gemrc (at least the :sources: chunk)

---
gem: --no-ri --no-rdoc
:benchmark: false
:verbose: true
:backtrace: false
:update_sources: true
:sources:
- http://gemcutter.org
- http://gems.rubyforge.org/
- http://gems.github.com

Install and setup Passenger (optional)

$ sudo apt-get install apache2-prefork-dev

$ sudo gem install passenger

$ sudo passenger-install-apache2-module

and follow the instructions passenger gives you. Most folks in the office run some form of an Apache-Passenger stack for development purposes. I can explain how that setup works in a future post.

Install geminstaller

$ sudo gem install geminstaller

Almost all of our legacy projects use geminstaller to manage gem dependencies. In the future, new projects will be moving to Bundler, but legacy projects matter and upgrading subsystems is not always justified.

“If it ain’t broke, don’t fix it” and the corollary: “because you probably won’t be able to bill it to the client” both apply.

Install common packages (optional, may be required by projects)

$ sudo aptitude install imagemagick

$ sudo gem install utility_belt open_gem redgreen

The first (imagemagick) is used by the Paperclip gem to manage files attached to Rails models. The second are tools that make Rails development more manageable on the command line. I highly recommend them. Besides, I’ve been sneaking redgreen in as a test dependency on our projects, so if you’re a developer here you’ll have to install it anyways.

Create an ssh key (follow instructions)

$ ssh-keygen

EXAMPLE PROJECT

All the stuff above won’t let you test-code-deploy without hassles, so here’s a sample run through of what the last mile of project-specific setup looks like.

Get the project. SVN, Git, whatever. Go to it.

$ svn co rails-project

$ cd rails-project

Run geminstaller

$ sudo geminstaller                                 # normal

$ sudo geminstaller -c config/test/geminstaller.yml # test env

Install necessary aptitude packages

$ sudo aptitude install texlive-latex-base

Don’t ask.

Create the test database

Mysql:

$ rake RAILS_ENV=test db:create

Postgres (using your username or DB username in place of “username”):

$ sudo -u postgres psql

postgres=# CREATE USER username SUPERUSER;

postgres=# CREATE DATABASE username OWNER username;

If you get an error regarding PGconn.quote_ident (Rails 2.3.x on Ubuntu 8.04) patch activerecord-2.3.4/lib/active_record/connection_adapters/postgresql_adapter.rb and add the following:

def PGconn.quote_ident(name)
  %("#{name}")
end

immediately after the “class PGresult” definition at the top of the file. Once that’s all done, the following:

$ rake RAILS_ENV=test db:create

$ rake RAILS_ENV=test db:schema:load

$ rake RAILS_ENV=test test

should work.

go have fun, write code, blow our minds

So that’s how we roll. Awkward requirements will still crop up as we pick up new projects (or very old projects) so you’ll always need a bit of apt/gem fu to get you through the day. Leave questions here or drop me a line at adam@smartlogicsolutions.com or http://twitter.com/abachman.

The post Setting Up Ubuntu 9.10 for Ruby On Rails Development appeared first on Web Development Advice and Tips.

“Byobu is a Japanese term for decorative, multi-panel screens that serve as folding room dividers. As an open source project, Byobu is an elegant enhancement of the otherwise functional, plain, practical GNU Screen.”

https://wiki.ubuntu.com/MeetingLogs/openweekKarmic/Byobu

SSH into your server and install the two packages:

sudo apt-get install byobu landscape-common update-motd

Next, run byobu-config:

Change byobu settings with byobu-config

You can change Byobu’s colors if you’d like. I like the “dark” theme.

The important setting to change is the last one: “Byobu currently does not launch at login“. Select that and press return. Now you can go back to the menu and exit byobu-config (tab to the exit choice and hit return).

Now log out and log back in again. You’ll now access Byobu automatically.

Byobu as your session manager

The new MOTD (that’s “Message of the Day”, which is the text that is displayed when you login) has system information, thanks to landscape-common. This will let you know if your almost out of memory or disk space, or if you’re under heavy load.

Now let’s look at the information Byobu provides. The bottom row of Byobu contains, from left to right, your Distribution logo and name, uptime, load, cpu frequencies, ram amount and usage, and the date.

The upper row are your windows. Press F2 to create a new window. Along the bottom of the screen you’ll now see two entries in the top row of the footer. Press F3 to move to the previous window and F4 to move to the next window.

Press F8, then give the window a name, like “echo”. I like to name each tab with the task I’m doing, like “console”, “mysql”, and “top”.

You can press F7 to enter copy/scrollback mode. Use the arrow and page-up/down keys to move around your session. Press the spacebar to start copying. Now move your cursor somewhere else. Then press space again to copy the selection. Byobu doesn’t have a paste hotkey, so you have to use the screen hotkey, which is CTRL-a ] (thats CTRL-a, then let go of control and press ]).

You can press F12 to quickly lock your terminal, to grab a quick cup of coffee (or read this blog!).

Let’s explore how to maintain a session. Issue the following command:

echo hi

That will print out “hi” on the screen. Now let’s exit this session. Press the Byobu hotkey F6 to “Detach” from Byobu.

You’ll notice you’re still logged in, this is because Byobu is an application that runs on top of your session. So now you have to exit again to get back to your local machine.

Now SSH back into your machine. You’ll notice that command is still on the screen! “Detaching” does not end your session, it just detaches you from it. That means you can log in, run a long running task, and then Detach and come back in the morning when your task is done.

For extended information on Byobu, check out its Ubuntu Man Page.

The post Using Byobu and Landscape to improve remote Ubuntu sessions appeared first on Web Development Advice and Tips.

2) Run:
> tune2fs -o journal_data_writeback /dev/sdXY
Where /dev/sdXY is replaced by the partition that you want to boost

4) Edit fstab

> nano -w /mnt/sdXY/etc/fstab

Find the line that references sdXY. It will look something like:

# /dev/sda2
UUID=be2f0ac2-4683-4550-bcd1-704a1a840b3e / ext4 relatime,errors=remount-ro 0 1

The first entry is the UUID (although on your system this could just be /dev/sdXY). The second entry is the path (/ for me). Third is the fstype (ext3/4). Fourth are the options. Fifth is for dump and sixth is pass. See man fstab(5) for more info.

Change the options to:

noatime,data=writeback,barrier=0,nobh,errors=remount-ro

(you can leave all of yours in place, if they weren’t the same as mine.

The main ones are replacing atime/relatime with noatime. This causes the FS to not write read-times to a file when read. Think about it. Writing to the FS for every read of the FS? crazy!

Next is data=writeback. This means that metadata for files can be written lazily after the file is written. This will not cause file system corruption, but it may cause the most recent changes to be lost in the event of a crash (so you may jump back into the past a bit).

Next is barrier, which is slightly more dangerous:

barrier=<0|1(*)> This enables/disables the use of write barriers in
the jbd code. barrier=0 disables, barrier=1 enables.
This also requires an IO stack which can support
barriers, and if jbd gets an error on a barrier
write, it will disable again with a warning.
Write barriers enforce proper on-disk ordering
of journal commits, making volatile disk write caches
safe to use, at some performance penalty. If
your disks are battery-backed in one way or another,
disabling barriers may safely improve performance.

Next is nobh:

bh (*) ext4 associates buffer heads to data pages to
nobh (a) cache disk block mapping information
(b) link pages into transaction to provide
ordering guarantees.
“bh” option forces use of buffer heads.
“nobh” option tries to avoid associating buffer
heads (supported only for “writeback” mode).

You can skip barrier and nobh if you’d like. noatime and data=writeback are the big ones.

6) Reboot to your system.

If you have any trouble booting, just boot a recovery disk and revert the fstab changes.

EDIT: Updated to no longer require recovery disk booting thanks to Nicolas Alpi’s response post.

The post Mount options to improve ext4 file system performance appeared first on Web Development Advice and Tips.

Slow-Actions is great tool for determining the slow areas of a rails application. Since it is built on a rails log parser, it can be used for many other things. For example, here is some standard output from slow-actions when you want to know the speed of different user sessions within a specific date range:

slow-actions log/development.log --sessions --start-date=2009-03-01 --end-date=2009-04-01
           Cost    Average Max
+ ece0e17b48a5fe0fc766fa42f199fb66 (51 entries, 58% Error)
| Total:   2883.262 732.948 3100.00 
| Render:  0.00000 0.00000 0.00000 
| DB:      55.8220 14.1900 99.0000 

+ e1181ad94b34b41aff2a74ee62d5fb57 (10 entries, 30% Error)
| Total:   1563.912 676.276 1426.00 
| Render:  0.00000 0.00000 0.00000 
| DB:      14.5350 6.28500 19.0000 

+ 867bc340e6d65e673f29c57189feaf2b (133 entries, 0% Error)
| Total:   1083.138 221.450 386.000 
| Render:  0.00000 0.00000 0.00000 
| DB:      44.2030 9.03700 90.0000 

Etc...

Now, if we grep for “entries” we get a line for each session:

slow-actions log/development.log --sessions --start-date=2009-03-01 --end-date=2009-04-01 | grep entries
+ ece0e17b48a5fe0fc766fa42f199fb66 (51 entries, 58% Error)
+ e1181ad94b34b41aff2a74ee62d5fb57 (10 entries, 30% Error)
+ 867bc340e6d65e673f29c57189feaf2b (133 entries, 0% Error)
+ 99b39c7a9af55fc01056b5f127dc4c98 (75 entries, 0% Error)
+ 6ee15c4f3bc804ac68e9e868d9c3f7e6 (14 entries, 0% Error)
+ 80928c25c83590c4f89b91f8d4c8896d (9 entries, 0% Error)
+ c78cfa1cb086b3f473015b212ca3da11 (12 entries, 0% Error)
+ 48cd02cf6544493077c84eb16821bda2 (3 entries, 66% Error)
+ ac027817dcf87d473efb54cd737b2a80 (2 entries, 0% Error)
+ dff2ca2678852b06bbb6c7ca0a85b0a9 (1 entries, 0% Error)
+ 08ed3bc031c23bd3b2572ef7ce60d066 (1 entries, 0% Error)

Pipe that to word count, and we have the number of unique sessions within a specified time period:

slow-actions log/development.log --sessions --start-date=2009-03-01 --end-date=2009-04-01 | grep entries | wc -l
11

On a 54m log file, that takes only 0.761s!

The post Find the Unique Sessions for a Rails Application appeared first on Web Development Advice and Tips.

I wanted to preview my HTML emails without having to fill up my online email inboxes with tons of email (and then I’d have to make filters too). I also didn’t want to manage actually sending real email. So, I set up my machine for local delivery. Read on for instructions.

I am running Ubuntu 9.04, however these instructions should be fine at least back to 8.04.

Step 1 – Install Postfix

sudo aptitude install postfix

During the install, choose “Local Delivery Only”

Step 2 – Configure Evolution for Receiving Locally

Open up Evolution “Applications->Internet->Evolution Mail”

Set up a new account and choose local delivery. When it asks you for a path, put in:

/var/mail/<username>

Note that this path may not exist if you’ve never received mail.

Step 3 – Configure Sanitize Email

In your rails test environment.rb put:

config.action_mailer.delivery_method = :sendmail

Then, in the initializer where you set up sanitize email:

ActionMailer::Base.sanitized_recipients = ["@"]
ActionMailer::Base.local_environments = %w( test )

If you need to know your username or hostname, just open up a terminal. The command “whoami” will give you your username, and “hostname” will give you the hostname.

Step 4 – Run your tests

Now, run your rails tests. You should have a test for every email you send, and that should trigger them to be sent locally. You may have to tell Evolution to Send/Receive. Keep in mind your tests will probably fail because they will expect email to be in test mode.

Step 5 – Set it back

Remember, when you’re done previewing your email, reset config.action_mailer.delivery_method!

The post Using Sanitize Email to Preview HTML Emails Locally appeared first on Web Development Advice and Tips.

Whenever Integrity would try to build my project, I’d get an error about rake not being able to be found: sh: rake: not found

This totally threw me. I had added it to /etc/environment

PATH="/opt/ruby/bin:$PATH"

and so it was certainly on my PATH:

john@john-ci:~$ echo $PATH
/opt/ruby/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

I had also added it to /root/.bash_profile so that root would have it picked up:

john@john-ci:~$ sudo su -
root@john-ci:~# echo $PATH
/opt/ruby/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

So why wasn’t Apache/Integrity picking it up? To seek this out, I opened up the Integrity source and edited /opt/ruby/lib/ruby/gems/1.8/gems/integrity-0.1.9.3/lib/integrity/project_builder.rb to output the values of `whoami` and $PATH to help me troubleshoot.

IO.popen("(echo `whoami` && echo $PATH && cd #{scm.working_directory} && $
          |output| build.output = output.read }

Doing this yielded the following output:

www-data
/usr/local/bin:/usr/bin:/bin

At this point, I wholly expected the PATH to not include /opt/ruby/bin, so this isn’t a surprise. But why is this being set?

The Real Problem

The real issue here was that I was trying to set the PATH in scripts that only get run when a shell is entered. Since apache starts up as a system process, it is not started from a shell, and does not have a PATH associated with it. In fact, if you open up /etc/init.d/apache, you’ll see on one of the first few lines that the PATH is distinctly set:

ENV="env -i LANG=C PATH=/usr/local/bin:/usr/bin:/bin"

This was precisely the PATH I was seeing when I hacked Integrity to output its PATH just prior to failing the Rake command. Now it’s fairly obvious that I just need to add in /opt/ruby/bin to the front of that PATH, and Integrity will be able to execute all of my ruby/rubygem executables (most importantly rake).

Recap of Steps to Install

1. Download and install REE
2. Create a symlink for /opt/ruby so it’s dead simple to upgrade when a new REE comes out: sudo ln -s /opt/ruby-enterprise-whatever-version-you-installed /opt/ruby
3. edit /etc/environment to add /opt/ruby/bin to your shell PATH : PATH=/opt/ruby/bin:$PATH
4. edit ~/.bash_profile to alias sudo so that sudo can inherit your environment : alias sudo='sudo env PATH=$PATH'
5. Reload your environment to pick up the new PATH: source /etc/environment && source ~/.bash_profile
6. Install Passenger: sudo gem install passenger && passenger-install-apache2-module
7. Install Integrity Gem: sudo gem install integrity
8. Install Integrity Home: sudo integrity install --passenger ~www-data/integrity
9. Install do_sqlite3 Gem: sudo gem install do_sqlite3
10. Prepare Integrity Database: cd ~www-data/integrity && sudo integrity migrate_db config.yml
11. Grant ownership of all integrity files to www-data: sudo chown -R www-data:www-data ~www-data/integrity
12. Create Apache config for Integrity: sudo nano -w /etc/apache2/sites-available/integrity:

    <VirtualHost *>
      ServerName ci.yourdomain.com
      DocumentRoot /var/www/integrity/public
    </VirtualHost>
    

13. Enable the site: sudo ln -s /etc/apache2/sites-available/integrity /etc/apache2/sites-enabled/002-integrity
14. Edit /etc/init.d/apache2 and add /opt/ruby/bin to the PATH as described above: ENV="env -i LANG=C PATH=/opt/ruby/bin:/usr/local/bin:/usr/bin:/bin"
15. Restart Apache: sudo /etc/init.d/apache2 restart
16. Navigate to http://ci.yourdomain.com/ and start adding projects.

Boy, that was a mouthful. Please let me know if you’ve also been successful, and if there are any other steps you take to set up Integrity with REE and Passenger.

The post Integrity CI on Passenger 2.2.2 with Ruby Enterprise Edition on Ubuntu 8.04 appeared first on Web Development Advice and Tips.

Now then, the guide:

1) Download the 9.04 daily live from here:

http://cdimage.ubuntu.com/daily-live/current/

2) Burn or mount in a VM
3) Boot
4) Run the installer. When it asks for partitioning, select manual.
5) Create a small (128m is good) Ext2 primary partition and mount it at /boot
6) Create a swap primary partition (1024m is a good min, up to however much ram you have is good. No more than 4096m is necessary).
7) Create a third primary partition that uses the rest of the space, type Ext3, mount as root.
8) Proceed through the rest of the install.
9) When it asks you if you want to reboot, say “Continue using livecd”
10) Open a terminal
11) Your root partition should be /dev/sda3 if you just had those three partitions. If not, substitude /dev/sdaX for /dev/DEV when I mention it.
12) On the terminal, run “sudo tune2fs -O extents,uninit_bg,dir_index /dev/DEV”
13) Then run:

sudo fsck -pf /dev/DEV
sudo mkdir /mnt/root
sudo mount /dev/DEV /mnt/root
sudo nano -w /mnt/root/etc/fstab

and change the “ext3″ to “ext4″ on the / partition.
14) Reboot

The post EXT4 On Ubuntu 9.04 appeared first on Web Development Advice and Tips.

Combining these tools together, we can do some pretty nifty code performance analysis in very few lines of code, e.g.

require 'rubygems'
require 'better-benchmark'

result = Benchmark.compare_realtime(:iterations => 10) { |iteration|
  save_the_world()
}.with { |iteration|
  save_the_world_and_save_the_girl()
}
Benchmark.report_on result

I have forked better-benchmark and wrapped the library up into a RubyGem.

Currently, the gem is available from github:

$> sudo gem install jtrupiano-better-benchmark

Based on preliminary discussions with Pistos, he intends to merge my changes back into his branch. I’ll update this post with the relevant details when that has been completed.

So, let’s get started by preparing your box for better-benchmark!

Installing R

On Mac OSX

This one’s simple. AT&T Research provides us with a DMG.

On Ubuntu 8.0.4

I first built this just fine on a Mac. But, I decided that I wanted a dedicated VM where I could run some fairly intense benchmarks. You’ll need to ensure you have the following apt packages installed:

* build-essential
* g77 (fortran compiler)
* x11-common

There may be a few others that are required, but I started on a pre-built box more or less ready to host rails apps, and so there may be a few other necessities I missed. The following steps will install R from source on Ubuntu (I was unable to get the necessary R headers installed using apt-get or apititude), and prepare you for installing the rsruby gem.

$> sudo apt-get install build-essential g77 x11-common
$> cd /opt
$> wget http://lib.stat.cmu.edu/R/CRAN/src/base/R-2/R-2.7.2.tar.gz
$> tar xzf R-2.7.2.tar.gz
$> cd R-2.7.2
# --enable-R-shlib is important...it signals the installer to build and make available libR.so
$> ./configure --enable-R-shlib
$> make
$> sudo make install

Installing rsruby

Assuming you installed R fine, rsruby should be easily installed using gem. If you have trouble, see the installation instructions on the project’s README.

Note that you’ll need to set the R_HOME environment variable prior to installing the gem. On Mac, R_HOME=/Library/Frameworks/R.framework/Resources. On Ubuntu (when installed from source), R_HOME=/usr/local/lib/R. I find it helpful to just drop this into /etc/environment (on Ubuntu) so that this variable is set upon login.

$> export R_HOME=/path/to/R/for/your/OS
$> sudo gem install rsruby -- --with-R-dir=$R_HOME

Installing better-benchmark

Currently, you can get this directly off of my fork on github.

$> gem sources -a http://gems.github.com/
$> sudo gem install jtrupiano-better-benchmark

Great, I have it, now what do I do with it??

So now we’re ready to benchmark something. The most important thing to understand when benchmarking is that you need to clearly identify what is part of the benchmark, and what is not. Let’s take a look at a real-world example. We’ll go through the following steps:

1. Hypothesize: define exactly what you’re looking to test, and take special care to describe what you are NOT testing.
2. Plan: write out step by step how you will accomplish your benchmark
3. Refine: identify which steps belong in the benchmark, and which are really setup/teardown aspects of the benchmark, and thus shouldn’t be included in the test.
4. Test: run the benchmarks
5. Rinse/Repeat: (as necessary) tweak your test parameters, tweak the tests, re-test

Real world example

I was building a set of services that needed to download tens to hundreds of thousands of feeds regularly and perform some post-processing. I pulled in the feed-normalizer gem. My implementation plan called for me to write separate services, one for “pulling down” the feed, and one for “processing” the feed.

My first approach entailed storing the feeds on the hard-drive as YAML (using the to_yaml function). After running into some obscure problems with YAML and multi-line strings, I started to explore alternative persistence formats. One that caught my attention was using the Marshall standard library to store the content as bytecode. This brings me to my hypothesis:

1) Hypothesis: Converting FeedNormalizer::Feed objects to bytecode using the Marshal library is faster than converting those same obejcts to YAML using the YAML library.

I also like to list out my goals at this stage:

1. We only want to test the #dump conversion (object –> string) and the #read conversion (string –> object). We do not want to test the write to disk and the read from disk portions.
2. We specifically want to test FeedNormalizer::Feed objects, since that’s what we’re using in our code.
3. Let’s benchmark YAML / Marshal #dump and #read methods specifically on FeedNormalizer::Feed objects

2) Plan:

1. We have already grabbed roughly 100 feed downloads, and they are sitting as raw xml in test/raw_rss.
2. We’ll read them in, then benchmark YAML.dump vs. Marshal.dump
3. Then we’ll independently benchmark YAML.read vs. Marshal.read

3) Refine:
Here’s where we need to ensure that unnecessary processing doesn’t make its way into our benchmarks. For instance, in order to benchmark Marshal#dump against YAML#dump, we’ll need to load up all of the FeedNormalizer::Feed objects (by reading them from disk and instantiating them) prior to starting the benchmark.

4) Test:
It’s a ruby script, so go ahead and run it. The set of options provided by better-benchmark are easily discernible from the source file.

5) Rinse/Repeat:
Same story as always.

Real Script Example

#!/usr/bin/env ruby
# Author: John Trupiano
# 2008-09-28
#
# In order to run this benchmark, you need to have the better-benchmark library installed (http://github.com/Pistos/better-benchmark/tree/master)
# better-benchmark depends on the rsruby gem (http://web.kuicr.kyoto-u.ac.jp/~alexg/rsruby/)
# rsruby depends on an installation of the computing package R (http://mirrors.ibiblio.org/pub/mirrors/CRAN/)
#
# For details on setting up R, rsruby, and better-benchmark, see this blog post: http://blog.smartlogicsolutions.com/2008/10/08/benchmark-ruby-code-with-r-rsruby-and-better-benchmark

# This is intended to be run from RAILS_ROOT, e.g.
# /path/to/rails/root $> ruby test/benchmark/yaml_bytecode_test.rb

require 'openssl'
require 'yaml'
require 'rubygems'
require 'feed-normalizer'
require 'lib/rss_parser'
require 'better-benchmark'

# GOALS 
# 1) We only want to test the #dump conversion (object --> string) and the #read conversion (string --> object).  We do not want to test the write to disk and the read from disk portions.
# 2) We specifically want to test FeedNormalizer::Feed objects, since that's what we're using in our code.
# 3) Let's benchmark YAML / Marshal #dump and #read methods specifically on FeedNormalizer objects

# PLAN
# 1) We've grabbed roughly 100 feed downloads sitting in test/raw_rss.
# 2) We'll read them in, then benchmark YAML.dump vs. Marshal.dump
# 3) Then we'll independently benchmark YAML.read vs. Marshal.read

fn_feeds = {}
root_dir = File.join("test", "raw_rss")
feed_ids = (1..62).to_a + (64..108).to_a

# build a global hash fn_feeds that contains the FeedNormalizer::Feed entries
feed_ids.each do |feed_id|
  infile = File.join(root_dir, feed_id.to_s, feed_id.to_s + '_dump.rss')
  fn_feeds[feed_id] = RssParser.parse(infile, :file)
end

#### BENCHMARK 1 ####
result = Benchmark.compare_realtime(
  :iterations => 25,
  :verbose => true
) { |iteration|
  fn_feeds.each_pair do |feed_id, feed|
    YAML.dump(feed)
  end
}.with { |iteration|
  fn_feeds.each_pair do |feed_id, feed|
    Marshal.dump(feed)
  end
}
Benchmark.report_on result


# now, let's create separate collections storing the dumps
yaml_dumps = {}
marshal_dumps = {}
fn_feeds.each_pair do |feed_id, feed|
  yaml_dumps[feed_id] = YAML.dump(feed)
  marshal_dumps[feed_id] = Marshal.dump(feed)
end

#### BENCHMARK 2 ####
result = Benchmark.compare_realtime(
  :iterations => 25,
  :verbose => true
) { |iteration|
  yaml_dumps.each_pair do |feed_id, yaml|
    YAML.dump(yaml)
  end
}.with { |iteration|
  marshal_dumps.each_pair do |feed_id, marshal|
    Marshal.dump(marshal)
  end
}
Benchmark.report_on result

And for good measure, let’s take a look at the results.

john-trupianos-macbook-pro:trunk john$ ruby test/benchmark/yaml_bytecode_test.rb 
.........................
Set 1 mean: 2.095 s
Set 1 std dev: 0.071
Set 2 mean: 0.123 s
Set 2 std dev: 0.019
p.value: 1.58214572048972e-14
W: 625.0
The difference (-94.1%) IS statistically significant.
.........................
Set 1 mean: 0.191 s
Set 1 std dev: 0.015
Set 2 mean: 0.022 s
Set 2 std dev: 0.012
p.value: 1.58214572048972e-14
W: 625.0

As R plainly tells us, the difference is statistically significant (those who remember their p-values from stats class raise your hands) for both benchmarks (note that I did run two separate benchmarks in this example, one for dumping and one for reading). That said, considering I don’t have a need for this data to be human readable on my filesystem, I can safely conclude that using the Marshal library in lieu of the YAML library will give me a performance boost on both the read/dump methods. Now, whether or not this boost is negligible in the scope of the greater system, well that’s a question for a separate benchmark.

(What I mean by this last part is that this #dump/#read portion of my whole system may be tiny. If it only represents 0.5% of the processing time, then the improvement my example shows may be more or less negligible in the context of the whole system. These are the types of questions you need to ask yourself when benchmarking.)

Primary take-home point: know what you’re benchmarking, and benchmark what you don’t know.

The post Benchmark Ruby Code with R, rsruby and better-benchmark appeared first on Web Development Advice and Tips.

john@john-ubuntu:~/passenger-recipes$ gem -v
1.1.1
john@john-ubuntu:~/passenger-recipes$ sudo gem update --system
Updating RubyGems
Nothing to update
john@john-ubuntu:~/passenger-recipes$ gem -v
1.1.1
john@john-ubuntu:~/passenger-recipes$ sudo gem update --system
Updating RubyGems
Nothing to update

The first few times I ran into this, I was too busy to figure out what was wrong and ignored it. However, there is one HUGE TIME SAVER in 1.2.0 that I just couldn’t wait for any longer. Up through 1.1.1, gem update and gem install calls both updated all locally cached gemspec’s from your gem sources. This not only led to large memory consumption (anyone on a VPS got a horror story?), but was also just a plain waste of time. If I know which gem I want to update, it should just update that gem. The onus should be on me specify that I want to fully update my cache.

That said, I finally came across Eric Hodel’s notice addressing this very problem. The solution:

john@john-ubuntu:~/passenger-recipes$ <strong>sudo gem install rubygems-update -v 1.1.1</strong>
Bulk updating Gem source index for: http://gems.rubyforge.org/
Successfully installed rubygems-update-1.1.1
1 gem installed
john@john-ubuntu:~/passenger-recipes$ <strong>sudo gem update --system</strong>
Updating RubyGems
Updating rubygems-update
Successfully installed rubygems-update-1.2.0
Updating version of RubyGems to 1.2.0
Installing RubyGems 1.2.0
...
... (success)
...
john@john-ubuntu:~/passenger-recipes$ <strong>gem -v</strong>
1.2.0

Happy upgrading.

The post I can’t upgrade RubyGems from 1.1.1 to 1.2.0 on Ubuntu appeared first on Web Development Advice and Tips.