Blag o' dkam

New post on Daemons over on the Booko blog.

Just posted on the Booko Blogo an article about implementing Users and Passwords.

When testing new stuff for Booko, I sometimes create a new slicehost and build myself a test box. In the past I’ve done this manually, which really isn’t very difficult, but it turns out that Slicehost have an API to allow you to automate this stuff.  With the API you can create, destroy, rebuild and reboot your VPS which is pretty cool.  It also lets you manipulate all your DNS settings. Check out the API for complete details. 

Here’s a script I put together to build a new dev host for me and set it up so it’s ready to use.  It performs the following steps:

1. Check the Domain name I’ve selected for the host is managed by Slicehost.
2. Create a new 256MB VPS with Ubuntu 8.04 installed.
3. Create the domain names for the host (including on the internal interface if required)
4. Wait for host to build and for Networking and SSH to startup.
5. Add my SSH key to the root user’s account for passwordless login.
6. Update and upgrade the host.
7. Install Puppet  

I plan on having a puppet server setup soon which will take over the rest of the setup, so I’ve installed that. I’ll probably use the Net::SSH stuff to get the host added to puppet (signing certificates and such.)
[sourcecode language='ruby']
#!/usr/bin/env ruby

require ‘rubygems’
require ‘activeresource’
require ‘net/ssh’
require ‘net/scp’

API_KEY=”your_key_goes_here”
SITE=”https://#{API_KEY}@api.slicehost.com/”
DEFAULT_TTL = 300

host_name=”hostname”
int_host_name=”hostname-int”
domain_name=”mydomain.com.au”

fqdn = host_name + “.” + domain_name
origin = domain_name + “.”

##
# Required definitions to access the Slicehost stuff
##
class Slice < ActiveResource::Base
self.site = SITE
end

# Address class is required for Slice class
class Address < String; end

class Zone < ActiveResource::Base
self.site = SITE
end

class Record < ActiveResource::Base
self.site = SITE
end

##
# Create or update a DNS record
##
def create_host_record(zone_id, host_name, ip_address, ttl = DEFAULT_TTL)
host_record = Record.find(:first, :params => { :name => host_name, :zone_id => zone_id } )
unless host_record.nil?
host_record.data = ip_address
host_record.record_type = “A”
else
host_record = Record.new(:ttl => ttl, :record_type => ‘A’, :zone_id => zone_id, :name => host_name, :data => ip_address)
end
host_record.save
end

puts “Getting Zone data for \”#{domain_name}\”.”
dom = Zone.find(:first, :params => { rigin => origin } )

raise “Domain \”#{domain_name}\” not found. Won’t be able to create host record.” if dom.nil?

puts “Domain exists. Creating slice.”

slice = Slice.new(:image_id => 10, :flavor_id => 1, :name => host_name)
slice.save

puts “Slice created. Creating DNS records while it builds.”

create_host_record(dom.id, host_name, slice.addresses[0])
create_host_record(dom.id, int_host_name, slice.addresses[1]) unless int_host_name.nil?

puts “DNS created. Waiting for host to build and become active.”

while slice.progress != 100 && slice.status != “active”
puts “Host is #{slice.progress}% complete – host status: #{slice.status}”
sleep 10
slice.reload
end

puts “Host built. Waiting for host to startup.”

begin
Net::SSH.start(slice.ip_address, ‘root’, {:auth_methods => ["password"], :password => slice.root_password}) do |ssh|
puts “Connected to new host. Bootstrapping.”

puts “Creating .ssh directory and uploading public key.”
ssh.exec “/bin/mkdir -p /root/.ssh/”
ssh.scp.upload!(“/Users/dkam/.ssh/id_dsa_omena.pub”, “/root/.ssh/authorized_keys”)
puts “Done!”

puts “Updating apt and upgrading system.”
ssh.exec!(“/usr/bin/aptitude update”)
ssh.exec!(“/usr/bin/aptitude dist-upgrade -y”)
puts “Host has been upgraded and updated.”

  puts “Installing puppet.”
ssh.exec!(“/usr/bin/aptitude install puppet -y”)
puts “Puppet Installed.”
end
rescue Errno::ENETUNREACH
puts “Host network not up. Waiting 10 seconds, then retrying”
sleep 10
retry
rescue Errno::ECONNREFUSED
puts “SSH not ready. Waiting 10 seconds, then retrying”
sleep 10
retry
end

puts “Done! Enjoy your new host #{fqdn}. You can now ssh root@#{slice.ip_address}”

[/sourcecode]

I’ve taken Phil’s advice and removed the Javascript from the body’s onload event handler, replacing it with Prototype’s event handler.

    <script type="text/javascript">
    Event.observe(
             window,
             "load",
         <%=  remote_function :url => { :action => "get_prices", :isbn13 => @book.isbn13  },:method => :get %>
        );
    </script>

Ruby On Rails apps are finally easy to install – mod_rails is here. Just installed it for the Blag ( Typo 5.0.1 on Gentoo Unstable ) and it looks to be working quite nicely. Only roadbump I encountered was problems getting static content to be served. Kept getting:

[error] [client 124.168.83.214] client denied by server configuration: /var/www/da.nmilne.com/htdocs/public/javascripts/typo.js

The problem was pretty straight forward – now that mongrel isn’t serving up the static content, I had to make sure that apache was configured to allow access. Added a this stanza:

  <Directory "/var/www/da.nmilne.com/htdocs/public">
        Options FollowSymLinks
        AllowOverride None
        Order allow,deny
        Allow from all
  </Directory>

Naturally it feels snappier. Nice.

I spent several hours yesterday fighting with RubyGems – I’d even written a vitriolic post about it – but I … did something … and bam, like that it was gone. RubyGem is Ruby’s version of Perl’s CPAN. It’s got a very annoying trait – its prodigious use of memory. For each gem ( a gem is a Ruby module – like rails or hpricot for example ) RubyGem would load the spec into memory in order ( I’m guessing after reading a bunch of forum posts ) to build a dependency tree. On a 256MB slice host, this pushes you into swap hell. On a 512MB host it would use up to 68% of memory.

So what can you do but rent a bigger slicehost? Moe Sizlack said it best: “I’m choking on my own rage over here!”

Naturally, it’s been fixed. Today.

If only I’d done something more constructive yesterday. Like played COD.

Well, apparently I’ve not been consistent enough – an important quality when one bags out others for being inconsistent. So – I’ve updated all references to “Bookie” to the now correct “Booko”.

While I was at it I updated the scraping code for Fishpond who have again updated their site. At least they’ve updated it for the better. Compare and contrast the old and new Hpricot XPath code for grabbing the book title and author.

The Old:

book.title     = (doc/"table/tr/td/div/h1").inner_html
book.author    = (doc/"table/tr/td/p[2]/a/font/u").inner_html

The New:

book.title     = (doc/"h1#product_title").first.inner_html
book.author    = (doc/"p#product_author/a").first.inner_html

Much nicer!

• Bug fixed which stopped Booko remembering which Book searching place you last used. ( Bug reported by teh timo )
• Re-factored the functions which go and get the prices from the stores.
• Added an attribute to “Book” such that it now remembers when its prices were last looked up. Accidentally introduced a bug where any new book initially had no idea when it was last looked up and crashed the “Recently Viewed” method.
• Todo – better testing

I was contacted by a chap from The Nile to let me know that they not only have an Affiliate program I could join, but they have an API to access their site! I’ve spent the last couple of days adding the affiliate links. You’ll see that several stores now appear to link to clixGalore – but fear not, the links will still get you to the page you’re after.

You’ll also notice there’s now a new store to search – The Nile. It’s the fastest of the search engines by a considerable margin. ( Which isn’t really fair – at least one of the sites which is searched is just a fancy screen scraper.)

Additionally, I’ve cleaned up the CSS and layout some. Hopefully you won’t have the horizontal scroll bar on your browser, regardless of how wide you open the window.

You’ll also notice that there’s a context sensitive help message in the top right-hand corner. This should make it clear what you can do at which ever page you happen to be on.

Try out the updated Booko and let me know what you think.

EDIT: Added a tool tip style mouse over for the Most Popular/Recent list so you can see the full title. Thanks to Niall & Riina for their help & ideas.

I added fragment caching to Booko the other day. I’ve added caching to four sections, the Recent Searchs, Most Popular, Shopping Cart and the section which displays the prices of a book.

Now, instead of calculating the most popular books every time someone views a page, we look to see if that part of the site has been created recently – if it has, use it again. The Shopping cart is another good example – it only changes when you add or remove a book from it. Now Booko avoids having to calculate the cost of your shopping trolley at all shops every time you view a page – it regenerates that part of the page only when you add or remove a book.

Kinda wishing I’d bench marked it before and after. I think it’s faster now – but that’s because I’m looking for it. It feels snappier. Anyway – let me know what you think, faster or slower?