Blag

I’ve taken Phil’s advice and removed the Javascript from the body’s onload event handler, replacing it with Prototype’s event handler.

    <script type="text/javascript">
    Event.observe(
             window, 
             "load", 
         <%=  remote_function :url => { :action => "get_prices", :isbn13 => @book.isbn13  },:method => :get %> 
        );
    </script>

I spent several hours yesterday fighting with RubyGems – I’d even written a vitriolic post about it – but I … did something … and bam, like that it was gone. RubyGem is Ruby’s version of Perl’s CPAN. It’s got a very annoying trait – its prodigious use of memory. For each gem ( a gem is a Ruby module – like rails or hpricot for example ) RubyGem would load the spec into memory in order ( I’m guessing after reading a bunch of forum posts ) to build a dependency tree. On a 256MB slice host, this pushes you into swap hell. On a 512MB host it would use up to 68% of memory.

So what can you do but rent a bigger slicehost? Moe Sizlack said it best: “I’m choking on my own rage over here!”

Naturally, it’s been fixed. Today.

If only I’d done something more constructive yesterday. Like played COD.

This Simple XMPP client for Ruby looks like a great way to get some IM loving into an application. I’d love a monitoring system which used IM to alert me to failures. Since IM is not exactly a perfect medium, sending a copy of the message via Email for critical alerts is probably a good idea too.

Could be fun to ask your monitoring IM buddy “Status?” and get a status report. X Emails delivered, Y Emails rejected, Z books looked up on Booko.

Fun. :-)

Well, apparently I’ve not been consistent enough – an important quality when one bags out others for being inconsistent. So – I’ve updated all references to “Bookie” to the now correct “Booko”.

While I was at it I updated the scraping code for Fishpond who have again updated their site. At least they’ve updated it for the better. Compare and contrast the old and new Hpricot XPath code for grabbing the book title and author.

The Old:

book.title     = (doc/"table/tr/td/div/h1").inner_html
book.author    = (doc/"table/tr/td/p[2]/a/font/u").inner_html

The New:

book.title     = (doc/"h1#product_title").first.inner_html
book.author    = (doc/"p#product_author/a").first.inner_html

Much nicer!

I added fragment caching to Booko the other day. I’ve added caching to four sections, the Recent Searchs, Most Popular, Shopping Cart and the section which displays the prices of a book.

Now, instead of calculating the most popular books every time someone views a page, we look to see if that part of the site has been created recently – if it has, use it again. The Shopping cart is another good example – it only changes when you add or remove a book from it. Now Booko avoids having to calculate the cost of your shopping trolley at all shops every time you view a page – it regenerates that part of the page only when you add or remove a book.

Kinda wishing I’d bench marked it before and after. I think it’s faster now – but that’s because I’m looking for it. It feels snappier. Anyway – let me know what you think, faster or slower?

Well, I’ve added another feature. You can now add books to a cart, which will calculate the price to buy all books in your cart at the various online book stores, including calculating shipping.

It might be slightly problematic for collecting referrer fees as people will probably not be clicking on each of the book links. I’ll have to have a think on that.

Next job is to make it look better.

Well, it’s been a while, but it looks like someone (Tim you’re the only person who looks at cooking books) found a bug. Searching for a book which has an ISBN which is only available via Fishpond and not Amazon, means that you could find the book, but that it didn’t have a title, author or image. Well, I’ve fixed that now.

Also, you can now click on the book cover image to take you to the price lookup page.

Lastly, shipping price can now be more than just a single number – I can define rules for shipping. So, for example ( the only one I’ve found so far ) Fishpond have free shipping for books over $50 – this is now correctly accounted for.

Watir ( “Web Application Testing in Ruby” – pronounced water ) is a sweet little system used to automate browsers for the purposes of testing your site. Unfortunately, it’s for IE on Windows. Despite being odd in that most Ruby software is Linux or MacOS it’s also frustrating because I don’t use Windows. Luckily the Tubes have granted us two alternatives, SafariWatir and FireWatir. They achieve browser automation in very different ways.

[More…]

I’ve updated Booko to have a default new search engine: fishpond.com.au. Thanks to Tim Evans for finding bugs in it immediately.

Unlike Amazon, Fishpond ( or any Australian bookstore as far as I can tell ) don’t provide an interface for searching their site. Fishpond take the inconvenience one step further by not providing a standard way of finding a book by ISBN. You have to search for the ISBN, then get the Fishpond ID of the book. ( BTW – searching for an ISBN can return multiple results – the same book, but often with different prices. Apparently it’s due to having multiple suppliers for the same book. ) Aside from these annoying bits, Fishpond seems to have a very complete listing. They have an associate program too, so sending traffic to their sites may provide some return.

So, I wrote a searching module for Fishpond. Let me know if you find bugs.

The domain name bookie.com.au was taken, so I figured I’d carry on the tradition of naming the product like it was a person whose name was Book…. Booko. So you can now find the cheapest place to buy books at booko.com.au.

Found another bug today when one of the remote shops was misbehaving. The open-uri library was throwing exceptions I wasn’t catching.

The original version in the shop model:

def get_pricing(book)
    eval( self.price_function + "(book)" )
end

Replaced by this version:

def get_pricing(book)
  begin
    eval( self.price_function + "(book)" )
  rescue Errno::ECONNRESET => e
    logger.info(">>> Connection Reset checking #{self.name}")
  rescue OpenURI::HTTPError => e
    logger.info(">>> Connection Reset checking #{self.name}")
  end
end

Seems to work quite well.

Seems like Sunday is my blog day. Yesterday I submitted a patch for Ruby’s Open-URI module. It’s a tiny, 2 line patch, which could probably be written better, but it works and scratches an itch of mine. I think this is the first patch I’ve submitted to a software project. Hopefully it won’t languish like another patch to the same module.

Bookie currently checks prices at 9 shops. When you view a new book or refresh the pricing, it checks each of those shops in sequence. A nice little optimisation is to check the shops in parallel. Threading in rails appears to be pretty straight forward – provided you clean up before the request-response cycle is complete.

Here’s the sequential version:

for shop in Shop.find(:all)
   shop.get_pricing(@book)
 end

And the threaded version:

threads =[]
for shop in Shop.find(:all) 
  threads << Thread.new(shop) do |lookup|
    lookup.get_pricing(@book)
  end
end
threads.each{|thr| thr.join}
ActiveRecord::Base.verify_active_connections!()

By default ActiveRecord doesn’t allow multiple threads to access its mysql connection, so you need to allow concurrency by adding the following line to your environment.rb file:

config.active_record.allow_concurrency = true

That last line in the example is used to close database connections which are no longer attached to a thread. You’d have thought that joining the thread would have closed it’s database connection, but there you go. Without this you’ll eventually exhaust your database connections.

I’ve updated Bookie by adding another online Australian retailer, Fishpond. They seem to have good pricing on some books, and as a big plus, they have an Affiliate program which I’ve signed up for. I’ll get a cut on any books purchased via Fishpond, Amazon UK & Amazon US.

Adding Fishpond wasn’t as straight forward as the other stores. When searching for an ISBN, you can get multiple results, with different product ids and different prices. (According to their support guys, it’s because they give different product ids to books provided by different suppliers.) Just to make life difficult, the product id needed to generate Associate Links isn’t simply the ISBN. After a little bit of Hpricot work, I figured out how to grab all the data required to select the cheapest book ( if there’s multiple results ) and to generate Associate Links.

Lots of you have been noticing that “Bookie” craps out pretty frequently – turns out it’s due to SQLite3:

ActiveRecord::StatementInvalid (SQLite3::BusyException: database is locked: SELECT * FROM books WHERE (books.”isbn10” = ‘978-0596005696’) LIMIT 1):

Session data ( stored in the DB ) was a major culprit, but it looks like more than one concurrent user ( It happens! ) can cause this problem. SQLite’s homepage states that write operations lock the whole table, but since this should only take milliseconds, it shouldn’t be a problem.

I’ve migrated over to MySQL and performance feels much snappier. Maybe that’s just because I was hoping it’d be faster. With luck, bookie should be far more stable now.

Couple of small updates to Bookie, the most interesting of which was to filter the “Recent Searches” list to exclude filthy words. Timo was both the cause of the problem and suggested the solution. Thanks. I think ;-)

Here’s how it used to look:

( Note: It’s searching the book prices as they maintain an “updated at” value )

Price.find(:all).sort_by {|p| p.updated_at }.reverse.collect {|p| p.book}.uniq.compact.slice(0..14)

To filter out books with bad words in their title, I changed it to this:

Price.find(:all).sort_by {|p| p.updated_at }.reverse.collect {|p| p.book}.uniq.delete_if { |b| b.title =~ / shit| piss| cunt/i }.compact.slice(0..14)

Adding the following will delete the Books containing rude words:

.delete_if { |b| b.title =~ / shit| piss| cunt/i }

How sweet is that? You can practically read it like a sentence.