Blag

I’ve taken Phil’s advice and removed the Javascript from the body’s onload event handler, replacing it with Prototype’s event handler.

    <script type="text/javascript">
    Event.observe(
             window, 
             "load", 
         <%=  remote_function :url => { :action => "get_prices", :isbn13 => @book.isbn13  },:method => :get %> 
        );
    </script>

Ruby On Rails apps are finally easy to install – mod_rails is here. Just installed it for the Blag ( Typo 5.0.1 on Gentoo Unstable ) and it looks to be working quite nicely. Only roadbump I encountered was problems getting static content to be served. Kept getting:

[error] [client 124.168.83.214] client denied by server configuration: /var/www/da.nmilne.com/htdocs/public/javascripts/typo.js

The problem was pretty straight forward – now that mongrel isn’t serving up the static content, I had to make sure that apache was configured to allow access. Added a this stanza:

  <Directory "/var/www/da.nmilne.com/htdocs/public">
        Options FollowSymLinks
        AllowOverride None
        Order allow,deny
        Allow from all
  </Directory>

Naturally it feels snappier. Nice.

I spent several hours yesterday fighting with RubyGems – I’d even written a vitriolic post about it – but I … did something … and bam, like that it was gone. RubyGem is Ruby’s version of Perl’s CPAN. It’s got a very annoying trait – its prodigious use of memory. For each gem ( a gem is a Ruby module – like rails or hpricot for example ) RubyGem would load the spec into memory in order ( I’m guessing after reading a bunch of forum posts ) to build a dependency tree. On a 256MB slice host, this pushes you into swap hell. On a 512MB host it would use up to 68% of memory.

So what can you do but rent a bigger slicehost? Moe Sizlack said it best: “I’m choking on my own rage over here!”

Naturally, it’s been fixed. Today.

If only I’d done something more constructive yesterday. Like played COD.

Well, apparently I’ve not been consistent enough – an important quality when one bags out others for being inconsistent. So – I’ve updated all references to “Bookie” to the now correct “Booko”.

While I was at it I updated the scraping code for Fishpond who have again updated their site. At least they’ve updated it for the better. Compare and contrast the old and new Hpricot XPath code for grabbing the book title and author.

The Old:

book.title     = (doc/"table/tr/td/div/h1").inner_html
book.author    = (doc/"table/tr/td/p[2]/a/font/u").inner_html

The New:

book.title     = (doc/"h1#product_title").first.inner_html
book.author    = (doc/"p#product_author/a").first.inner_html

Much nicer!

• Bug fixed which stopped Booko remembering which Book searching place you last used. ( Bug reported by teh timo )
• Re-factored the functions which go and get the prices from the stores.
• Added an attribute to “Book” such that it now remembers when its prices were last looked up. Accidentally introduced a bug where any new book initially had no idea when it was last looked up and crashed the “Recently Viewed” method.
• Todo – better testing :-(

I was contacted by a chap from The Nile to let me know that they not only have an Affiliate program I could join, but they have an API to access their site! I’ve spent the last couple of days adding the affiliate links. You’ll see that several stores now appear to link to clixGalore – but fear not, the links will still get you to the page you’re after.

You’ll also notice there’s now a new store to search – The Nile. It’s the fastest of the search engines by a considerable margin. ( Which isn’t really fair – at least one of the sites which is searched is just a fancy screen scraper.)

Additionally, I’ve cleaned up the CSS and layout some. Hopefully you won’t have the horizontal scroll bar on your browser, regardless of how wide you open the window.

You’ll also notice that there’s a context sensitive help message in the top right-hand corner. This should make it clear what you can do at which ever page you happen to be on.

Try out the updated Booko and let me know what you think.

EDIT: Added a tool tip style mouse over for the Most Popular/Recent list so you can see the full title. Thanks to Niall & Riina for their help & ideas.

I added fragment caching to Booko the other day. I’ve added caching to four sections, the Recent Searchs, Most Popular, Shopping Cart and the section which displays the prices of a book.

Now, instead of calculating the most popular books every time someone views a page, we look to see if that part of the site has been created recently – if it has, use it again. The Shopping cart is another good example – it only changes when you add or remove a book from it. Now Booko avoids having to calculate the cost of your shopping trolley at all shops every time you view a page – it regenerates that part of the page only when you add or remove a book.

Kinda wishing I’d bench marked it before and after. I think it’s faster now – but that’s because I’m looking for it. It feels snappier. Anyway – let me know what you think, faster or slower?

I’ve updated Booko’s cart feature. You now get a link to Amazon ( com or co.uk ) which will prompt you to add multiple items to your Amazon cart and set me as the Associate. Should make buying a bunch of items way easier. Now I just need to figure out how to do the same thing with Fishpond.

Also, fixed a bug with a shop’s scrapping code.

Well, my first attempt at a shopping cart was pretty naf. There was no feedback to tell you that something was happening when adding or removing books from your cart, and every book added was listed individually.

I’ve added some feedback now for adding and removing from your cart, and added the number of items beside each item. It’s quite a bit better. Still looks crap tho.

Well, I’ve added another feature. You can now add books to a cart, which will calculate the price to buy all books in your cart at the various online book stores, including calculating shipping.

It might be slightly problematic for collecting referrer fees as people will probably not be clicking on each of the book links. I’ll have to have a think on that.

Next job is to make it look better.

I’ve just finished the biggest update to Booko since it went live. I never liked the fact that you had to wait for all the results to be found before displaying prices. Having each shop get updated was problematic as the end result wasn’t sorted by price. So I found a way to provide feedback while book sites are being searched and having the result sorted by price.

I’ve found a fixed a few bugs from the last release too around providing a link for the image in the search page.

So, go to town and let me know if you find any bugs. Any feature requests while I’m at it? I’m thinking of a personal list of viewed books or maybe a shopping cart type feature which calculates the cheapest place to buy all the books on your list – inclusive of shipping.

Well, it’s been a while, but it looks like someone (Tim you’re the only person who looks at cooking books) found a bug. Searching for a book which has an ISBN which is only available via Fishpond and not Amazon, means that you could find the book, but that it didn’t have a title, author or image. Well, I’ve fixed that now.

Also, you can now click on the book cover image to take you to the price lookup page.

Lastly, shipping price can now be more than just a single number – I can define rules for shipping. So, for example ( the only one I’ve found so far ) Fishpond have free shipping for books over $50 – this is now correctly accounted for.

Watir ( “Web Application Testing in Ruby” – pronounced water ) is a sweet little system used to automate browsers for the purposes of testing your site. Unfortunately, it’s for IE on Windows. Despite being odd in that most Ruby software is Linux or MacOS it’s also frustrating because I don’t use Windows. Luckily the Tubes have granted us two alternatives, SafariWatir and FireWatir. They achieve browser automation in very different ways.

[More…]

I’ve updated Booko to have a default new search engine: fishpond.com.au. Thanks to Tim Evans for finding bugs in it immediately.

Unlike Amazon, Fishpond ( or any Australian bookstore as far as I can tell ) don’t provide an interface for searching their site. Fishpond take the inconvenience one step further by not providing a standard way of finding a book by ISBN. You have to search for the ISBN, then get the Fishpond ID of the book. ( BTW – searching for an ISBN can return multiple results – the same book, but often with different prices. Apparently it’s due to having multiple suppliers for the same book. ) Aside from these annoying bits, Fishpond seems to have a very complete listing. They have an associate program too, so sending traffic to their sites may provide some return.

So, I wrote a searching module for Fishpond. Let me know if you find bugs.

The domain name bookie.com.au was taken, so I figured I’d carry on the tradition of naming the product like it was a person whose name was Book…. Booko. So you can now find the cheapest place to buy books at booko.com.au.

Found another bug today when one of the remote shops was misbehaving. The open-uri library was throwing exceptions I wasn’t catching.

The original version in the shop model:

def get_pricing(book)
    eval( self.price_function + "(book)" )
end

Replaced by this version:

def get_pricing(book)
  begin
    eval( self.price_function + "(book)" )
  rescue Errno::ECONNRESET => e
    logger.info(">>> Connection Reset checking #{self.name}")
  rescue OpenURI::HTTPError => e
    logger.info(">>> Connection Reset checking #{self.name}")
  end
end

Seems to work quite well.