Along with the new stores, I’ve fixed a bunch of little bugs and added more availability messages to prices (Just hover your mouse over the price – not super obvious I’ll grant 😉 )
Just pushed out some more updates to Booko:
- Moved Booko’s gem dependencies into environment.rb. Should have done this ages ago. (Read about it here)
- Started moving from Hpricot to Nokogiri. Hpricot was awesome, but it’s been choking on some websites and crashing the price grabbers.
- Started moving from timeout.rb to SystemTimer because I would like my timeouts to, you know, timeout. Sheesh.
- Updated the memcache-client to use SystemTimer for the same reason.
- Added more testing
Booko’s Fishpond parser was creating links to books on Fishpond which went into a never ending cycle of redirects, stop only when the browser was sick of it. Hopefully this should all be fixed now. A quick glance through the referrer section of Fishpond shows that this has been happening for at least a week. Sorry about that!
Recently I set up email notification for errors with help from this blog post. Since then I’ve been getting the occasional email from Booko telling me when things go wrong. It includes the URL accessed and a bunch of useful information for tracking down bugs. Highly recommended for anyone running a RoR site. Helps you catch all sorts of errors which you can then, naturally, write tests for. So as a result, I’ve fixed several bugs recently, introduced one that slowed search down by a factor, then removed it.
In addition, I’ve added shop number 30 – Text Books Oz who specialise, (this won’t be a surprise) in text books.
It’s a few days now since Booko was brought to its knees by an enormous (by Booko standards) spike in traffic. It’s an exciting way to have your web project fail – a massive number of users who are trying to use your site. Traffic’s dropped off to a fair degree, but is still far above pre-spike levels which is great.
The largest problem was the method the price grabber used to find books which need to have their prices looked up. It worked fine with just a few users, but didn’t scale at all well. This was fixed pretty quickly – it was making the site unusable – prices just weren’t getting looked up. The next bug was in the code which calculates the price of the books in the cart – this was doing great big table scans which was running the machine out of memory – it was a pretty simple fix once I’d identified it. It was disabled for a few days while I fixed and tested it. It’s now back in action. There have been a number of performance fixes too which should speed up the site.
Booko is on track to have received 10x the traffic it received on its busiest day ever. In an hour today it received more traffic than the busiest day previously in total. Several bugs have been found and squashed – naïve approaches to problems have been refined. Problems still remain however which I’m working on.
The biggest problem I’m having is the decreasing responsiveness and increasing memory usage of the application. Twice today the host rebooted due to lack of memory. Naturally I suspect a memory leak somewhere, but I’ve had very little success finding any problems. Of particular interest however is some weird strace output – lots and lots of this:
rt_sigreturn(0) = 4
--- SIGVTALRM (Virtual timer expired) @ 0 (0) ---
Some Google work leads me to think this may be thread related – there were reports of decreasing responsiveness, so I’ve removed threading support for the one place it’s used in the Rails app – the search field. Search will now be quite a bit slower, but hopefully, everything will continue to work.
Booko now calculates the savings you’ll make between the price of the cheapest store, and the most expensive store, including the percentage saved. Plus, I’ve setup affiliate links for Better World Books – not that that’s a feature as such, but it should hopefully help me pay for hosting Booko.
I’ve also discovered a couple of bugs with Booko which need fixin’, one online shop changed their site meaning I need to fix the price scraper, plus I upgraded my Ruby install without fully testing it. The ActiveRecord component of Ruby on Rails introduced database connection pooling – which is great but the default values were too small for Booko. Some of you may have discovered only half the prices being downloaded for books – sorry about that.
Thanks to Andrew from Online Bargain Megastore, I’ll hopefully be adding http://oo.com.au/ to Booko sometime soon, bringing the total number of stores checked to 19.
Well, I think I’ve found the bug that makes IE not work so well with Booko. The section of the page which displays the prices is updated via AJAX (with a GET post), but the first chunk of text for that section (with the spinning icons) gets cached by IE. The solution is to add an Expires header to the response. The actions which respond to GET requests got the following code:
headers["Expires"] = "Mon, 26 Jul 1997 05:00:00 GMT"
I should probably put a more correct value in there – but it’s always going to be set in the past at some point.