All posts by Dan Milne

Booko – now with better fragment caching!

Booko

One of the more processor intensive sections on the Booko site is the “Most Viewed” section.  This section is generated by finding all the books which have been viewed in the last 7 days, and ranking them by their frequency. This query can take up to 1.6 seconds according to Benchmark.realtime {}. I had set up fragment caching for this, but every time a book is viewed, Booko notes down the ISBN and the time, then, because the data has changed, goes and recalculates the most popular book again.  So the caching really isn’t helping anyone who’s looking at books – it may help someone who’s on the front page, but that’s it. 

Ideally I wanted this section of HTML to have a time based expiry but that functionality isn’t part of the regular Rails caching system and I don’t want to install plugins to do this for me. After a bit of searching, Google found me a very relevant page which discussed the issue I was facing.

The goal is for a time based expiry – let’s say 15 minutes or 900 seconds. When you call the cache function, you pass in some data to help it create a unique key for the HTML being cached.  If that name already has some cached HTML associated with it, that HTML will be rendered rather than generating the HTML again. Previously, I was just naming this cache “Popular” – the code looked like this:
[sourcecode language=’ruby’]
<% cache("Popular") do %>
# Generate the list of most popular books
# Print out the popular books
<% end %>
[/sourcecode]
Instead, we’ll do this:
[sourcecode language=’ruby’]<% cache(["Popular", Time.now.to_i / 900 ].to_s) do %>
# Generate the list of most popular books
# Print out the popular books
<% end %>[/sourcecode]
This version will append a number to the cache key which will change every 900 seconds, or 15 minutes. Here’s what it generates as a key:

>> ["Popular", Time.now.to_i / 900 ].to_s
=> "Popular1369073"

So, now we have a new key for the cache every 15 minutes. When the key changes, the cache system won’t be able to find the HTML and will regenerate it.   This would be a terrible idea if you’re using file based caching BTW, since you’d be constantly creating new cache files.  However, if you’re using memcached, like Booko does, you’re home free. Memcached automatically expires objects from its cache using an LRU algorithm. So, with any luck, Booko’s performance should improve.

Bookstore features to help out Affiliates

Booko, Shops

I was talking with Jeremy (the brains behind Booktagger) about what kind of features an online bookstore could have to help affiliates do a better job. After documenting some, I thought they might make an interesting blog post. So, good features for a book site to have, to help affiliates do a better job:

1) An API.  The API should accept two types of query – ISBN and a text based search. They could be separate URLs or they could be combined into a single URL and just intelligently recognise if an ISBN or search string has been submitted.
1a) If an ISBN has been submitted, return an XML formatted document including ISBN, title, author, publisher, format, availability, link to cover image and price.  If the ISBN isn’t found, return an XML document explaining that there were 0 matching records.
1b) If a search term has been submitted, do a search and return a list of matching items.  The items could either be complete – multiple records of the type above in 1a, or an abridged version, requiring the user to resubmit the individual ISBN to get a detailed version of the book.

2) A method of allowing an affiliate to add multiple items to a users carts.  To see an example of this, add a few books to Booko’s cart – make sure the books are all available at one of the sites which include this functionality such as the Nile or an Amazon. Once you’ve added a few items, click the “the Nile” or “Amazon ..” link in the cart and you’ll be taken to a page where you confirm that you’d like to add all the following items.

3) Easy to read and generate links to books which can be created based on the ISBN. Like, http://site.com/books/isbn/123123123.php

4) An Affiliate program.  The affiliate program should make it easy to create dynamic links to any book in store. Perhaps within the URL or as an argument ?refid=123 style.

These features would make it a book store easy to work with, and helps the end user’s experience by integrating the shopping carts.

More shops and various updates

Booko, Shops

I’ve added DStore to Booko this morning. Two other shops, Emporium Books and Glee Books were also added earlier in the year bringing Booko to a total of 25 stores. 

Thanks to Anish for a couple of suggestions I’ve implemented including Favicon to help those of you with more browser tabs than desktop space and clearer page titles.

Booko review

Booko

Thanks to my referrer logs, (well, thanks to Piwik) I found another great review for Booko. The site, “Frock You – The Blog” had this to say:

There’s millions of sites out there (aside from the mega giganticAmazon; Fishpond, The Nile, Booktopia and even Dymocks are all good online stores, though you sometimes have to wait awhile to receive your goodies) and I have just discovered a site to take all the hard work out of cyber-browsing: Booko.com.au is a search-and-compare site with direct links to the books you want to buy! I have just been testing it and give it two very enthusiastic thumbs up. Price, shipping, and total price inc shipping are all displayed in AU$, which makes it quick and easy to make the best purchasing decision possible!

Traffic Aftermath

Booko

It’s a few days now since Booko was brought to its knees by an enormous (by Booko standards) spike in traffic. It’s an exciting way to have your web project fail – a massive number of users who are trying to use your site. Traffic’s dropped off to a fair degree, but is still far above pre-spike levels which is great.

The largest problem was the method the price grabber used to find books which need to have their prices looked up. It worked fine with just a few users, but didn’t scale at all well. This was fixed pretty quickly – it was making the site unusable – prices just weren’t getting looked up. The next bug was in the code which calculates the price of the books in the cart – this was doing great big table scans which was running the machine out of memory – it was a pretty simple fix once I’d identified it. It was disabled for a few days while I fixed and tested it. It’s now back in action. There have been a number of performance fixes too which should speed up the site.

Booko on the Net

Booko

Looks like Booko’s been mentioned out there on the Tubez today resulting in a 10x increase in my previous maximum visitors.  Oz Bargain Blog and Lifehacker are today’s top two referrers – Hello Guys! Sorry for the slowness – I’m doing my best to get things sorted out.  For those of you of a graphical nature, here’s a visual representation of the traffic for Booko.

traffic

Awesome!

What a Day

Booko

Booko is on track to have received 10x the traffic it received on its busiest day ever. In an hour today it received more traffic than the busiest day previously in total.  Several bugs have been found and squashed – naïve  approaches to problems have been refined. Problems still remain however which I’m working on.

 The biggest problem I’m having is the decreasing responsiveness and increasing memory usage of the application. Twice today the host rebooted due to lack of memory. Naturally I suspect a memory leak somewhere, but I’ve had very little success finding any problems. Of particular interest however is some weird strace output – lots and lots of this:

rt_sigreturn(0) = 4
--- SIGVTALRM (Virtual timer expired) @ 0 (0) ---

Some Google work leads me to think this may be thread related – there were reports of decreasing responsiveness, so I’ve removed threading support for the one place it’s used in the Rails app – the search field.  Search will now be quite a bit slower, but hopefully, everything will continue to work.