How Good Is Good Reads?

A while back, Andy McKenzie wondered where "the imdb of books" is. By which he means a site that presents an authorative ranking of books on the basis of a large sample of votes, combined with a clever algorithm for scoring these. One of the candidates he considered for such a role was the site Good Reads:

Upside: As far as I can tell, this is the largest "bookshelf" site with the most user ratings. Huge potential. Downside: They've made no attempt to publish a list of the highest rated books across the site! All I can ask is, what is holding you back, GoodReads editors? Qualms about alienating authors whose works won't make the list? Fears of being labelled imperialistic? These are both hogwash. Our time is scarce and in order to be informed consumers we need to know what the best books are. If you are worried about the arbitrariness of the minimum votes cut-off, then publish multiple lists with different scaling parameters. You will thank me later when the list gets out-of-control traffic. Indeed, a group of passionate GoodReads users recently called for such a list. To this valiant effort I can only say, Viva la Résistance!

As it turns out, Good Reads has now come round to providing such lists. Andy won't be to thrilled, however, and neither am I. That's because they're doing it all wrong. Two big problems: One, if you want to calculate a score on the basis of multiple votes, your measure needs to be metric. But Good Reads provides labels for each point of its five-point scale, and if voters take those labels seriously, the scale is decidedly nonmetric (the labels are "it was amazing", "really liked it", "liked it", "it was o.k.", "didn't like it"). Second, and worse, their scoring method is quite obviously not a variant of the ingenious Bayesian formula used by imdb. I don't know how exactly the Good Reads algorithm works, but it seems to give a lot of weight to the total number of votes, so you won't be surprised to find that their Best Books Ever list looks like this.

Now, I'll admit I have an Aspergery fascination with lists (including references sections in academic texts. Really.), but their main use is giving recommendations that I like. Good Reads has hence a chance to redeem itself because it's heavy on recommendations. That system doesn't look to hot either (it seems to be based simply on matches with books one has rated 3 or higher), but there's no need to speculate on its quality: let's make it empirical! Here are six books that the site has recommended to me, that hadn't been on my mental "to read" list and that looked interesting:

• The Man Who Was Thursday, by G.K. Chesterton
• The Speed Queen, by Stewart O'Nan
• If on a Winter's Night a Traveler, by Italo Calvino
• Hawksmoor, by Peter Ackroyd
• Snow Crash, by Neal Stephenson
• The End of the World News, by Anthony Burgess

In order to test how useful Good Reads is for me, I'll read five of those and see how much I like them. If the average rating will be 3 or above, I'll consider the site's recommendations a failure, as I can easily do that well without it. If the average rating will be 4 or above, I'll consider the system a big success. In between will be, well, in between. Ratings will be conceived of as metric. Half points are allowed. If I don't like a book enough, I'm going to put it down and rate it on estimated overall quality.

To wit, these are books I picked from the recommendations list on the basis of anticipated enjoyment. While it might seem "fairer" to pick a random selection of recommendations, such a test would have very low ecological validity: I want to know how useful the site will be to me in the future, and in the future I will not be picking books at random from the list.

Testing will be finished when I've rated five of the six books on the list. Given that I won't restrict my reading to these titles, and have other stuff to do as well, this will probably take a few months' time. I'll keep you posted. Anyone reading this is invited to play along.

And now, a literature review: I finished James Joyce's Dubliners today. The last two pages are really good.