Yes. I am. And I agree with you. A friend of mine, Peter Turney, who also reads this blog, might answer something along the line that an incomplete metric is better than no metric at all.

I think that the rating prediction accuracy is a vastly overrated metric for evaluating recommender systems. This metric ignores all sorts of aspects of recommendation that can add or detract from the quality of recommendation: novelty, transparency, resistance to hacking and shilling, diversity all contribute to the quality of a recommendation.

I agree 100%. I have written about this on my blog in the past.

The canonical wisdom for CF systems is that more data is better – and if you are just predicting ratings, then I agree, but I think we’ve seen many examples of recommendation in the wild where more users result in poorer recommendations. Just look at the diversity of recommendations at sites like Digg or Last.fm. As their user base goes up, the diversity of recommendations goes down, the recommender hacks goes up, and the overall recommender experience gets worse. Look at the top 10 tracks at last.fm this week. As the size Last.fm user base has increased it has become a very homogenized music site.

Very interesting comment. And I agree.

I am interested in how you use ‘accuracy’ here – since there is no ‘right’ answer for a recommender, accuracy is hard to measure, let alone improve. I suspect that you are really talking about predicting ratings (such as one can do for the Netflix prize).

I think that the rating prediction accuracy is a vastly overrated metric for evaluating recommender systems. This metric ignores all sorts of aspects of recommendation that can add or detract from the quality of recommendation: novelty, transparency, resistance to hacking and shilling, diversity all contribute to the quality of a recommendation.

The canonical wisdom for CF systems is that more data is better – and if you are just predicting ratings, then I agree, but I think we’ve seen many examples of recommendation in the wild where more users result in poorer recommendations. Just look at the diversity of recommendations at sites like Digg or Last.fm. As their user base goes up, the diversity of recommendations goes down, the recommender hacks goes up, and the overall recommender experience gets worse. Look at the top 10 tracks at last.fm this week. As the size Last.fm user base has increased it has become a very homogenized music site.

http://www.last.fm/music/+charts/track/

(well, sorry for the rant, thanks for the interesting and provocative list).