After Netflix? What next?
The Netflix competition is nearly concluded. We have learned that ensemble methods are the solution for more accuracy.
The recommender system community moves on. Immediate questions come to mind:
- Researchers continue to use the Netflix data set. Will it remain freely available?
- We need to study the effect of a 10% accuracy gain (measured by RMS) on user satisfaction. How do we go about it?
Otherwise, it seems that future research is bound by the available data. Models and theory alone have never had much of an impact on the field. Accordingly, there has been a surge of research on recommender systems using social network sites as a data source. Alas, social data is sparse, heterogeneous and ephemeral. Are all the low-hanging fruits gone?
Montreal, Canada 
Facebook
Friendfeed
LinkedIn
SlideShare
Delicious
Hi all
What about research on the upper performance limit of RC systems? If there is a good model, telling you the performance expectations for a given setup (’data topology’) according certain metrics and methods,we can slow down with the trial and error procedures. I think it would give us new inputs and insights..
Cheers
Marcel
Comment by marcel blattner — 10/7/2009 @ 6:22
Testing user satisfaction should be fairly straightforward. I’d use a before/after method, with same people in the two conditions. You can either directly ask people how good the recommendations are, or you can do a behavioural approach, where you ask people to click on the items that interest them and see if there are more items in the after than in the before.
Comment by Sylvie Noel — 10/7/2009 @ 7:56
@blattner
From a Machine Learning perspective, it is likely to be extremely difficult to compute such a bound on the accuracy. Unless you specify what type of algorithm you are allowed to use or can make some assumption about the data. Indeed, it is always possible that there is some unknown exotic structure within the data that can be exploited. How do you prove that there is no such structure? Hard.
From a practical user perspective, we can obviously bound the accuracy by how well human beings can guess their own ratings. Frankly, when I got back to my past ratings, I am sometimes surprised by how highly or lowly I rated certain items. However, this “accuracy” will depend on the user and its context. For example, maybe one person always give a rating of 3 to all items, no matter what. In this case, clearly, it is easy to make perfect predictions. Other users will be more frivolous, changing their minds from day to day. So, it is unlikely that there is some universal constant regarding the unaccuracy out there.
Comment by Daniel Lemire — 10/7/2009 @ 8:33
@lemire
. One big class of algorithms use an overlap based approach (common rated items between users, or common audiance shared by two objects in question). For such a class we could do some assumptions: the way people rate objects depends obviously on what prior information they have about objects they rate. Let’s take movies: everybody does a pre-selection and is influenced by many sources. So the probability density over the rating space is clearly influenced by that fact. And I would expect a right shifted (gaussian?) distribution over the rating space. Furthermore we could setup different ways (distributions) what movie will be rated by a user. From these simple facts only, we could do a small model and compute the expected error (i.e. RMS) for different levels of correlation.
I agree on your thoughts about “self-correlation” (see also http://www.apparentwind.com/navigation/videos.html section Reliability).
However, as a physicist I believe in simple, but controllable models. I think on can do pretty much
Now take something like jokes. You don’t have any prior information when somebody is telling you a joke. So I would expect a much broader distribution over the rating space. And indeed, when I compare movielens and jester distributions, they differ in that manner. I don’t think we could built a model telling us the whole story about every RC system. But I think we could do a good one, telling if a certain method makes sense in a particular situation (data).
cheers
Marcel
Comment by marcel blattner — 10/7/2009 @ 9:27
@blattner
Interesting take on the subject.
Comment by Daniel Lemire — 10/7/2009 @ 9:42
I haven’t been following this real closely, but are any of the winning algorithms actually cheap enough for Netflix to use in production?
Comment by Mr. Gunn — 14/7/2009 @ 23:30
@Gunn
This is a valid question. Netflix will probably not put these algo. in practice “as-is” due to scalability and business reasons.
Comment by Daniel Lemire — 15/7/2009 @ 8:28
All meaning is context dependent. Nothing has inherent meaning. Which leads me to think that meaning is in the relationship and interaction between things. It is dynamic and fluid rather than being concrete and decided.
This comment was originally posted on http://apperceptual.wordpress.com/)“>Apperceptual
Comment by mariana — 24/7/2009 @ 21:15
Yes, I like this a lot. Actually my learning style typically requires me to read many books on the same topic. I get stuck somewhere in a book, move on to the next one, and the next one, and the next one… then suddenly something clicks, and I can go back to the first book and push a bit further… and so on. I found one of the most useful things about my PhD supervisor was that he’d just push me to keep going, even if something felt impossible to understand.
This comment was originally posted on http://apperceptual.wordpress.com/)“>Apperceptual
Comment by Andy — 25/7/2009 @ 7:30
Unfortunately, our current culture puts too little value in robustness and diversity. Our dramatic financial failure (2008-2009) was due to few dominant viewpoints determining the behavior of most investors. Had we cultivated a few contrarian viewpoints, we might have planned better and avoided the crash.
Fortunately, (senior?) scientists can cultivate contrarian viewpoints without paying too high a price. But there is evidence that in some fields, too few scientists are willing to pay the price and go against the current. Or just “think for themselves”.
I think that this post is closely related to Martin’s paper Research productivity: some paths less travelled (see my blog post Research productivity: some paths less travelled). In particular, research is not necessarily about “truth seeking” and it requires creativity (thus diversity).
This comment was originally posted on http://apperceptual.wordpress.com/)“>Apperceptual
Comment by Daniel Lemire — 25/7/2009 @ 14:14
In mathematics, the structure of a manifold is encoded by a collection of charts that form an atlas. If understanding is a manifold, then each analogy is a chart, and all analogies together form the atlas of the manifold.
This comment was originally posted on http://apperceptual.wordpress.com/)“>Apperceptual
Comment by jofr — 26/7/2009 @ 4:58
I got this from peter gardenford
Six Tenets of Cognitive Semantics, page 160
i) Meaning is a conceptual structure in a cognitive system (not truth conditions in possible worlds)
ii) Conceptual Structure are embodied (meaning is not independent of perception or of bodily experience).
iii) Semantic elements are constructed from geometrical or topological structures (not symbols that can be composed according to some system of rules).
iv) Cognitive models are primarily image-schematic (not propositional). Image-schemas are transformed by metaphoric and metonymic operations (which are treated as exceptional features on the traditional views).
v) Semantics is primary to syntax and partly determines it (syntax cannot be described independently of semantics).
vi) Concepts show prototype effects (instead of showing the Aristotelian paradigm based on necessary and sufficient conditions).
This comment was originally posted on http://apperceptual.wordpress.com/)“>Apperceptual
Comment by mariana — 26/7/2009 @ 18:44
I got this from peter gardenford
For more on Peter Gärdenfors, see my blog post, Three Levels of Thought.
This comment was originally posted on http://apperceptual.wordpress.com/)“>Apperceptual
Comment by Peter Turney — 26/7/2009 @ 20:42
I prefer to apply Ulam’s description: (via Scott Kelso’s Dynamic Patterns [a great book!]). It’s more used in a visual perception sense but I think it is spot on for understanding meaning in text as well. Take a quick look at the bottom of this page starting at section ‘The Barrier of Meaning, Perceptual Dynamics I’ thru first paragraph of following page from the book (pages 188/189, thanks google!)
http://tinyurl.com/nayptt
This comment was originally posted on http://apperceptual.wordpress.com/)“>Apperceptual
Comment by Matt S — 28/7/2009 @ 17:17
I prefer to apply Ulam’s description: (via Scott Kelso’s Dynamic Patterns [a great book!]).
I agree with Ulam that the important thing is context and relations, not objects. See:
Context
Attributes and Relations: Redder than Red
Structural Realism
This comment was originally posted on http://apperceptual.wordpress.com/)“>Apperceptual
Comment by Peter Turney — 28/7/2009 @ 17:40
I prefer to apply Ulam’s description: (via Scott Kelso’s Dynamic Patterns [a great book!]).
I agree with Ulam that the important thing is context and relations, not objects. I’ve discussed this in several previous posts:
Context
Attributes and Relations: Redder than Red
Structural Realism
This comment was originally posted on http://apperceptual.wordpress.com/)“>Apperceptual
Comment by Peter Turney — 28/7/2009 @ 17:40
I prefer to apply Ulam’s description: (via Scott Kelso’s Dynamic Patterns [a great book!]).
I agree with Ulam that the important thing is context and relations, not objects. I’ve discussed this in several previous posts:
Context
Attributes and Relations: Redder than Red
Structural Realism
Symbol Grounding and Proportional Analogy
This comment was originally posted on http://apperceptual.wordpress.com/)“>Apperceptual
Comment by Peter Turney — 28/7/2009 @ 17:40