(…) the machine learning community, much like the information retrieval community, generally prefers black box approaches, (…) If the goal is to optimize one-shot recommendations, they are probably right. But I maintain that the process of picking a movie, like most information seeking tasks, is inherently interactive, (…)

I disagree with him. Even for non-interactive recommendations, the Machine Learning community is off-track for two reasons:

• They fail to take into account diversity. In Information Retrieval, we know that if precision is high (all documents are relevant) but recall is low (few of the relevant documents are presented), then the system is poor. There is no such balance in collaborative filtering. Precision above all else is the goal. This is wrong. Diversity metrics must be used.
• They work over static data sets. A system like Netflix is not static and so, accuracy on a static data set might be a good predictor for real-world performance. The problem is intrinsically nonlinear. People will rate different items, and they will rate differently, if you change the recommender system. The feedback loop may work against you or in your favour. The effect might be large or small. As far as I can tell, I am the only one who keep pointing out this fundamental, but never addressed limitation of working over static data sets.

See also my post Netflix: an interesting Machine Learning game, but is it good science?

Disclaimer: I organized the ACM KDD Workshop on Large-Scale Recommender Systems and the Netflix Prize Competition along with people like Yehuda Koren. Yahuda is among the candidates to win the Netflix prize. I do not encourage the Netflix competition. I just do not think that it will solve our big problems.

Solution to economic crisis: sack everyone who has an MBA. (Tim Bray)

The naïve way to solve this problem goes as follows:

1. For each element…
2. check whether sqrt(x) is an integer.

This may prove too expensive since the square-root operation must be computed using a floating-point algorithm.

A better way is to look at the first 4 bits of each integer. If the integer is a square, then the first 4 bits must have value 0, 1, 4, or 9. If you have a random distribution of numbers, this means that you can quickly discard 3 out of 4 numbers.

It is not immediately obvious that you will speed up the retrieval because inserting this check will add some overhead. However, it doubles the speed according to John. It is even less obvious that checking the first 8 or 16 bits instead of just the first 4 bits, can help. John says it does not help in one C++ implementation, but it does in a C# implementation.

This sort of strategy is entirely general. The research question is how much work should you do on fast dismissal? Too much effort toward dismissing lots of candidates might be counterproductive. Too little and your performance might not improve optimally.

Recently I started to wonder whether we could make it multipass: you first dismiss a few candidates with a cheap test, then on the survivors you use a more expensive test and so on. For example, you first check the first 4 bits, and if you cannot dismiss the candidate, you check the next 4 bits and so on. It is not a surprising idea, but figuring out whether it is worth the effort is a research question.

To make my point, I have worked on fast retrieval under the Dynamic Time Warping (DTW) distance, a nonlinear distance measure between time series. The DTW does not satisfy a triangle inequality. It is commonly used as a pattern recognition technique when comparing time series. It was initially designed to compare voice samples, allowing for changes in voice rhythm.

Eamonn Keogh from UCI has come up with a simple but nearly optimal way to compute a lower bound to the DTW between any two times series, called LB_Keogh (named after himself). Just like in the John Cook algorithm, this lower bound quickly discards the false negatives. If you are interested, Eamonn has applied LB_Keogh to just about every time series problem you can think of.

I improved over LB_Keogh as follows. If LB_Keogh is not good enough (and only if it is not good enough), I compute a tighter lower bound (called LB_Improved). Surprisingly, in many cases, I can improve the retrieval time by a factor of two or more.

I have published my work as a software library, but also as the following paper:

Daniel Lemire, Faster Retrieval with a Two-Pass Dynamic-Time-Warping Lower Bound, to appear in Pattern Recognition.

This sort of work is much more difficult than it appears. I could have easily made my method look good by optimizing it, while leaving the competing methods unoptimized. By publishing my implementation, I go a long way toward keeping me honest. If I fooled myself and the reviewers, someone might find out by surveying my source code.

I actually take my research very seriously.

I like to distinguish abstract from concrete research. Concrete research is when you seek to obtain results in special cases. For example, an AI researcher may try to first understand how we can detect spam. Eventually he might move on to even more sophisticated tasks. In such a form of research, there are no overarching formal plans. You could say it is inductive, maybe. Researchers are often driven to this form of research because the deeper problems are simply too difficult to address directly. (I define a problem to be too difficult when you cannot make noticeable progress in a matter of months.) They hope for a breakthrough to an important problem to come as they work on a narrow issue.

Abstract research derives from a formal plan. Semantic Web is one such a plan. Tim Berners-Lee even drew diagrams early on of what the beast should look like. The research issues are clearly laid out. As a researcher you are tackling an extremely difficult problem, unsure whether you will ever make any noticeable progress. Researchers follow this path because they believe that only a focused effort in a definite direction can solve the difficult problems. Funding agencies love abstract research.

It might be a matter of biology, but my brain has always been much more productive in concrete research. I resist the inductive/deductive classification because it feels wrong. However, times and times again, working on a tractable, but possibly insignificant problem, has lead me to understand a deeper issue. When the problems are too big, my brain gets into circular and incorrect arguments. I need to chop down the problems to a manageable size. The problems need to be hard enough to push me to the limits, but easy enough that I can make weekly progress. Moreover, I cannot never know exactly what I will be doing a month later, as a researcher.

I will make a stronger claim: abstract research is never done. Researchers will give the illusion that they are working directly on some grand problem (like world hunger), but, in reality, they will work at a much smaller scale. And when a researcher solves a grand problem in what seems like a short time, and with few concrete possibly irrelevant steps, I attribute it to luck or lies.

See also my post my research process.

My business colleagues argue in favour of research methodology courses. Apparently, students need to learn how to conduct interviews and such. In any case, I then pointed out that my master degree did not contain any such course. One of my business colleague then said a deadly thing:

Of course, you got a technical master degree!

This got me really angry. Really, really, really angry. I do not think I ever got so angry in my life.

For the record, my master degree was in Mathematics at the University of Toronto. Is Mathematics technical? If technical is to have a “practical” connotation, I can tell you that none of my graduate courses were technical. Are fewnomials practical? I think not.

But the deeper implication was that anything having to do with Science was technical. That is, it deals with nuts and bolts. And I think that it is squarely wrong. From my view point, business is far more technical. And I ran my own business for several years. The business side of things was always the boring-but-easy component.

There is a distinct feeling in North America that business is king, and science & technology are things monkeys or foreigners can do. Yet, in my experience, it is a lot harder to design a usable web application than negotiate a business deal. I believe that India and China are getting a sweet deal by doing our science & technology while we focus on business. A very sweet deal indeed.

I think that Amazon, Google, Cisco, Microsoft and so on, thrive because many of their engineers have a deep knowledge of Computer Science. Kill the science and you kill the business.

But even if you discard science. Writing good source code is hard. Very hard. And it is not hard for technical reasons, not any more than painting, movie-making and sculpture are technical challenges.

In any case, I believe that North America is headed for a wall if it fails to recognize that its prosperity is due to culture, science and technology. And given that 40% of all students at my school go for a business degree, I am nervous.

See also my post Career Swings where I wrote:

I cannot believe that in 2015, we’ll all be lawyers, business managers, salesman, and medical doctors. I cannot believe that technology will stand still and mathematics beyond basic algebra will be a lost art. I cannot believe my two sons will have business degrees and make three times my salary by managing a bunch of underpaid Indian programmers.

It is what I would call a “temporal disparity” novel. Earth becomes suddenly surrounded in a temporal shield that slows time down for human beings. Alas, the Sun is aging very fast for the poor human beings. Are we going to die? Who is creating this field?

This is almost exactly the reverse story from Georges-Jean Arnaud’s La grande séparation (1971-1973). In Arnaud’s story, a planet has a similar temporal field, but it accelerates time on the planet. Even though the planet has primitive technology, it is constantly surveyed for any sign of technological development. Spin offers the counterpart story.

A temporal disparity leads to a technological disparity: a small band of savages can evolve into a technologically superior race while you are having coffee.

Pros

The novel is very good. The author writes with good scientific rigour. The writing is supported by repeatedly introducing new mysteries in every chapter… to keep you coming for more. The characters are believable and well drawn.

Cons

The author tried to limit the scope of the story to few characters, but not all of them are good characters. The writing style reminds me a bit of Card’s Ender’s game series. There is the extra smart kid who grows up to be the is the only one able to see through what is happening. I found this particular element of the novel irritating. A major catastrophe hits the Earth and only one man seems to be able to put it all together? I am a bit disappointed by how the author dealt with anything outside the Earth, including the Martians. He could have done so much more!

Sequels are upcoming.

• The Noisy Channel: Daniel Tunkelang, chief Scientist at Endeca. He works in information retrieval.
• Population of One: Sylvie Noël, research scientist at the government of Canada. She works in HCI.
• Ongoing: Tim Bray, director of Web Technologies at Sun Microsystems. He helped create XML and Atom.

It would be nice to be able to combine a true search engine with any database engine.

If you are willing to use Java, then Glen Newton from NRC has the solution: LuSql. It allows you to index with Lucene any database accessible by Java (through JDBC). He says it has been extensively tested. It is open source and free.

Recognizing analogies, synonyms, antonyms, and associations appear to be four distinct tasks, requiring distinct NLP algorithms. In the past, the four tasks have been treated independently, using a wide variety of algorithms. These four semantic classes, however, are a tiny sample of the full range of semantic phenomena, and we cannot afford to create ad hoc algorithms for each semantic phenomenon; we need to seek a unified approach.

I do not work in Natural Language Processing (NLP) per se, but this sounds like commoditization to me in the sense that you no longer need to design, learn and tweak custom algorithms. If you have enough data, you can do NLP after learning one (remarkably simple) family of algorithms. Peter Norvig might approve.

In the database research world, commoditization is already an accomplished fact. Database researchers have been wondering about their relevance for about ten years.

Peter might argue that in such a context, researchers should become bold and daring. Computer Science researchers should choose crazy problems.

Reference: Peter Turney, A Uniform Approach to Analogies, Synonyms, Antonyms, and Associations, Coling 2008 August 2008.

Source: Standard Deviations.

El Naschie is editor in chief of the journal Chaos, Solitons and Fractals. This journal is published by Elsevier, one of the biggest players in the science publishing business. But here’s where things get interesting: this journal also lists 322 papers with El Naschie as an author!

The journal has a high impact factor and a high rating by the Australian Academy of Sciences.

Source: Nielsen.

See also my posts The insane world of academic publishing, Why aren’t there more scientific breakthroughs? and Peer review is an honor-based system.

Imagine that you bought a couple of albums from Celine Dion and you liked them a lot. Then the best answer might be to suggest you buy all the other Dion albums. Or is it?

No. You do not want to optimize accuracy above all else. You need to balance accuracy and diversity. To any user, it is obvious. Researchers often prefer to ignore diversity because it is harder to measure.

Several people, me included, have argued in favour of diversity, but metric proposals were still missing. I have had on my to-do list to write a paper on measuring the diversity of recommender systems. Unfortunately, I cannot cope with more than a few projects at any one time. Fortunately, it looks like I will not have to write this paper. Zhang and Hurley have done a good job at it:

Zhang, M. and Hurley, N. 2008. Avoiding monotony: improving the diversity of recommendation lists. In Proceedings of the 2008 ACM Conference on Recommender Systems (Lausanne, Switzerland, October 23 - 25, 2008). RecSys ‘08. ACM, New York, NY, 123-130.

Do not be put off by the mathematics: they are a tad formal, but the right ideas are there, just read slowly sections 3 and 4. Basically, diversity is measured as the average dissimilarity between items. That is a standard form of diversity measure. This strategy to measure diversity is not novel, but to my knowledge, they are the first to apply it to collaborative filtering.

What is next? You are looking for a paper idea?

• Take Zhang and Hurley’s class of diversity measures, and apply them to existing recommender systems. Show that there is an accuracy-precision trade-off. All you need is a dissimilarity measure between items.
• Do user studies to prove people prefer a balance between diversity and accuracy.

Requirement: if you steal anyone of these ideas, you have to email me a copy of your paper once it is written.

What do I mean by sorting the rows? It turns out that finding the best row reordering is a NP-hard problem. You might as well use something cheap like lexicographical sort as a heuristic. Finding an equally scalable technique that work much better is probably impossible. Ah! But there are different ways to sort lexicographically!

We wrote a few papers on this issue including one for DOLAP 2008. Here are the slides of our presentation:

It appears that I am not alone. Here is Zeilberger on why errors in 99.9% of mathematical papers are no big deal:

Most mathematical papers are leaves in the web of knowledge, that no one reads, or will ever use to prove something else. The results that are used again and again are mostly lemmas, that while a priori non-trivial, once known, their proof is transparent. (Zeilberger’s Opinion 91)

I agree that getting a formula wrong is no big deal. As long as we have a diverse set of researchers working on different problems, important errors are not going to survive long. Fortunately.

Cheating and biases is what we need to worry about.

In a previous post, I explained that only two factors make indexing possible:

• your index expects specific queries
• or you make specific assumptions about the data sets.

In other cases, you are better off just scanning the entire data set.

What makes database indexes work?

As far as I know, there are only 6 strategies that make indexes work. By combining them in different ways, we get all of the various existing schemes. (I would love to hear your feedback on this claim!)

1. You expect specific queries: restructure your data!

Suppose you know ahead of time that you will only need to select some of the elements in your data set. Then you can taylor an index for such queries and thus avoid scanning much of the content. For example, an inverted index in full-text search will select which documents contain the various keywords. Instead of working with all documents, you will only worry about the ones matching at least one keyword. Indexing a column with a B-tree or a hash table is another scenario where you try to immediately go to the relevant rows in a table.

Of course, if I look for all documents containing the words “the” and “will”, and want to know how many there are and what is their average length, such a form of indexing will not help.

2. You expect specific queries: materialize them!

Another commonly used strategy is view materialization. If 10% of all visitors on Google type in the word “sex”, they might as well precompute the result of the query. In Business Intelligence, if you can expect your users to mostly care about results aggregated over weeks, months or years, it makes sense to precompute these values instead of always working from the raw data. Alternatively, you can materialize intermediate elements that are needed to compute your results. For example, even if people do not need data aggregated per day, precomputing it might be useful for computing weekly numbers faster.

This form of indexing tends to work well to address the most popular queries, but it fails when people have more specific needs.

3. You expect specific queries: redundancy is (sometimes) your friend

When you do not know exactly which queries to expect, you can try to index the data in different ways, for different queries. For example, you could both use a B-tree and a hash table, and determine at query time which is the best evaluation strategy. You might even determine that the best way is to forgo the indexes and scan the raw data!

4. Use multiresolution!

Suppose that you look for specific images, but you may still need to scan 50% of them. An index that would point you to only the relevant images might not be effective. Instead, you should try to quickly discard the irrelevant candidates. What you could do is create thumbnails (low resolution images). Then you can dismiss quickly the images that are obviously not a good match. Naturally, you can have progressively finer resolutions.

Database indexes often bin values together. For example, if you could bin all workers earning between $10,000 and $30,000, then all workers earning between $30,000 and $50,000, and so on. If you are looking for workers earning between $40,000 and $45,000, you can first find all works that are in the $30,000-$50,000 bin, and then look up their actual salaries, one by one. You can adapt the bins either to the data distribution or to the types of queries you expect.

For more examples, see my post How to speed up retrieval without any index?.

5. Your data is not random: compress it!

Most real-world data is highly compressible. By compressing the data, you can make it so that your CPU and IO subsystem process less data. However, you have to worry about bottlenecks. Too much compression may overload your CPU. Too little compression and most time will be spent in loading the data from disk. Two techniques are often combined to get good results out of compression: sorting and run-length encoding.

6. In any case: optimize your code

You should be using cache-aware and CPU-aware indexes. Be aware that comparing two bits together may take nearly as long as comparing two integers. Be aware that jumping all over the place (as in a B-tree) takes longer than processing the data by tiny chunks.