Comments on: Productivity measures are counterproductive?

By: Daniel Lemire

Daniel Lemire — Tue, 16 Oct 2007 19:02:54 +0000

Cyril: journals can game the system by favoring articles citing a large number of their own articles. A journal I often review asks me every time whether the article cites a large number of articles from the journal itself. The message is clear: if you want your paper to be accepted, cite a large number of papers from the journal itself.

As for what happens when the concept of publication morphs due to the Web… A related question that I find very interesting is why we publish in the first place. If our goal is to be read by others, and people can more easily read us if they can freely download our paper, then why do we lock our papers behind a publisher’s firewall?

I am not worried, there will always be metrics. And it is a good thing. And competing for better score is a good thing too. But we can go too far and make the game boring for the next generation.

By: Cyril

Cyril — Tue, 16 Oct 2007 05:11:21 +0000

Do you have examples of journals that game the impact factor (and how they do it)?

And what happens to these citation-based metrics when the notion of “publications” slowly disappears with the widespread availability of preprints, notes, blogs, etc. on the web?

By: Peter Turney

Peter Turney — Tue, 16 Oct 2007 01:44:53 +0000

I think you would like Lee Smolin’s latest book, The Trouble With Physics. Firstly, it is a great survey of the state of modern physics for non-physicists. But more relevantly for your post, there is a common theme that the institutional structure that supports physics research is unintentionally suppressing originality and stifling progress.

By: Michael Stiber

Michael Stiber — Sat, 13 Oct 2007 16:18:27 +0000

“… the often quoted fact that over 50% of all married people cheat on their spouse…”

Is that like the oft-quoted statistic that 73% of all statistics are made up on the spot? :^)

Seriously, I wonder about that relationship between judging science via H-index and technical investing. In the latter case, investors ignore things like company financial fundamentals, what product they make, what industry sector they’re in, who runs the company, etc. and just focus on stock price and volume. They assume that all available information, plus market psychology and even insider information, is already captured in price and volume movements.

It seems to me that the key difference is that a measure like H-index is monotonic. We could make it non-monotonic by only including citations made in the last n years (to all papers written by the author). Would that make it too difficult to game the system?