Comments on: Are we destroying research by evaluating it?

By: Vellino

Vellino — Fri, 20 Apr 2007 16:40:26 +0000

I remember when we were trying to publish material on interval constraints - about which there had been a sum total of 2 publications in both math and C.S. and having all the trouble in the world getting anybody to understand what we were trying to say and why it was interesting.

Failing the peer-review method we went about on a person-person campaign to explain our ideas and did some “intellectual lobbying”… after which, these ideas started spreading and more stuff got published by good people.

I think your point Daniel is a very good one. The problem is that if the subject matter isn’t already in the realm of “normal science” (an accepted problem / solution space in the community) then it’s very hard to actually get new stuff out with the existing peer-review process and incentives. Not that it’s always bad, as Peter points out. It’s just not always good from the point of view of creative, new ideas.

By: Jean Robillard

Jean Robillard — Mon, 16 Apr 2007 20:14:12 +0000

This is a very serious issue. What is at stake is not only the validity of concepts and theories, but also the social ethics that lay behind any social system based on evaluation and judgement on or about one’s works and actions. But the fundamental aspect of peer reviewing is its ability to reproduce parts or totality of a science, just by assessing ideas as conform to accepted models (Kuhn coined the term “paradigm” at the end of the sixties, which describes this reproductive function of normal science, in the perspective of science’s social organization). I think that if researchers tend to view their productions as salami that can be sliced ad infinitum, or almost as an ever ending story to surf on, it is easy to see that the reason of this behaviour is the fact that the peer review system is probabilistic in its structure: with few reviewers picking up ideas in an “urn” like stock of ideas (the total amount of submitted articles added to the total amount of actually published articles which is a priori lesser than the first sum), it is likely that the ideas that will be picked up will tend to be conditioned by their correspondence to the total sum of available ideas. For new ideas and theories are rare, and difficult to develop but more importantly to grasp and understand, and this is also why so many people write commentaries on other’s ideas, not texts offering fresh ideas (I am not even talking of revolutionary ideas, ones that change the course of a discipline or domain…). So, what is there to be done? First, I think that the reaffirmation of research’s fundamental ethical principles should help (not the morality of science as when it comes to use animals or human subjects for instance); and this can only be done by introducing a compulsory formation in ethics. And reviews that don’t respect these principles should be denounced. No one would then be associated with an organization that doesn’t respect deontology and ethics. And to reproduce only what is already known is certainly not a good principle from which derive some editorial guidelines.

By: Geoff Wozniak

Geoff Wozniak — Mon, 16 Apr 2007 18:40:07 +0000

This is something that has bothered me about a possible career in research. When I started graduate work, I was pretty sure I wanted a research position. As time has progressed, I’ve found that the ideas I want to explore don’t fit well into the “salami” model of publishing, which has greatly altered my career path.

Not only that, I find it difficult to bring myself to publish the same thing over and over. Some people I’ve seen get incredible traction out of a single idea and the papers written by them on the subject rarely build on the idea; instead, they rephrase it. I can’t say I like that approach very much. It doesn’t sit well with me.

By: Peter Turney

Peter Turney — Mon, 16 Apr 2007 16:59:36 +0000

There is no doubt that the current system has flaws, but I don’t think “leave them alone” is the answer. Let’s ignore conferences for a moment and focus on journals. In my experience as a reviewer of journal papers, most papers are neither accepted “as is” nor rejected; instead, they are accepted with major revisions or rejected with encouragement to resubmit. In my opinion, the second version of a paper is always quite a bit better than the first version, and it is usually accepted. I believe that many of my own papers have been very much improved by this reviewing process, so I would not want to see it eliminated. Also, papers that are rejected even after revision tend to appear later, in some less prestigious journal. In summary, I’m sympathetic to these criticisms, but “leave them alone” is not the answer.