unless it is reviewed and accepted by a journal with an established track-record for quality standards, most busy researchers will not invest their scarce time and resources into reading and using it

Just like none of them are reading my blog?

(Lots of famous people comment on my blog, beside you: http://www.daniel-lemire.com/blog/my-readers/ And my blog is not exceptional in any way.)

Should we also each individual hand-test our mushrooms, to see whether their are toxic o safe to eat, rather than relying on experts, because they are human, and might have occasional lapses?

You are misrepresenting my point. And by your arguments, wikipedia ought to be highly unreliable compared to expert-reviewed solutions, which it isn’t.

My quote about Wikipedia was directly from Daniel Lemire…

No.

I did not write or imply anywhere that people should replace journals with wikipedia.

By your arguments, busy people would not bother with wikipedia. They would prefer the more rigidly reviewed alternatives (of which several are free).

My answer to the rest of your comments took the form of a new blog post:

Become independent of peer review
http://www.daniel-lemire.com/blog/archives/2009/10/26/become-independent-of-peer-review/

The contention is not that there cannot be the occasional rare unrefereed gem.

The contention is that (1) unrefeeed postings are not a viable way of reporting, following and using the vast daily body of scholarly and scientific research; (2) post-posting “corrective” commentary comes too late (if it comes at all); and (3) no one has tested and demonstrated the capability and of any alternative to classical peer review to generate a research literature that is of at least the quality and useability as the one we have now, in our 25,000 peer-reviewed journals.

That’s the literature that the Open Access movement is trying to set free from fee-based access barriers — not from peer review.

This is a blatant misrepresentation of the position, and Stevan Harnad should know better. There are many other forms of online publication other than wikipedia.

DL: “wikipedia has been shown to be more reliable than the traditionally-reviewed alternatives”

Really? So researchers have abandoned their peer-reviewed journals and are now relying instead on the anonymous gremlins of Wikipedia, or ought to? (That’s an interesting stance…)

DL: “Traditional peer review is not the way we determine what is wrong and what is right in science”

But it’s our best bet for filtering in advance what is worth investing our time to read and taking the risk of trying to use and build upon…

DL: “’Empirical evidence on expert opinion shows that it is extremely unreliable.’ http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0020124 ”

Less reliable than inexpert opinion, or no binding, answerable advance expert filtering at all?

DL: “You’d be silly to trust your health to peer review.”

Trust it instead to Wikipedia, or web chat groups?

DL: “What you can trust, a lot more, is independent verification by several teams of the same findings.”

It’s whether or not enough of that verification has already happened that peer review is meant to determine.

DL: “If many people have independently verified something, then chances are that it is right.”

That’s what the collective, cumulative, self-corrective process of scientific publication is meant to converge on in the long run. But newly published research findings are not the long run, and active research cannot wait for everything to be eventually checked out in the long run. Peer review is an advance quality-control filter tomaximize in advance the probability that findings are sound enough to warrant reading and using now.

DL: “that a bunch of anonymous professors read a paper and think it is good enough to appear in Nature… does not say anything about the truth of the paper.”

It says a lot more than posting raw papers on the web and waiting to see if there turns out to be anything wrong with them.

DL: “There is no way reviewers can know whether the experiments were done properly.”

Even less can the reader and user know.

DL: “Often [reviewers] don’t even read carefully the papers, and sometimes they are not even qualified to read them.”

True. And your point is…?

Editors and the journal’s reputation and track-record are answerable for the reliability of their peer review. Different journals have different quality standards, but if those standards fall (not occasionally, because of lapses in human nature, but systematically, because of declining rigor), so does their readership.

DL: “There is no way reviewers can verify the truth. Reading over a paper is not the same as independently verifying it.”

There is even less way for potential readers and users to verify the truth. Do you propose that everyone must first try to independently verify every purported new finding rather than continue to rely on the advance filtering provided by classical peer review? But why on earth would you propose that?

(Should we also each individual hand-test our mushrooms, to see whether their are toxic o safe to eat, rather than relying on experts, because they are human, and might have occasional lapses?)

DL: “the reason I still go through traditional peer review is that this is the only way that I can force people to criticize my research.”

Another reason is that unless it is reviewed and accepted by a journal with an established track-record for quality standards, most busy researchers will not invest their scarce time and resources into reading and using it (unless you are famous).

DL: “It seems to me that we have Open Access already, for any author who wants it.”

You miss the point. It is *users* who want Open Access, and only 15% of authors are *providing* it (except if mandated). Hence users do not have it.

DL: “We are building Open Scholarship right now, but we still don’t have all the pieces. (Including the part where there is no good alternative to traditional peer review for many of us.)”

I can only repeat: You are dreaming if you think that the already overloaded specialists in each field have nothing better to do than to review everything that is posted (unbidden, and at the risk of having their recommendations unheeded) — when they can barely keep up with the formal requests to peer review papers from the editors of journals who seek their expertise and will see to it that their advice is heeded…

I’ll make a new blog post out of this.

I’d still rather have external quality control — and not only through hearing about how many users poisoned themselves from trying out a new drug without external quality testing. (Sorry for the shrillness of the analogy. but do we really rate
scholarly/scientific reliability so much lower than health-safety?)

How can you say this when wikipedia has been shown to be more reliable than the traditionally-reviewed alternatives?

Traditional peer review is not the way we determine what is wrong and what is right in science, it is also a recent “innovation” (very recent in the history of science), and it is certainly not reliable:

http://www.daniel-lemire.com/blog/archives/2009/01/09/the-purpose-of-peer-review/

Here is another reference:

“Empirical evidence on expert opinion shows that it is extremely unreliable.”

Reference: Why Most Published Research Findings Are False, http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0020124

To put it bluntly, you’d be silly to trust your health to peer review. What you can trust, a lot more, is independent verification by several teams of the same findings.

Basically, there is strength in numbers. If many people have independently verified something, then chances are that it is right (but it may still be wrong!!!).

But simply because a bunch of anonymous professors read a paper and think it is good enough to appear in Nature… that does not say anything about the truth of the paper.

There is no way reviewers can know whether the experiments were done properly. Often they don’t even read carefully the papers, and sometimes they are not even qualified to read them. There is no way reviewers can verify the truth. Reading over a paper is not the same as independently verifying it.

That is not to say that feedback from your peers is not extremely valuable.

There is nothing stopping scholars from freeing their scholarship from peer review today — except possibly, as Stephen Downes pointed out, their career prospects.

Stephen has not freed himself from peer review.

And I don’t advocate that anyone frees himself from peer review.

Heck! All my research papers even go through traditional peer review. But I have no illusion that it makes them “right”.

And the reason I still go through traditional peer review is that this is the only way that I can force people to criticize my research. (I’m not famous or infamous enough to be able to pull a Perelman and just have the whole world scrutinizing my arxiv submissions).

We have “Open Scholarship” already, for any author who wants it; the barriers are to the access that you find so “short-sighted”…

It seems to me that we have Open Access already, for any author who wants it. We are building Open Scholarship right now, but we still don’t have all the pieces. (Including the part where there is no good alternative to traditional peer review for many of us.)

DL: “I refuse to be limited to traditional peer review.”

No one is blocking you. (But my access to peer-reviewed articles is being blocked by toll-barriers until they are made OA).

DL: “The high standard I adhere to is to produce high quality research outputs. The rest is unneeded artificial scarcity.”

The same is probably said by drug-manufacturers — but I’d still rather have external quality control — and not only through hearing about how many users poisoned themselves from trying out a new drug without external quality testing. (Sorry for the shrillness of the analogy. but do we really rate scholarly/scientific reliability so much lower than health-safety?)

(By the way, I’d say the scarcity of high quality research is not artificial but a natural consequence of traits that tend to be distributed guassianly or even logarithmically in the population: even the criterion for “high” is relative to the distribution…)

DL: “As an aside, OA presently offers drastically limited access, by your own arguments, stated on your own blog. Open Access is to be limited to reseach articles, and should exclude books (your own books are not OA), conference proceedings and so on.”

OA offers nothing. Authors offer drastically limited access to their own research, because most are not yet self-archiving their work to make it OA. That’s why the mandates are needed. But it is the mandates that are limited to refereed research articles, not OA, which any author can provide to whatever work (of their own) they wish to make freely accessible online (just as you can post whatever you wish, without having to submit to peer review).

The problem is the toll-access barriers, and authors’ failure to free what they want to give away already (namely, all their refereed journal articles, written only for usage and impact, not for income). But for most authors that does not mean their books (or music, or films, or software).

I have not written any books, only edited a few. The one book (edited by Okerson and O’Donnell) the lion’s share of whose contents I wrote is completely free online. So are all the chapters I have written in edited books, including the ones I edited myself. But again, I self-archived those facultatively; it is premature to speak of self-archiving content that most authors don’t want to give away free today when we have not even mandated self-archiving the content that they all do want to give away free today.

DL: “If I am stranded without an acquisition budget, I can only access a small fraction of the science. I do not get the full picture. That is especially true in Computer Science where 80% of the research is published in conference proceedings which are excluded from Open Access policies.”

Daniel, forgive me, but what you are saying makes no sense. Authors can self-archive anything they want to give away free online. The purpose of the institutional policies is to mandate self-archiving journal articles, which they all already want to give away free, but are not doing it. Many may also want to give away their book chapters, and some may even want to give away their books. None of that can be mandated, because it may conflict with what the author does and does not want to give away, but it is not “excluded,” any more than self-archived unrefereed content is excluded. (By the way, most Green OA mandates also cover refereed conference article.)

DL: “By preserving clique-based peer review and locking up books and proceedings, we are failing to move toward Open Scholarship which should be our only goal.”

You are conflating the problem of access with the problem of quality control. They are orthogonal, and insisting on freedom from peer review as part and parcel with freedom from toll-access neither makes sense nor does it do the cause of OA any good. (There is nothing stopping scholars from freeing their scholarship from peer review today — except possibly, as Stephen Downes pointed out, their career prospects.)

DL: “Open Scholarship is the noble long term goal we must work toward. Open Access policies are short-sighted because they fail to address the real problem which is our very own culture and reward system.”

We have “Open Scholarship” already, for any author who wants it; the barriers are to the access that you find so “short-sighted”…

They can also be used to demonstrate that Wikipedia cannot work. After all, an online encyclopedia where anyone can edit the content is going to be dominated by “the obsequious or contentious”?

And what might happen if anyone can post a research paper on arxiv, without an editor’s stamp of approval? Surely, most work will be of low quality? No?

Well. Gregory Perelman happened, that’s what. Arguably the most important mathematical result of the century was posted on arxiv, without any trace of formal peer review.

You see, once you lift the artificial barriers to publication, there is little incentive to publish nonsense for serious people. People who want to get ahead are forced to focus on the quality of their work once there is no artificial scarcity in the number of publication slots.

You can no longer stand out by how many papers you published or by where you published them. You are left to worry about producing work that people will want to read, work that will prove useful.

Artificial scarcities are inefficient. They lower the overall quality. No matter how we justify them.

I refuse to be limited to traditional peer review, that is filter early, as the high standard. We inherited this from another era and the time has come to question it.

The high standard I adhere to is to produce high quality research outputs. There rest is unneeded artificial scarcity.

As an aside, OA presently offers drastically limited access, by your own arguments, stated on your own blog. Open Access is to be limited to reseach articles, and should exclude books (your own books are not OA), conference proceedings and so on. If I am stranded without an acquisition budget, I can only access a small fraction of the science. I do not get the full picture. That is especially true in Computer Science where 80% of the research is published in conference proceedings which are excluded from Open Access policies.

By preserving clique-based peer review and locking up books and proceedings, we are failing to move toward Open Scholarship which should be our only goal.

Open Scholarship is the noble long term goal we must work toward. Open Access policies are short-sighted because they fail to address the real problem which is our very own culture and reward system.