Anonymity And "peer Review"

[Guest]

Transformer has it right.

I think I described this in the Ketchum thread at some point, but as a reviewer, it's every bit incumbent upon me to back up my assertions with literature (and sometimes data!) when I've got a major issue to raise with a submitted manuscript. It's NOT just a matter of my opinion. When that does happen, the editor has his own means to gauge the quality and maturity of the review - not the least of which will be the comments from the other reviewers! I'll sometimes compose multiple pages of text in preparing a review, and I'm actually really proud of the work I've done on some of them, especially when I see that much-improved paper I had reviewed actually make into the journal one day.

If the editor has no qualms about the reviews submitted, the last check on review quality comes from the author. Authors are provided with the comments of all reviewers as well as the synthesized decision letter prepared by the editor. The author absolutely has the right to challenge the review, especially in the case of some suspected bias against the author or the work. One of the reasons I'm underpublished compared to some of my peers is that I'm not given to pounding my first and bullying editors into publishing my stuff. I have colleagues who do engage in such behaviors and, sadly, are often as not rewarded with a reversal of the editor's decision.

I usually remain anonymous in my reviews, but not always. I have a friend who does not wish to be anonymous, and he intentionally signs every review. In general, anonymity is essential to the process, especially for younger reviewers who could very easily be intimidated by senior scientists.

[Guest]

Keeping anonymity is a way to prevent issues such as bribery or intimidation at the far end of the scale. Anonymous review is common practice and is done to protect the process. The editors are responsible for managing the reviewers and making sure everything is on the up and up. This is also a way to prevent a buddy-buddy system whereby a reviewer can give a favorable review in return for a favorable review. Remember, reviewers also submit papers for consideration. Saskeptics and Parnesus's numbers are a good case in point, they have submitted papers and also do reviews. It would be unseemly at the least to be put in a position where they would want to be lenient with someone that will later be doing a review of their own work.

I understand your concern Mulder. However, the policy of anonymity is a good one and is to the great benefit of the review process.

There is a phrase in politics that applies here "Sunshine is the best disinfectant." Corruption, favoritism, etc cannot flourish in a transparent process with publicly accountable participants.

[Guest BFSleuth]

Saskeptic, when you review a paper are you also privy to the identity of the author?

[Guest]

^Yes, though I don't know that that's the norm in other disciplines. I'd almost rather I didn't know, and I generally don't consider it at all in my reviews. I just refer to them as "the authors" and rarely pay any attention to who they are.

The reason we're given the authors' identities is so we have the opportunity to turn down the request if there's a conflict of interest. For example, I wouldn't want to review any of my friends' papers if for no other reason than it might create the impression of bias.

Remember for this discussion that authors are very often requested by the editor to provide 2 or 3 names of potential reviewers. I always do whether I'm asked or not.

[Guest BFSleuth]

Well that actually gives me more confidence in the peer review process, to have it normally be a blind review. While Mulder has a point in regard to "sunshine is the best disinfectant", that applies well to the political process where anonymity is impossible and not desirable in a democracy. However, in terms of peer review anonymity allows reviewers to focus on the science and not the politics. Kind of the ideal model of "justice is blind", to put it in political terms.

[southernyahoo]

I agree with your last point for sure. If the paper is published in a reputable journal then the science and conclusions have been accepted by her peers and I will be be happy to point out to my friends that my interest in sasquatch has not been a waste of time.

Me too!

[Guest]

Well that actually gives me more confidence in the peer review process, to have it normally be a blind review. While Mulder has a point in regard to "sunshine is the best disinfectant", that applies well to the political process where anonymity is impossible and not desirable in a democracy. However, in terms of peer review anonymity allows reviewers to focus on the science and not the politics. Kind of the ideal model of "justice is blind", to put it in political terms.

That's only valid if you have basic faith in some sort of intrinsic objectivity in Science.

I've read too much about the politics of, and biases in science to be that trusting.

[Guest BFSleuth]

That's quite the change for your avatar, Mulder! Yes, I understand your basic arguments in regards to distrusting science in general. I might recommend a good read for you (if you haven't read it already): Where the Wasteland Ends, by Theodore Roszak. Classic book discussing some of the points you have been making. Science is a human activity with prebuild blinders in terms of allowed observations. However, with attention to making sure the process is done correctly it is in my opinion one of the best tools in the history of mankind for understanding our world and universe. Having multiple people consider and review a problem (or a research paper) is most likely to yield good results, as long as the process is largely devoid of politics (as you have so eloquently noted).

[Guest Peter O.]

Thanks for the book tip, BFSleuth, just ordered it. Epistemology is a complicated subject I don't know much about. Historical materialism, on the other hand,...

[Guest BFSleuth]

Thanks, Peter. It was a very formative read for me back when it first came out.

[Guest]

Who's reviewing the reviewers science? Do you think they could blow smoke at the editors? If we accept that your argument works both ways, then let us remember that if and when this paper publishes, the quality of the science would then be verified.

I think you answered your own question. I can't claim to know anything above the norm about scientific peer review for publication and widespread acceptance. Everything I write at work is indeed peer reviewed but all those documents are internal and are only intended to be used as protocol by our staff. Clearly the level of scrutiny provided is much more rigorous for publication in a journal like Science or Nature. That rigorous review is very clearly for the benefit, not the detriment of the author.

It also benefits the publication and I've got to imagine the most respected publications are the one that provide the most rigorous review. I've never written anything that was distributed nationwide for ten of thousands of my cohorts to read and critique. Everything I write is internal. But no one in my position wants decisions and potentially millions of dollars to be spent on reccomendations we make when those decisions are not best that could have been made via other methods or other rationalizations. As a result, I don't seek out my collegues who provide little to no critical feedback. I solicit input from individuals whom I know will critique all my conclusions, rationalizations and justifications and who demand multiple rewrites when I've failed to write with absolute clarity. Why? Because it ultimately makes everything I produce better and gives me more confidence that the final product is based on the best available information and that my rationalizations are sound. That portion of the peer review process I'm certain is homogenous regardless of whether the author is writing something as earth shattering as a lifesaving as a cure for cancer or something as mundane as an article in the local paper about an annual charity event.

I guess the problem I'm seeing is that most proponents that seem to have an emotional horse in this race look at the peer review process prior to publication acceptance as a purposeful and malicious roadbook for Ketchum. If she is a true scientist, rigorous review is something I'm pretty certain wants and appreciates.

I also think that most proponents see publication as the "end game". In reality, it really is little more than the first step in the review process. If you don't believe that go back and look at the Science article where Fitzpatrick and company from Cornell proclaimed that the Ivory-billed Woodpecker still is alive and well in North America. I've got little doubt Fitzpatrick and company popped a few corks when they got word Science was going to run the article. I've also got few doubts that they probably wish the nasty skeptics that exposed the fallacy in their wing beat analysis were on the original prepublication review panel.

As far as anonimity goes, how many reviewers do you or any of the other Ketchum supporters think any Journal would be able to field if the default practice was to reveal the identity of the reviewers? Think about legal implications if said review ends up in essence endorsing something that proves to be invalid post publication.

[Guest]

That's quite the change for your avatar, Mulder!

I'm on a Robotech kick at the moment...

Yes, I understand your basic arguments in regards to distrusting science in general. I might recommend a good read for you (if you haven't read it already): Where the Wasteland Ends, by Theodore Roszak. Classic book discussing some of the points you have been making. Science is a human activity with prebuild blinders in terms of allowed observations. However, with attention to making sure the process is done correctly it is in my opinion one of the best tools in the history of mankind for understanding our world and universe. Having multiple people consider and review a problem (or a research paper) is most likely to yield good results, as long as the process is largely devoid of politics (as you have so eloquently noted).

I might try to find that one.

I recently read this one, and found it instructive:

http://www.amazon.co...s/dp/0316023787

Mind you, I have no problem with (as you put it) "having multiple people consider and review a problem (or a research paper)". I simply feel that the public is best served with a transparent (to the extent possible) process with open accountability.

Put another way: if, as several have maintained, the data is all (as it should be), then why wouldn't a reviewer be more than happy to put their name on the review?

I guess the problem I'm seeing is that most proponents that seem to have an emotional horse in this race look at the peer review process prior to publication acceptance as a purposeful and malicious roadbook for Ketchum. If she is a true scientist, rigorous review is something I'm pretty certain wants and appreciates.

No one is decrying the act of peer review at this point (at least I'm not). I'm not happy with the idea that such a potentially groundbreaking paper is possibly being reviewed by what amounts to a "star chamber".

As far as anonimity goes, how many reviewers do you or any of the other Ketchum supporters think any Journal would be able to field if the default practice was to reveal the identity of the reviewers? Think about legal implications if said review ends up in essence endorsing something that proves to be invalid post publication.

No more or less than those the submitting scientist/scientists face.

Also, a lot would depend on the nature of the problem. Did the reviewers make an honest, understandable error, or were they shoddy and sloppy with their work? If the later, wouldn't they deserve to face some sort of repercussions, legal and otherwise?

Edited April 12, 2012 by Mulder

[Guest]

Did the reviewers make an honest, understandable error, or were they shoddy and sloppy with their work?

Did the authors make an honest, understandable error, or were they shoddy and sloppy with their work?

[Guest]

^I already stipulated that the submitters are taking a risk professionally...

Why should the reviewers be free of the same risk?

[Guest]

Did the authors make an honest, understandable error, or were they shoddy and sloppy with their work?

In the case of the Clovis VS. Solutrean it does on the surface appear as tho plenty of folks stuck with the ''status'' quo instead of following the evidence and it's soon to ALL be overturned. Tons of ''out of place'' artifacts were found and dismissed as irrelevant. So Mulder's statement is a fair one given that it happens more than the layperson is aware across many fields of study.

If anyone wants ''tenure'' they toe the party line. Better science be dammed since rocking the boat is a career killer. Altho I'll qualify that by saying it's the same in almost every career field.