Peer Review

If you follow the various science-vs-pseudo-science arguments, you will likely have heard someone bring up "peer review", the process by which research papers or other documents are reviewed by experts in the field in order to see if they pass muster and essentially make sense. Peer review is often held up as one of the most important institutions in the sciences and in research fields in general (including history, literary criticism, etc.), and so, as I am currently going through the peer review process for a paper that I have submitted for publication, I thought it might be worth describing and de-mystifiying the process a bit.

To start with, I submitted my paper to the editors of the issue of the journal in which it will be published. They looked it over, determined that my basic arguments and ideas stood up to scrutiny and therefore were publishable, and returned it to me with comments that suggest that I read other books and papers relevant to the topic but not included in my references cited, and that I consider expanding the discussion of certain topics within the paper. For example, the paper discusses interaction between villages in prehistoric California, but only discusses the landscape in terms of geographic obstacles and pathways (valleys, mountains, etc.) and resources available (seeds in grassland, acorns in oak woodland, fish and shellfish at the coast, and so on). Implicit in the discussion was the matter of landscape as a social phenomenon (village territories, family land holdings, do coastal and inland people identify themselves differently?) but it was not explicitly addressed in my original paper, so it was requested that I address that more directly. Also, because of the work schedule that I have had over the last 18 months, there have been two books and a few articles that had escaped my attention but which addressed similar issues, so I was directed towards those and asked to include discussion of and references to them in the paper. They then suggested that I needed to explain why my conclusions were necessarily better or more likely than other possible conclusions derived from the same data, as well as qualify a few weaknesses in the data that weren't as clearly flagged in the original version of the paper, making my argument seem less robust but my paper far more honest (one of the most important services and editor can provide). The editors also commented on the clarity (or occasional lack thereof) of some of my writing, helping me to identify places where I know what I'm trying to get across, but it might be confusing to a reader.

So, with these recommendations in hand, I set about revising the paper. I incorporated their comments (or, in a couple of cases, added material to the paper in such a way as to explain why the type of criticism behind the comments didn't apply to the discussion at hand), gave it another go-over myself and altered some things that I thought needed to be changed but on which the editors did not comment, and handed it back over.

I then received another round of comments. These ones were less about the substance of the paper (as would be expected as this had already been discussed) and more about the clarity of the writing, ensuring that I was saying what I wanted to say and that the paper would be clear to readers unfamiliar with my topic. I incorporated these comments, and re-submitted the paper again. It was bundled up with the other papers for the issue and sent to the journal's main editor.

Okay, so I should explain the multiple layers of editors here. In this particular case, the two editors to whom I initially sent the paper have been trying to get a book published for a couple of years, and a group of archaeologists, myself among them, have been working on material to contribute to the book. When two different book deals failed to materialize, they spoke with the editors of a prominent archaeology journal who agreed to publish the material written for the book. So, now the book editors have become essentially guest editors for this issue of the journal. However, the journal's primary editor still has to do his duties. Also, it should be mentioned that these editors are not literary/journalistic editors - though many journals have such people on staff - they are respected archaeologists with strong research records who have the breadth and depth of knowledge necessary to provide useful criticism.

So, the journal's editor has received the papers. He is in the process of sending them out for external review. What this entails is that he will identify a few archaeologists who are knowledgeable about the geographic area, cultural group, or research topic of each paper, and send the papers to them to see what criticisms they have of them. I will then be sent these comments and will use them to create final revisions to the paper. After all of that is done, the paper will finally be published.

As you can see, the primary activity done by reviewers and editors in this process is criticism. Criticism, as both a concept and a word, gets abused alot in our society - people who are critical are considered undesirable company, critical comments are frowned upon, and we assume that professional critics (of film, literature, etc.) hate the medium that they criticize. But the truth is that criticism is important and valuable, and much of what people call criticism is not criticism at all but rather abuse. Criticism is not the automatic nay-saying or attacking of a subject, but it is rather the thoughtful consideration of a subject, an evaluation that takes into account both strengths and weaknesses, flaws and merits. Some objects, arguments, and ideas are so deeply flawed that there is little to no room for positive criticism, and a much smaller number are so virtuous that it is difficult to make negative criticisms, but most fall somewhere between these two extremes.

To a researcher, negative criticism, while sometimes hard to take, is vital. If I have made a flawed argument or am misunderstanding data, it is important that I know about that, and it is to my benefit that a fellow researcher point it out. It is also vital to a research discipline, as without critical assessment of data and arguments, an "anything goes" attitude develops in which the agenda or ideology of the author can easily take precedence over the reality of the subject being studied. In an environment that prohibits criticism, there is little possibility for advancing study as there is no clear criteria by which the validity of an idea or argument can be measured, and consistency and coherency of arguments, adherence to data, and clarity of thought decay. Criticism is a good thing, a vital thing, and a major force in advancing an area of research.

It is also important that criticism come from multiple experts. Any one expert will have their own view on a given subject, ideally motivated by their valid interpretation of evidence but potentially also motivated by external pressures (their employer, their ideological leanings, their religious beliefs, etc.). When one introduces multiple experts with different views and from different backgrounds, however, non-data driven views will be diluted and can be more easily parsed and dealt with by the author of the work being reviewed as well as the editor(s) of the publication. This doesn't always work, of course, but it works pretty well most of the time.

So, basically, being a responsible and legitimate researcher means leaving yourself open to criticism, and taking criticism into account. That is what the peer-review process is really all about. Research is not a feel-good sand box game in which we promote self-esteem, but a serious investigation of a subject. It can be, and usually is, alot of fun. But is can also be tough, and if you are not able to sustain a bruised ego or give up a cherished idea that doesn't stand up to scrutiny, then you are not cut out for research.

This is not to say that peer-review is a fool-proof enterprise that only ever benefits a field of study. It's a human activity, and as such is not perfect. Sometimes, bad* papers get through, and bad* arguments are not rooted out. But these are the exceptions, and most of the time peer review makes sure that published research is rigorous and well-supported**. Ultimately, this strong effort to adhere to reality as it can be observed rather than as we would like it to be is the difference between scholarship and pseudo-scholarship. And it is telling that the frequent complaint of pseudo-scholars is that they big bad "establishment" researchers criticized them.





*And note, I mean "bad" not as in "I don't like it" or "it makes me uncomfortable" but rather in that "it doesn't make any sense and takes liberties with observable reality."

**Which is different than saying that it is definitely right. All good research is based on information available at the time that it is produced, and new information may disprove existing ideas, no matter how well established, and many well-accepted ideas (stable-state universe, recent [within 2000 years] population of the Americas, and non-moving continents) have been overturned by new information...again, in contrast to pseudo-scholarship where no amount of data can change the minds of proponents (homeopathy, anyone?).