Peer Review

If you follow the various science-vs-pseudo-science arguments, you will likely have heard someone bring up "peer review", the process by which research papers or other documents are reviewed by experts in the field in order to see if they pass muster and essentially make sense. Peer review is often held up as one of the most important institutions in the sciences and in research fields in general (including history, literary criticism, etc.), and so, as I am currently going through the peer review process for a paper that I have submitted for publication, I thought it might be worth describing and de-mystifiying the process a bit.

To start with, I submitted my paper to the editors of the issue of the journal in which it will be published. They looked it over, determined that my basic arguments and ideas stood up to scrutiny and therefore were publishable, and returned it to me with comments that suggest that I read other books and papers relevant to the topic but not included in my references cited, and that I consider expanding the discussion of certain topics within the paper. For example, the paper discusses interaction between villages in prehistoric California, but only discusses the landscape in terms of geographic obstacles and pathways (valleys, mountains, etc.) and resources available (seeds in grassland, acorns in oak woodland, fish and shellfish at the coast, and so on). Implicit in the discussion was the matter of landscape as a social phenomenon (village territories, family land holdings, do coastal and inland people identify themselves differently?) but it was not explicitly addressed in my original paper, so it was requested that I address that more directly. Also, because of the work schedule that I have had over the last 18 months, there have been two books and a few articles that had escaped my attention but which addressed similar issues, so I was directed towards those and asked to include discussion of and references to them in the paper. They then suggested that I needed to explain why my conclusions were necessarily better or more likely than other possible conclusions derived from the same data, as well as qualify a few weaknesses in the data that weren't as clearly flagged in the original version of the paper, making my argument seem less robust but my paper far more honest (one of the most important services and editor can provide). The editors also commented on the clarity (or occasional lack thereof) of some of my writing, helping me to identify places where I know what I'm trying to get across, but it might be confusing to a reader.

So, with these recommendations in hand, I set about revising the paper. I incorporated their comments (or, in a couple of cases, added material to the paper in such a way as to explain why the type of criticism behind the comments didn't apply to the discussion at hand), gave it another go-over myself and altered some things that I thought needed to be changed but on which the editors did not comment, and handed it back over.

I then received another round of comments. These ones were less about the substance of the paper (as would be expected as this had already been discussed) and more about the clarity of the writing, ensuring that I was saying what I wanted to say and that the paper would be clear to readers unfamiliar with my topic. I incorporated these comments, and re-submitted the paper again. It was bundled up with the other papers for the issue and sent to the journal's main editor.

Okay, so I should explain the multiple layers of editors here. In this particular case, the two editors to whom I initially sent the paper have been trying to get a book published for a couple of years, and a group of archaeologists, myself among them, have been working on material to contribute to the book. When two different book deals failed to materialize, they spoke with the editors of a prominent archaeology journal who agreed to publish the material written for the book. So, now the book editors have become essentially guest editors for this issue of the journal. However, the journal's primary editor still has to do his duties. Also, it should be mentioned that these editors are not literary/journalistic editors - though many journals have such people on staff - they are respected archaeologists with strong research records who have the breadth and depth of knowledge necessary to provide useful criticism.

So, the journal's editor has received the papers. He is in the process of sending them out for external review. What this entails is that he will identify a few archaeologists who are knowledgeable about the geographic area, cultural group, or research topic of each paper, and send the papers to them to see what criticisms they have of them. I will then be sent these comments and will use them to create final revisions to the paper. After all of that is done, the paper will finally be published.

As you can see, the primary activity done by reviewers and editors in this process is criticism. Criticism, as both a concept and a word, gets abused alot in our society - people who are critical are considered undesirable company, critical comments are frowned upon, and we assume that professional critics (of film, literature, etc.) hate the medium that they criticize. But the truth is that criticism is important and valuable, and much of what people call criticism is not criticism at all but rather abuse. Criticism is not the automatic nay-saying or attacking of a subject, but it is rather the thoughtful consideration of a subject, an evaluation that takes into account both strengths and weaknesses, flaws and merits. Some objects, arguments, and ideas are so deeply flawed that there is little to no room for positive criticism, and a much smaller number are so virtuous that it is difficult to make negative criticisms, but most fall somewhere between these two extremes.

To a researcher, negative criticism, while sometimes hard to take, is vital. If I have made a flawed argument or am misunderstanding data, it is important that I know about that, and it is to my benefit that a fellow researcher point it out. It is also vital to a research discipline, as without critical assessment of data and arguments, an "anything goes" attitude develops in which the agenda or ideology of the author can easily take precedence over the reality of the subject being studied. In an environment that prohibits criticism, there is little possibility for advancing study as there is no clear criteria by which the validity of an idea or argument can be measured, and consistency and coherency of arguments, adherence to data, and clarity of thought decay. Criticism is a good thing, a vital thing, and a major force in advancing an area of research.

It is also important that criticism come from multiple experts. Any one expert will have their own view on a given subject, ideally motivated by their valid interpretation of evidence but potentially also motivated by external pressures (their employer, their ideological leanings, their religious beliefs, etc.). When one introduces multiple experts with different views and from different backgrounds, however, non-data driven views will be diluted and can be more easily parsed and dealt with by the author of the work being reviewed as well as the editor(s) of the publication. This doesn't always work, of course, but it works pretty well most of the time.

So, basically, being a responsible and legitimate researcher means leaving yourself open to criticism, and taking criticism into account. That is what the peer-review process is really all about. Research is not a feel-good sand box game in which we promote self-esteem, but a serious investigation of a subject. It can be, and usually is, alot of fun. But is can also be tough, and if you are not able to sustain a bruised ego or give up a cherished idea that doesn't stand up to scrutiny, then you are not cut out for research.

This is not to say that peer-review is a fool-proof enterprise that only ever benefits a field of study. It's a human activity, and as such is not perfect. Sometimes, bad* papers get through, and bad* arguments are not rooted out. But these are the exceptions, and most of the time peer review makes sure that published research is rigorous and well-supported**. Ultimately, this strong effort to adhere to reality as it can be observed rather than as we would like it to be is the difference between scholarship and pseudo-scholarship. And it is telling that the frequent complaint of pseudo-scholars is that they big bad "establishment" researchers criticized them.





*And note, I mean "bad" not as in "I don't like it" or "it makes me uncomfortable" but rather in that "it doesn't make any sense and takes liberties with observable reality."

**Which is different than saying that it is definitely right. All good research is based on information available at the time that it is produced, and new information may disprove existing ideas, no matter how well established, and many well-accepted ideas (stable-state universe, recent [within 2000 years] population of the Americas, and non-moving continents) have been overturned by new information...again, in contrast to pseudo-scholarship where no amount of data can change the minds of proponents (homeopathy, anyone?).

Karen Armstrong's Short History of Myth

I have an ambivalent relationship with the writings of Karen Armstrong, and not just because of her name (she and I share a surname, and my mother’s name is Karen). Those books and articles of hers that I have read are full of fascinating ideas, and I think that one of her recurring themes – that modern religion has placed claims of factual accuracy of religious texts over the value of these texts as explanatory of what it means to be human – is quite correct. However, she also has a tendency to look towards past religion and mythology with rose-colored glasses, claiming that prior to the beginnings of science the question of whether or not Biblical stories were literally true was seen as irrelevant (no doubt there are many who did feel this way, but the history of the Church’s persecution of heretics indicates that they were concerned about the literal truth of church teachings) and that modern society has problems that past societies didn’t because of the lack of compelling myth (there may be some merit to the basic idea, but Armstrong draws conclusions from it that are demonstrably false).

First, a definition. The term “Myth” is used here, and in the book being reviewed, not in its popular way, to indicate a false story being propagated, but in the sense that anthropologists and folklorists tend to use it – a story that holds symbolic meaning beyond the literal meaning of the story. Myths can refer to events that never occurred (the Paiute culture hero Helldiver retrieving the Earth from the bottom of a cosmic sea) or they can refer to events that are real but have become imbued with meaning beyond their original significance (the Emancipation Proclamation being viewed as a symbol of the equality of all people, when its actual purpose was mired in the more mundane politics of the day).

In A Short History of Myth, Karen Armstrong provides the reader with an overview of the development of mythology from our ancient ancestors up through the present day. The book lives up to its name, and is very brief, meaning that Armstrong deals in broad themes over wide spans of time, and tends to generalize. In and of itself, this isn’t too much of a problem, Armstrong acknowledges it, and it does serve to provide a framework for studying the development of mythology, so that is all to the good.

However, Armstrong’s mistakes begin in the first section on our ancient ancestors. She over-generalizes, implying that archaeological evidence of particular customs indicates that they were universal when they weren’t necessarily, and she draws rather broad conclusions that, while consistent with the evidence, are not necessarily supported by it. For example, she discusses flexed burials (where a body is bunched into a fetal position) and the placement of grave goods as indicating that the body was being prepared for rebirth in the next life, and holds that this is evidence for a general belief in a particular sort of afterlife among ancient cultures. Such an interpretation is not unreasonable, but it ignores such facts as that burial in the flexed position requires the excavation of a smaller grave, requiring less work on the part of the survivors, that the burial of goods may represent something other than the need for these goods in the next life (among a mobile population, for example, it may represent the need to dispose of goods so that the survivors are not carrying extra weight during their yearly rounds), and among sedentary populations such graves with grave goods may have served to legitimize land holding (this is the land of my ancestors and I can prove it! Look in this hole!) rather than veneration of the dead or a hope for a better afterlife.

Additionally, type of inhumation varies over both time and place (flexed burials, burial of an outstretched individual, burial of different body parts in different locations, cremations, etc.), as does the practice of depositing grave goods (some cultures don’t leave grave goods, others leave an abundance, some leave only certain types, and some leave all manner of household and daily task tools). So, it’s best not to assume too much based on mortuary evidence.

Another problem is that Armstrong seems to assume that there are basic over-arching myths that will be found expressed in remarkably similar ways across all cultures. Again, this is not an unreasonable notion, but it is one that requires more exploration rather than simple assertion that all hunter/gatherer and early farmer mythology is essentially the same. There are common themes, even some common images, that show up time and again across time and space, BUT there are also definite differences, some of them quite important that show up even between neighboring groups. Although I don’t think it is her intention, it is easy to read Ms. Armstrong’s books and come away with the impression that there was a single ancient religion practiced by all early peoples, a notion that is complete nonsense. In fact, the ancient religion that Armstrong discusses is heavily biased in favor of what many European archaeologists and folklorists have attempted to reconstruct for Europe and (to an extent) Africa, but it may have been quite alien to ancient Asia, Australia, and the Americas.

Still, to be fair, she is working with the accounts of a specific group of archaeologists and folklorists working in Europe, and the errors and mis-steps that she makes in this section of the book are likely due to the fact that the people she is relying on for sources tend to do the same. In other words, this is probably a problem more with her sources of information than with her reporting. Nonetheless, it would do her well to be more critical when examining this information, while some of it is quite solid, much of it is little more than conjecture, and in this book it is reported as fact. Of course, that assumes that she wants to be factually accurate, rather than writing a polemic disguised as a scholarly work, and this latter idea might be a more accurate description of her intention.

The next sections deal with changes to mythology that occur as people become sedentary, and begin to develop farming and then civilization. Although some of the same errors are present here as well, these sections are generally better, both because the quality of the archaeological data improves (it’s easier to see patterns in the remains of a long-term permanent settlement than in the remains of a hunter-gatherer encampment), and because the eventual invention of writing allows records of both the myths and the rituals surrounding these myths to be kept. Her discussion of the Axial age, in which mythology had to undergo a change and become more abstract and philosophical, as opposed to something that can be easily applied to everyday life, is interesting and does help to explain much about modern religions. However, I am not qualified to judge the accuracy of her assertions in this section, as my training and experience in prehistoric archaeology does not cover it, nor does my knowledge of history from the Roman Empire on. It’s a blind spot in my education, and it is possible that Armstrong commits the same types of errors here that she did in the earlier portions of the book.

As the book approaches the modern day, Armstrong laments that so many of the problems of our modern day – a great social ennui, genocidal regimes, and weapons of mass destruction – are due to our disconnect from myth. It’s not that we lack myth, but rather that we fail to understand it correctly that leads to violence on a mass scale and widespread social ills. This, she offers, is due to a commitment to scientific rationalism that both negates myth, thus alienating the rationalists from the meaning of the myth and causing destructive despair or misunderstandings, and creates a destructive literalist backlash from those who consider the myths sacred.

It is here that the author begins to spew bullshit in massive waves. First off, it is clear from the section on early hunter gatherers that Armstrong either doesn’t quite grasp the role of myth in the lives of these people, or else disregards what it was in favor of her pet hypothesis that humans are moving away from myth to their detriment. She is correct in stating that humans need myth, but they need it in ways both profound and profane, and early myth was as much about orienting humans to the world around them in a practical sense as in a psychological sense. In the practical sense, we now have much more trustworthy methods of orienting ourselves, and so we are left with the need of psychological orientation – what Joseph Campbell spent so much time dealing with.

Despite Armstrong’s claim, modern humans have not become detached from our use of myth for this purpose. Consider US history – we make ready use of the story of the founding of the nation, of events such as the Civil War and the Great Depression, of documents such as the Emancipation Proclamation not simply as historical facts, but as evidence of who we are as a people, what our destiny is, and how we should interact with the world around us. Armstrong may be right that many of us have lost the old myths, or in the case of Biblical literalists lost the point of the myths even if the myths themselves are retained, but we have formed new myths and use them in much the same way – and this is the process that is occurring at all times throughout the history of humans as a species.

Armstrong would likely concede that point, but in discussing groups such as the Nazis and their myths of Aryan superiority, she claims that we have become unmoored from our use and understanding of myths and that this is responsible for the evils of groups such as the Nazis and Stalin’s regime. But that this is absolutely wrong should be obvious to anyone who gives the idea a moment’s thought.

Genocidal war, hatred for the outgroup, irrational and homicidal claims of ethnic superiority, and internicene warfare even within ethnic groups are nothing new. Ethnographers have discovered this amongst hunter gatherers living isolated from the rest of the world, amongst primitive farmers throughout only recently contacted by those outside of their culture, and among all other social and technological organizations up through and including industrial and post-industrial nations. Archaeological evidence indicates strongly that these tendencies have been present for much, if not all, of human history, and tend to increase as populations grow and come into more regular contact with each other. And, of course, the written record is filled with such atrocities – the destruction of Carthage, the wars that led to the fall of Nineveh, and of course the genocidal campaigns approved by God in the Bible (which Armstrong would know about, being a former nun and all) – all committed by groups who Armstrong argues would not have had such violent dysfunctions of behavior because they were in touch with their myths.

Really, it’s not the myths that have changed, or even our direct relationship to them, it’s the technology and population size that allows us to carry out atrocities of a level not possible, but certainly dreamed of, in the past. The notion that getting back to our mythological roots will somehow put an end to, or even reduce the severity of, this violence is absurd and shows either an ignorance of history or, more likely in the case of Karen Armstrong, an intentional distortion of the facts to support a pet hypothesis that simply doesn’t hold water.

Not surprisingly, this book has gotten bad reviews from historians and anthropologists, but generally good reviews from academics who do not deal as critically with the past and from many people in the general media.

If you want to hear what the professional reviewers have to say about this book, go here.