This feels weird to me too.
I refer, of course, to an essay by Andrew Schlafly titled Quantifying Openmindedness. In it he proposes, in essence, that since we have scales that supposedly measure IQ** or Body Mass, we should be able to develop a way of measuring openmindedness. Essentially, he is arguing for the development of a new psychometric technique. I see no necessary reason why this should be impossible and, indeed, think it is an interesting idea. So how might we go about it?
Well, first, we need to define the thing that we're trying to measure. What is "openmindedness"? Schlafly defines it as follows:
By "openmindness" I mean a genuine willingness to consider the evidence before rejecting an idea. I do not mean tolerance, or a rejection of absolute truth, or skepticism. Openmindness means here what the dictionary says: "receptive to arguments or ideas."
So, the aspect that Schlafly is most interested in is the simple willingness to consider evidence before judging an idea.*** So, for all intents and purposes, we are interested in assessing whether or not the process of reacting to information has a certain character but are not interested in the outcome of that process. Put another way, we don't care whether or not the subject makes a correct determination about the evidence but only whether or not they consider it.**** I might modify this slightly to make it more amenable to testing by simply stressing that openmindedness entails a willingness to seriously consider the views of those with whom you disagree. Having reached, more or less, a definition of the concept, we can now turn to operationalization, or converting the concept into measurable indicators.
Again, Schlafly beats us to the punch suggesting a scale for openmindedness that is based on the idea that openmindedness is the converse of closedmindedness. In other words, we can measure unwillingness to consider ideas and assume that the less unwillingness is present, the more openminded a person is. This is a fairly strong assumption but we'll take it as a given. Using this perspective, in the scale below, each time you report that an item is impossible you get 1 point. The higher your score, the lower your level of openmindedness. Note, as well, that some items are reverse-coded. Ignore, as well, that some questions should be answered "yes" or "no."*****
Do you think it is impossible that more widespread gun ownership reduces the rate of crime?
When President Ronald Reagan told Mr. Gorbachev to tear down the Berlin Wall, did you think that it was impossible for the Berlin Wall to be torn down?
Did you think, or still think, that the Strategic Defense Initiative is impossible?
Do you think that it is impossible that the Shroud of Turin is authentic?
Do you think that there must be a material explanation for remarkable homing and migration behavior of birds and butterflies?
Do you think that it is impossible for the speed of light to have been different in the past?
Do you think that it is impossible to measure openmindedness?
Do you think that it is possible that there is no god?
Do you think that it is possible that evolution did not occur?
Do you think that is impossible for the power of 2 in Newtonian gravity, whereby the gravitational force is proportional to 1/r2, to be more precise with an exponent that is slightly different from 2, such as a gravitational force proportional to 1/r2.00000001?
This scale isn't a bad first effort, but there are some serious problems with it. First and foremost, many of the questions have the form "Is it imposssible that X is possible?" This is a confusing way of phrasing the question and the whole process might be better served if each item were a statement and respondents simply indicated whether that statement were possible or impossible.****** A second issue is that many of these questions deal with matters of empirical fact. The first item, for example, could be resolved with adequate data. As a result, a particular answer may simply indicate awareness of research. The same issue is true of the question about bird and butterfly migration, as I have touched on elsewhere. The migration question is, additionally, double barreled: It's impossible that it MUST or it's possible that it MUST? I don't know how to respond to a question that links possibility to certainty in such a fashion. Similarly, the last item simply indicates a willingness to accept the possibility of measurement error- hardly a sign of openmindedness. Finally, the questions encode Mr. Schlafly's perspective on things. So, for example, item 8 asks if it is possible that there is no god. For Mr. Schlafly, a theist, an answer of "yes" certainly indicates openmindedness. For myself, an atheist, the same answer indicates nothing of the sort.******* Many other items on the list are similarly flawed such that dogmatic acceptance of certain ideas might be confused with openmindedness.
To correct this problem, my wife and I have developed a possible alternative methodology. In our approach subjects to be tested will be exposed to a two-stage computer-mediated procedure. In the first stage the subjects will receive a battery of questions, some of which will determine their views on certain socially significant issues (e.g. abortion, stem cell research, etc). These opinions will then influence the second stage. In the second stage, the subjects will be asked to read several persuasive essays and report on how convincing they are. Four essays will then be presented: one essay supporting a postion the subject agrees with, one essay that they neither agree nor disagree with, and two essays supporting positions they disagree with. The computer will then time how long it takes for each respondent to complete reading each essay. A person who is more openminded should spend as long or longer reading the essays they disagree with as they do the essays they agree with or are ambivalent about. A person who is less openminded, in contrast, will spend less time reading (and considering the arguments of) the essays they disagree with. Note that we don't actually care how persuasive the essays are perceived as being- although that would be interesting data to analyze- but only how long they spend considering the essays. We are, as discussed earlier, interested in the process rather than the conclusion.
This approach has several advantages including that it does not encode a perspective into its questions, that it is more difficult to deliberately confound (since it measures something the subjects are not attending to), and it more directly assesses reactions to ideas that a subject disagrees with. On the other hand, it is more time consuming, more difficult to administer, and will require more preparation. Particularly, the essays would need to be normalized to the same level of reading difficulty and persuasiveness and pretesting would need to be carried out to determine mean reading times for those who both agree and disagree. This would, however, permit the usage of fairly standard statistical techniques to see if an individual is significantly more or less openminded than the population at large.
In all likelihood there are problems with this approach but I do think it solves many of the problems with Mr. Schlafly's approach and could warrant further development. And, if nothing else, it shows that every now and then Conservapedia does produce something that's worth taking seriously.
* What I mean is that the points made in some of the articles are logically unrelated to one another. It is as though I asked, "How much do those books weigh?" and you responded, "Ham!"
** Of course, there is the slight issue that we don't really know what IQ actually measures. Indeed, E.G. Boring once commented that IQ tests measure whatever it is that IQ tests measure. I digress, however...
*** Technically, he is interested in a willingness to consider evidence before rejecting an idea. However, if rejection is a foregone conclusion then I think we must agree that the subject is hardly openminded. I'm also forced to wonder why Schlafly is uninterested in the alternative meaning of openminded: "unprejudiced; unbigoted; impartial."
**** If we were more interested in correct determinations, we would probably be keenly interested in skepticism: "A doubting or questioning attitude or state of mind; dubiety."
***** Ignore, as well, that Mr. Schlafly has obviously never been taught how to construct surveys or psychometric instruments.
****** e.g. "There is an all-powerful God. __Possible __Impossible"
******* If the question were phrased as "Is it possible that there IS a god" I would still answer "yes." That is a side issue, however.
As a Side Note: For those who are curious, my answers to Schlafly's scale are (impossible=1 and possible=0): 0, 0, 0, 1, 0, 0, 0, 0, 0, 0. I think it impossible that the Shroud to Turin is "authentic" (whatever that means) because there is ample evidence indicating that it is of more recent origin than the first century AD. So, my score is 1/10 or .10. I'm not going to touch the "follow-up questions" because they are needlessly convoluted.