Total Drek

Or, the thoughts of several frustrated intellectuals on Sociology, Gaming, Science, Politics, Science Fiction, Religion, and whatever the hell else strikes their fancy. There is absolutely no reason why you should read this blog. None. Seriously. Go hit your back button. It's up in the upper left-hand corner of your browser... it says "Back." Don't say we didn't warn you.

Monday, December 24, 2007

This feels weird to me too.

Regular readers of this blog know that, from time to time, I comment on things going on over at Conservapedia. Regular readers also realize that, more often than not, these comments are somewhat negative as many Conservapedia articles are so rife with logical inconsistency that I have a difficult time parsing them.* So, it may come as a surprise to many of you that I recently came across something that I find genuinely... interesting.

I refer, of course, to an essay by Andrew Schlafly titled Quantifying Openmindedness. In it he proposes, in essence, that since we have scales that supposedly measure IQ** or Body Mass, we should be able to develop a way of measuring openmindedness. Essentially, he is arguing for the development of a new psychometric technique. I see no necessary reason why this should be impossible and, indeed, think it is an interesting idea. So how might we go about it?

Well, first, we need to define the thing that we're trying to measure. What is "openmindedness"? Schlafly defines it as follows:

By "openmindness" I mean a genuine willingness to consider the evidence before rejecting an idea. I do not mean tolerance, or a rejection of absolute truth, or skepticism. Openmindness means here what the dictionary says: "receptive to arguments or ideas."

So, the aspect that Schlafly is most interested in is the simple willingness to consider evidence before judging an idea.*** So, for all intents and purposes, we are interested in assessing whether or not the process of reacting to information has a certain character but are not interested in the outcome of that process. Put another way, we don't care whether or not the subject makes a correct determination about the evidence but only whether or not they consider it.**** I might modify this slightly to make it more amenable to testing by simply stressing that openmindedness entails a willingness to seriously consider the views of those with whom you disagree. Having reached, more or less, a definition of the concept, we can now turn to operationalization, or converting the concept into measurable indicators.

Again, Schlafly beats us to the punch suggesting a scale for openmindedness that is based on the idea that openmindedness is the converse of closedmindedness. In other words, we can measure unwillingness to consider ideas and assume that the less unwillingness is present, the more openminded a person is. This is a fairly strong assumption but we'll take it as a given. Using this perspective, in the scale below, each time you report that an item is impossible you get 1 point. The higher your score, the lower your level of openmindedness. Note, as well, that some items are reverse-coded. Ignore, as well, that some questions should be answered "yes" or "no."*****

Do you think it is impossible that more widespread gun ownership reduces the rate of crime?

When President Ronald Reagan told Mr. Gorbachev to tear down the Berlin Wall, did you think that it was impossible for the Berlin Wall to be torn down?

Did you think, or still think, that the Strategic Defense Initiative is impossible?

Do you think that it is impossible that the Shroud of Turin is authentic?

Do you think that there must be a material explanation for remarkable homing and migration behavior of birds and butterflies?

Do you think that it is impossible for the speed of light to have been different in the past?

Do you think that it is impossible to measure openmindedness?

Do you think that it is possible that there is no god?

Do you think that it is possible that evolution did not occur?

Do you think that is impossible for the power of 2 in Newtonian gravity, whereby the gravitational force is proportional to 1/r2, to be more precise with an exponent that is slightly different from 2, such as a gravitational force proportional to 1/r2.00000001?

This scale isn't a bad first effort, but there are some serious problems with it. First and foremost, many of the questions have the form "Is it imposssible that X is possible?" This is a confusing way of phrasing the question and the whole process might be better served if each item were a statement and respondents simply indicated whether that statement were possible or impossible.****** A second issue is that many of these questions deal with matters of empirical fact. The first item, for example, could be resolved with adequate data. As a result, a particular answer may simply indicate awareness of research. The same issue is true of the question about bird and butterfly migration, as I have touched on elsewhere. The migration question is, additionally, double barreled: It's impossible that it MUST or it's possible that it MUST? I don't know how to respond to a question that links possibility to certainty in such a fashion. Similarly, the last item simply indicates a willingness to accept the possibility of measurement error- hardly a sign of openmindedness. Finally, the questions encode Mr. Schlafly's perspective on things. So, for example, item 8 asks if it is possible that there is no god. For Mr. Schlafly, a theist, an answer of "yes" certainly indicates openmindedness. For myself, an atheist, the same answer indicates nothing of the sort.******* Many other items on the list are similarly flawed such that dogmatic acceptance of certain ideas might be confused with openmindedness.

To correct this problem, my wife and I have developed a possible alternative methodology. In our approach subjects to be tested will be exposed to a two-stage computer-mediated procedure. In the first stage the subjects will receive a battery of questions, some of which will determine their views on certain socially significant issues (e.g. abortion, stem cell research, etc). These opinions will then influence the second stage. In the second stage, the subjects will be asked to read several persuasive essays and report on how convincing they are. Four essays will then be presented: one essay supporting a postion the subject agrees with, one essay that they neither agree nor disagree with, and two essays supporting positions they disagree with. The computer will then time how long it takes for each respondent to complete reading each essay. A person who is more openminded should spend as long or longer reading the essays they disagree with as they do the essays they agree with or are ambivalent about. A person who is less openminded, in contrast, will spend less time reading (and considering the arguments of) the essays they disagree with. Note that we don't actually care how persuasive the essays are perceived as being- although that would be interesting data to analyze- but only how long they spend considering the essays. We are, as discussed earlier, interested in the process rather than the conclusion.

This approach has several advantages including that it does not encode a perspective into its questions, that it is more difficult to deliberately confound (since it measures something the subjects are not attending to), and it more directly assesses reactions to ideas that a subject disagrees with. On the other hand, it is more time consuming, more difficult to administer, and will require more preparation. Particularly, the essays would need to be normalized to the same level of reading difficulty and persuasiveness and pretesting would need to be carried out to determine mean reading times for those who both agree and disagree. This would, however, permit the usage of fairly standard statistical techniques to see if an individual is significantly more or less openminded than the population at large.

In all likelihood there are problems with this approach but I do think it solves many of the problems with Mr. Schlafly's approach and could warrant further development. And, if nothing else, it shows that every now and then Conservapedia does produce something that's worth taking seriously.

* What I mean is that the points made in some of the articles are logically unrelated to one another. It is as though I asked, "How much do those books weigh?" and you responded, "Ham!"

** Of course, there is the slight issue that we don't really know what IQ actually measures. Indeed, E.G. Boring once commented that IQ tests measure whatever it is that IQ tests measure. I digress, however...

*** Technically, he is interested in a willingness to consider evidence before rejecting an idea. However, if rejection is a foregone conclusion then I think we must agree that the subject is hardly openminded. I'm also forced to wonder why Schlafly is uninterested in the alternative meaning of openminded: "unprejudiced; unbigoted; impartial."

**** If we were more interested in correct determinations, we would probably be keenly interested in skepticism: "A doubting or questioning attitude or state of mind; dubiety."

***** Ignore, as well, that Mr. Schlafly has obviously never been taught how to construct surveys or psychometric instruments.

****** e.g. "There is an all-powerful God. __Possible __Impossible"

******* If the question were phrased as "Is it possible that there IS a god" I would still answer "yes." That is a side issue, however.

As a Side Note: For those who are curious, my answers to Schlafly's scale are (impossible=1 and possible=0): 0, 0, 0, 1, 0, 0, 0, 0, 0, 0. I think it impossible that the Shroud to Turin is "authentic" (whatever that means) because there is ample evidence indicating that it is of more recent origin than the first century AD. So, my score is 1/10 or .10. I'm not going to touch the "follow-up questions" because they are needlessly convoluted.

Labels: , , ,


Anonymous Andy Schlafly said...

Thanks for considering and analyzing my suggestion to quantify openmindedness. Your score was quite good, and if it withstands scrutiny that you should be applauded for being relatively openminded. However, your comments on the Shroud of Turin and migration/homing show signs of a lack of openmindedness, and I wonder if your better answers to the more political questions merely reflect a lack of interest in those issues. In other words, you seemed less openminded on issues you seem to care more about, the inverse of what one would expect from an openminded, curious mind. Regardless, thanks again and Merry Christmas. Andy Schlafly

Tuesday, December 25, 2007 6:50:00 PM  
Blogger Drek said...


I'm pleased that you enjoyed my analysis, but the remainder of your response is somewhat problematic. First off, I'm not sure what to make of your "withstand scrutiny" remark. Presumably you are suggesting that I am not as openminded as the test indicates. Perhaps so but, if this is the case, is it not a problem with the test rather than me?

Secondly, I think you will find that I care rather a great deal about whether or not god exists and whether or not evolution did, indeed, take place as biological science indicates. Yet, I apparently managed to be judged as "openminded" on both of those questions. My interest in an issue does not appear to have a strong relationship to my degree of openmindedness as judged by your test.

Thirdly, I think I would caution you against referring to some answers as "better." Remember, you are interested in a process, not an outcome. A person who is openminded but concludes, based on evidence, logic, or both, that a given conclusion is correct is not any less openminded. If we require that a person not reach a conclusion then we are measuring indecisiveness, not openmindedness.

Finally, I think you would be wise to focus on developing and validating an appropriate testing methodology before you try to judge people based on their degree of "openmindedness." Psychometric testing is not a trivial matter and should not be embarked upon casually.

Wednesday, December 26, 2007 11:45:00 AM  
Blogger Mister Troll said...

Hmm, interesting. I scored even higher than Drek on the open-mindedness scale.

Now what does that get me*?

* a liberal, atheist scientist

Sunday, December 30, 2007 10:39:00 AM  

Post a Comment

<< Home

Site Meter