Total Drek

Or, the thoughts of several frustrated intellectuals on Sociology, Gaming, Science, Politics, Science Fiction, Religion, and whatever the hell else strikes their fancy. There is absolutely no reason why you should read this blog. None. Seriously. Go hit your back button. It's up in the upper left-hand corner of your browser... it says "Back." Don't say we didn't warn you.

Tuesday, July 28, 2009

Well, this is a pain in the arse.

Folks with an interest in the sociology of science and knowledge may have noticed this little gem that appeared in BMJ. It's an article by Steven Greenberg titled, "How citation distortions create unfounded authority: analysis of a citation network." The abstract, in somewhat abridged form, is nothing if not intriguing:

ABSTRACT
Objective To understand belief in a specific scientific claim by studying the pattern of citations among papers stating it.

Design A complete citation network was constructed from all PubMed indexed English literature papers addressing the belief that β amyloid, a protein accumulated in the brain in Alzheimer’s disease, is produced by and injures skeletal muscle of patients with inclusion body myositis. Social network theory and graph theory were used to analyse this network.

...

Conclusion Citation is both an impartial scholarly method and a powerful form of social communication. Through distortions in its social use that include bias, amplification, and invention, citation can be used to generate information cascades resulting in unfounded authority of claims. Construction and analysis of a claim specific citation network may clarify the nature of a published belief system and expose distorted methods of social citation.


In short, Greenberg identifies several mechanisms that can impair the ability of the scientific process to reach an accurate result. "Citation Bias" is the preferential citation of supporting evidence (whatever supporting is in your case) and the discounting or ignoring of critical evidence. This is particularly the case with critical primary (i.e. using data) research. "Citation Diversion" is the citing of papers that say something relevant to your claim, but which do not say precisely what you imply or state they say. Note that this is not lying about what the citation supports but more akin to stretching it through interpretation. "Invention" relates to the use of citations to introduce new material. Often this is an over-statement of what is contained in the cited article. So, for example, the article says, "We hypothesize that x is related to y" while the citation of that article reads, "So and so demonstrated that x is related to y". The newer, and much stronger, claim has essentially been invented. Finally, "amplification" emerges when articles that lack primary data (e.g. review articles) are cited as authoritative sources for demonstrating that some particular thing is true. In that these articles are effectively just recycling and collating earlier work, they provide a sort of echo chamber that lends additional legitimacy to a claim without lending it additional substantive support.

In combination, the mechanisms above can generate chains of citation between papers that seem to produce tremendous legitimacy for a claim when, in fact, that claim may be unreliable. This is obviously a problem. Greenberg identifies a few reasons why this may be prone to occur, including the structure of rewards in science (e.g. successful grant proposals often must be built on existing work- in other words, existing citation chains), the natural truncation that follows negative results (e.g. you don't publish multiple papers on a negative finding, but can build a career on a positive finding), and on more mundane causes like the confirmation bias (e.g. you like your hypothesis/theory so you tend to discount the articles that imply it may be incorrect). In total, however, the paper casts a certain amount of doubt on the cumulative reliability of journal results.

So is this the death knell for science as we know it? Eh. Probably not, if for no other reason than that we don't have anything better waiting in the wings. Solutions, however, will be difficult. Mostly I think it's beholden on authors to be more careful with their citations so as to avoid this sort of thing. Doubtless it will continue, and maybe one of these days we'll figure a way to check every citation of every paper that is submitted, but I don't think reviewers have that kind of time right now and are unlikely to discover it in the near future. We could probably resolve at least some of these issues if the perennial joke publication, "The Journal of Null Findings," were to get off the ground. Then we might have an easier way to counter-balance some of the exciting new error-prone revelations that can get us all haring off in some random direction. Finally, however, I wonder if it wouldn't help if we all tried to de-emphasize the idea that each and every important paper has to produce some amazing new theoretical insight. It's hard for science to accumulate when we get rewarded for new conceptual leaps but not for the dirty, difficult work of sorting the wheat from the chaff. I, for one, would certainly be in favor of that if only because it would make life easier when I'm searching in vain for good empirical overviews of certain areas.

And as a final side note, if you haven't already felt amused at the fact that I cited an article that reports on the unreliability of citations, there's no time like the present to start.

Labels: , , ,

0 Comments:

Post a Comment

<< Home

Site Meter