« Obama's second scandal: Secret Service | Main | NPR interview on "The Death of Facts" »

April 27, 2012


Your proposal for pre-accepted academic articles is certainly a step in the right direction. However, as long as there remain substantial incentives for positive results--better job offers, tenure, well-funded chairs, praise and glory among one's professional colleagues--is it wrong to suspect that, acting out of self-interest, many will try to game the system?

How could one do that? Well, prior to writing the all-but-results article, one could conduct one or more pilot studies to test the hypothesis and/or research design. Try the idea out on twenty students, for example, not with the goal of getting statistically significant results but to see whether one's guess about the hypothesis or research design is achieving generally positive results. (Note that pilot studies or focus groups may well play a legitimate function in social science research to improve questionnaire design and reduce ambiguity in phrasing of questions or for other reasons; that's not what I'm concerned with here.)

Probably there are other ways of increasing the likelihood that the all-but-results article one submits will ultimately enjoy statistically significant positive results; people with more familiarity than I with quantitative research can use their imaginations.

I hope I'm wrong about the propensity of many scholars to act in their own self-interest, but frankly I haven't seen much to suggest that academics are immune from human nature. If that's right, the ultimate solution is a change in the differential professional rewards received by those who publish positive results. Happily, those rewards are almost entirely within the control of academics themselves. All that's needed is a change in attitudes.

Here are a few thoughts, mostly from my wife Joan, who as biostatistician co-authored lots of medical research papers:

1. When a funding agency, such as NIH, funds some research, they have made their evaluation based on study design, prior to results. So, it can be done.

2. A limiting factor is the time and energy of Professors who volunteer to serve on journals' Committees on Review of Papers and who may also serve as volunteers to help funding agencies evaluate proposals.

3. If the study involves the hiring of employees, once it's funded it generally needs to start right away. The research can't wait for journal pre-approval, or the employees may not be available when needed.

4. I suggested that Brendan's suggestion might be implemented by having agencies like NIH simply publish on the web all the research they have funded. Joan was dubious, because the NIH isn't a publishing organization and because of the possible shortage of review committees.

5. Joan said that once her groups' research had been done, they generally or always found some journal that would publish it, even if the results didn't break new ground.

6. There would need to be a process for reasonable modifications. E.g., if they're unable to get as many subjects as planned, there may be a way to modify the studey design so it's still useful.

7. Sometimes a secondary aspect of a study turns out to be more valuable than the primary point of interest. There needs to be a way for this situation to get appropriate publication.

The comments to this entry are closed.