Randomized Control Trials: The Strange Case of the Contradictory Graphs (was In Defense of the NRC's Scientific Research in Education)

Some blog followers might be interested in a recent post “Randomized Control Trials: The Strange Case of the Contradictory Graphs (was In Defense of the NRC's Scientific Research in Education)” [Hake (2012)]. The abstract reads:

**************************************************
ABSTRACT: Susan Skidmore at http://bit.ly/Uov4sU alerted the Math-Teach list to her valuable articles with Bruce Thompson in the June/July 2012 issue of the Educational Researcher: (a) “Propagation of Misinformation About Frequencies of RFTs/RCTs in Education: A Cautionary Tale” [Skidmore & Thomson (2012a)] at http://bit.ly/SPN361, and (b) “Things (We Now Believe) We Know” [S&T (2012b)] at http://bit.ly/ZSQ5v5.

S&T (2012a) discuss the CONTRADICTORY GRAPHS of cumulative numbers of Randomized Control Trials (RCT’s) vs time for Criminology, Education, Psychology, and Social fields (showing education first, tied for second, and last) presented by influential scholars in prominent settings [that, along with the attendant sequence of events] “may have gratuitously damaged the already fragile reputation of education research as a field.”

After reviewing the history, S&T (2012b) conclude: “We believe that the errors were unintentional . . . . . . “ But the history as recounted by S&T (2012a) [and in the same Educational Researcher issue by Robinson (2012) at http://bit.ly/WHhdiU and Petrosino (2012) at http://bit.ly/SU3K3O] seems to contradict S&T’s conclusion.

Thomas Cook submitted an article with the title “A critical appraisal of the case against using experiments to assess school (or community) effects” [Cook (2001a)] at http://bit.ly/Uyd3CY with NO GRAPH to the Hoover Institution's Education Next http://educationnext.org/ Evidently without Cook's knowledge, his academic article was heavily edited and published as “Sciencephobia: Why education researchers reject randomized experiments” [Cook (2001b)] at http://bit.ly/SQox50 WITH A GRAPH of cumulative numbers of Randomized Control Trials (RCT’s) vs time for Criminology, Education, Psychology, and Social fields showing education LAST, consistent with the provocative new title. The graph was erroneously attributed to Boruch, De Moya, & Snyder (2001) - the data should have been 2002) - at http://bit.ly/UoX3sA, despite the fact that the Boruch et al. graph showed education tied for second, not last. Are we to believe that Education Next's degradation of the accurate academic Cook (2001a) to the inaccurate hooverized Cook (2001b) was unintentional?

A side issue: to those who regard RCT’s as the “gold standard” of education research, the higher the curve of cumulative numbers of Randomized Control Trials (RCT’s) vs time for a field, the higher the merit of research in that field. But not everyone would agree - see e.g., “A Summative Evaluation of RCT Methodology: & An Alternative Approach to Causal Research” [Scriven (2008] at http://bit.ly/93VcWD, “Seventeen Statements by Gold-Standard Skeptics #2” [Hake (2010)] at http://bit.ly/TNpTR9, and the present signature quote of Thomas Cook and Monique Payne.
**************************************************

To access the complete 46 kB post please click on http://bit.ly/TQdfhX.


Richard Hake, Emeritus Professor of Physics, Indiana University
Links to Articles: http://bit.ly/a6M5y0
Links to Socratic Dialogue Inducing (SDI) Labs: http://bit.ly/9nGd3M
Academia: http://bit.ly/a8ixxm
Blog: http://bit.ly/9yGsXh
GooglePlus: http://bit.ly/KwZ6mE
Twitter: http://bit.ly/juvd52

“In science education, there is almost nothing of proven efficacy.”
- Grover Whitehurst, former director, Institute of Education Sciences, USDE, as quoted by Sharon Begley (2004)

“In some quarters, particularly medical ones, the randomized experiment is considered the causal ‘gold standard.’ It is clearly not that in educational contexts, given the difficulties with implementing and maintaining randomly created groups, with the sometimes incomplete implementation of treatment particulars, with the borrowing of some treatment particulars by control group units, and with the limitations to external validity that often follow from how the random assignment is achieved.”
- Thomas Cook and Monique Payne in Evidence Matters [Mosteller & Boruch (2002)]

“Physics educators have led the way in developing and using objective tests to compare student learning gains in different types of courses, and chemists, biologists, and others are now developing similar instruments. These tests provide convincing evidence that students assimilate new knowledge more effectively in courses including active, inquiry-based, and collaborative learning, assisted by information technology, than in traditional courses.”
- Wood & Gentile (2003)


REFERENCES [URL’s shortened by http://bit.ly/ and accessed on 17 Nov 2012.]
Begley, S. 2004. “To Improve Education, We Need Clinical Trials To Show What Works,” Wall Street Journal, 17 December, page B1; online as a 41 kB pdf at http://bit.ly/SSmaym, thanks to David Klahr.

Hake, R.R. 2012. “Randomized Control Trials: The Strange Case of the Contradictory Graphs (was In Defense of the NRC's Scientific Research in Education)” online on the OPEN! AERA-L archives at http://bit.ly/TQdfhX. Post of 17 Nov 2012 10:45:11-0800 to AERA-L and Net-Gold. The abstract and link to the complete post are being transmitted to several discussion lists.

Mosteller, F. & R. Boruch, eds. 2002. Evidence Matters: Randomized Trials in Education Research. Brookings Institution, publisher's information at http://bit.ly/UoX3sA. Amazon.com information at http://amzn.to/n6T0Uo. An expurgated Google book preview is online at http://bit.ly/RX1k3u.

Wood, W.B., & J.M. Gentile. 2003. “Teaching in a research context,” Science 302: 1510; 28 November; online as a 213 kB pdf at http://bit.ly/SyhOvL thanks to Portland State's “Ecoplexity” site.