Randomized Control Trials - The Tarnished Gold Standard

Some blog followers might be interested in a recent post “Randomized Control Trials - The Tarnished Gold Standard” [Hake (2012a)]. The abstract reads:

************************************************
ABSTRACT: In response to “The Randomistas' War On Global Poverty - Erratum & Addendum” at http://bit.ly/YfMESg, Art Burke of the EvalTalk list pointed to an NYT piece “Applying Evidence to Social Programs” by Jon Baron at http://nyti.ms/Um9vVI.

Baron wrote (slightly edited): “Scientifically rigorous studies - particularly, the 'gold standard' of Randomized Controlled Trials (RCT’s) - are a mainstay of medicine, providing conclusive evidence of effectiveness for most major medical advances in recent history. In social spending, by contrast, such studies have only a toehold. Where they have been used, however, they have demonstrated the same ability to produce important, credible evidence about what works - and illuminated a path to major progress.”

In this post I cite arguments that the “gold standard” RCT studies may not be as lustrous as claimed by Baron:

(1) Ever since the pioneering work of Halloun & Hestenes (1985a) at http://bit.ly/fDdJHm, physicists have been engaged in social science of Physics Education Research (PER) that has made useful, reliable, and nonobvious predictions without resort to RCT’s - e.g. “Why Not Try a Scientific Approach to Science Education?” [Wieman (2007)] at http://bit.ly/anTMfF.

(2) In “A Response to ‘It's Not All About Class Size’ ” [Hake (2009)], I pointed out that according to the California Class Size Reduction Research Consortium [CCSRRC (2002)] at http://bit.ly/V923Ms, California's attempt to duplicate the vaunted Tennessee RCT study of reduced class size benefits results yielded no conclusive evidence of increased student achievement.

(3) In “A Summative Evaluation of RCT Methodology: & An Alternative Approach to Causal Research” [Scriven (2008] at http://bit.ly/93VcWD wrote: “In standard scientific usage, experiments are just carefully constrained explorations, and the RCT is simply a special case of these. To call the RCT the only ‘true experiment’ is part of an attempt at redefinition that distorts the original and continuing usage, and excludes experiments designed to test many simple hypotheses about - or simple efforts to find out - what happens if we do this.”

(4) In “Seventeen Statements by Gold-Standard Skeptics #2” [Hake (2010)] at http://bit.ly/TNpTR9 I cite, among others, the comments of the American Education Research Association; the American Evaluation Association; the National Education Association; the European Evaluation Society; Thomas Cook and Monique Payne, Hugh Burkhardt & Alan Schoenfeld; Margaret Eisenhart & Lisa Towne; Burke Johnson; Annette Lareau & Pamela Barnhouse; Joseph Maxwell; Dennis Phillips; Barbara Schneider, Martin Carnoy, Jeremy Kilpatrick, William Schmidt, & Richard Shavelson; Mack Shelley, Larry Yore, & Brian Hand; Deborah Stipek; and Carol Weiss.

(5) In "Why Most Published Research Findings Are False," John Ioannidis' (2005 at http://1.usa.gov/YxUxkL states ". . . . there is strong evidence that selective outcome reporting, with manipulation of the outcomes and analyses reported, is a common problem even for randomized trails [Chan et al. (2004)] at http://1.usa.gov/X8SB1T.

(6) the present signature quote of Thomas Cook and Monique Payne.
************************************************

To access the complete 18 kB post please click on http://bit.ly/VzVc0K.


Richard Hake, Emeritus Professor of Physics, Indiana University
Links to Articles: http://bit.ly/a6M5y0
Links to Socratic Dialogue Inducing (SDI) Labs: http://bit.ly/9nGd3M
Academia: http://bit.ly/a8ixxm
Blog: http://bit.ly/9yGsXh
GooglePlus: http://bit.ly/KwZ6mE
Twitter: http://bit.ly/juvd52

"In some quarters, particularly medical ones, the randomized experiment is considered the causal 'gold standard.' It is clearly not that in educational contexts, given the difficulties with implementing and maintaining randomly created groups, with the sometimes incomplete implementation of treatment particulars, with the borrowing of some treatment particulars by control group units, and with the limitations to external validity that often follow from how the random assignment is achieved."
- Thomas Cook and Monique Payne in "Evidence Matters" [Mosteller & Boruch (2002)]


REFERENCES [URL shortened by http://bit.ly/ and accessed on 04 Dec 2012.]
Hake, R.R. 2012a. "Randomized Control Trials - The Tarnished Gold Standard" online on the OPEN! AERA-L archives at http://bit.ly/VzVc0K. Post of 4 Dec 2012 19:26:48-0800 to AERA-L and Net-Gold. The abstract and link to the complete post are being transmitted to several discussion lists.

Hake, R.R. 2012b. “The Randomistas' War On Global Poverty - ERRATUM & ADDENDUM,” online on the OPEN! AERA-L archives at http://bit.ly/YfMESg. Post of 30 Nov 2012 12:15:33-0800 to AERA-L and Net-Gold. The abstract and link to the complete post are being transmitted to several discussion lists, and are also on my blog “Hake'sEdStuff” at http://bit.ly/11c5w3e with a provision for comments.

Mosteller, F. & R. Boruch, eds. 2002. Evidence Matters: Randomized Trials in Education Research. Brookings Institution, publisher's information at http://bit.ly/UoX3sA. Amazon.com information at http://amzn.to/n6T0Uo. An expurgated Google book preview is online at http://bit.ly/RX1k3u.

Re: Why Most Published Research Findings Are False

Some blog followers might be interested in a recent post “Re: Why Most Published Research Findings Are False” [Hake (2012a)]. The abstract reads:

**********************************************
ABSTRACT: In “The Randomistas’ War On Global Poverty - ERRATUM & ADDENDUM” [Hake (2012b] at http://bit.ly/YfMESg I pointed to:

(a) a PLoS Medicine article by John Ioannidis (2005) http://bit.ly/Vb1u70 titled “Why Most Published Research Findings Are False” at http://1.usa.gov/YxUxkL, brought to my attention by Guy Brandenberg; and

(b) a good discussion of the important work of Ioannidis in the Atlantic: “Lies, Damned Lies, and Medical Science” by David Freedman (2010) at http://bit.ly/11aAmt0. Freedman wrote that Ioannidis (2005):

(1) claims that as much as 90 percent of the published medical information that doctors rely on is flawed;

(2) states that randomized controlled trials. . . . ended up being wrong some of the time;

(3) laid out a detailed mathematical proof that, assuming modest levels of researcher bias, typically imperfect research techniques, and the well-known tendency to focus on exciting rather than highly plausible theories, researchers will come up with wrong findings most of the time;

(4) is the most downloaded article in the history of PLoS Medicine.

BTW: Freedman claims that meta-research experts have confirmed that similar issues distort research in all fields of science, from physics to economics (where DeLong & Lang at http://bit.ly/SpNMww showed how a remarkably consistent paucity of strong evidence in published economics studies made it unlikely that any of them were right).
**********************************************

To access the complete 9 kB post please click on http://bit.ly/Ve4Qnk.

Richard Hake, Emeritus Professor of Physics, Indiana University
Links to Articles: http://bit.ly/a6M5y0
Links to Socratic Dialogue Inducing (SDI) Labs: http://bit.ly/9nGd3M
Academia: http://bit.ly/a8ixxm
Blog: http://bit.ly/9yGsXh
GooglePlus: http://bit.ly/KwZ6mE
Twitter: http://bit.ly/juvd52

“It is not enough to observe, experiment, theorize, calculate and communicate; we must also argue, criticize, debate, expound, summarize, and otherwise transform the information that we have obtained individually into reliable, well established, public knowledge.”
- John Ziman. 1969. "Information, Communication, Knowledge," Nature 224: 318-324; abstract online at http://bit.ly/cNPB1d.

REFERENCES [URL’s shortened by http://bit.ly/ and accessed on 02 Dec 2012.]
Hake, R.R. 2012a. “Re: Why Most Published Research Findings Are False,” online on the OPEN! AERA-L archives at http://bit.ly/Ve4Qnk. Post of 02 Dec 2012 11:39:22-0800 to AERA-L and Net-Gold. The abstract and link to the complete post are being transmitted to several discussion lists.

Hake, R.R. 2012b. “The Randomistas’ War On Global Poverty - ERRATUM & ADDENDUM,” online on the OPEN! AERA-L archives at http://bit.ly/YfMESg. Post of 30 Nov 2012 12:15:33-0800 to AERA-L and Net-Gold. The abstract and link to the complete post are being transmitted to several discussion lists and are also on my blog “Hake'sEdStuff” at http://bit.ly/11c5w3e with a provision for comments.

The Randomistas' War On Global Poverty - ERRATUM & ADDENDUM

Some blog followers might be interested in a recent post “The Randomistas’ War On Global Poverty - ERRATUM & ADDENDUM” [Hake (2012a)]. The abstract reads:

*******************************************************
ABSTRACT: My post “The Randomistas' War On Global Poverty” [Hake (2012b)] at http://bit.ly/V93tXl:

I. Contains an ERRATUM: I wrote: “. . . . . there's no indication that the ‘Center for Economic and Policy Research’ (CEPR) http://bit.ly/U1DTSU, FOR WHICH DUFLO IS A PROGRAM DIRECTOR FOR DEVELOPMENTAL ECONOMICS http://bit.ly/VgQ3y8, is aware of the overriding influence of poverty on the educational achievement of U.S. children. . . .” Contrary to that statement, Duflo is a program director for the EUROPEAN “Centre for Economic Policy Research” http://www.cepr.org/ & http://bit.ly/TwYuzM, NOT the U.S. “Center for Economic and Policy Research” (CEPR) http://www.cepr.net/ & http://bit.ly/U1DTSU. So Duflo bares ZERO responsibility for the U.S. CEPR’s seeming unawareness of “the overriding influence of poverty on the educational achievement of U.S. children.”

II. Now requires an ADDENDUM: In response to Hake (2012), Guy Brandenberg (2012) of the EDDRA2 list pointed to a PLoS Medicine article by John Ioannidis http://bit.ly/Vb1u70 titled “Why Most Published Research Findings Are False” at http://1.usa.gov/YxUxkL, wherein Ioannidis wrote: “. . . . there is strong evidence that selective outcome reporting, with manipulation of the outcomes and analyses reported, is a common problem even for randomized trails [Chan et al. (2004)] at http://1.usa.gov/X8SB1T.” For a good discussion of the important work of Ioannidis see the Atlantic article “Lies, Damned Lies, and Medical Science” by David Freedman (2010) at http://bit.ly/11aAmt0.
*******************************************************

To access the complete 9 kB post please click on http://bit.ly/YfMESg.


Richard Hake, Emeritus Professor of Physics, Indiana University
Links to Articles: http://bit.ly/a6M5y0
Links to Socratic Dialogue Inducing (SDI) Labs: http://bit.ly/9nGd3M
Academia: http://bit.ly/a8ixxm
Blog: http://bit.ly/9yGsXh
GooglePlus: http://bit.ly/KwZ6mE
Twitter: http://bit.ly/juvd52

REFERENCES [URL shortened by http://bit.ly/ and accessed on 30 Nov 2012.]
Hake, R.R. 2012a. “The Randomistas’ War On Global Poverty - ERRATUM & ADDENDUM,” online on the OPEN! AERA-L archives at http://bit.ly/YfMESg. Post of 30 Nov 2012 12:15:33-0800 to AERA-L and Net-Gold. The abstract and link to the complete post are being transmitted to several discussion lists.

Hake, R.R. 2012b. “The Randomistas’ War On Global Poverty (was Chocolate Makes You Smart),” online on the OPEN! AERA-L archives at http://bit.ly/V93tXl. Post of 29 Nov 2012 14:27:16-0800 to AERA-L and Net-Gold. The abstract and link to the complete post are being transmitted to several discussion lists and are also on my blog “Hake'sEdStuff” at http://bit.ly/Rm6Oqw with a provision for comments.