Some blog followers might be interested in a recent post “Grover Whitehurst Testifies Against Class Size Reduction” [Hake (2012)]. The abstract reads:
*********************************************
ABSTRACT: Diane Ravitch in her blog entry “When Grover Whitehurst Testified Against Class Size Reduction” at http://bit.ly/QRniWu pointed to Leonie Haimson’s http://huff.to/12fHNyy “Grover Whitehurst's big pay day, testifying class size doesn't matter” at http://bit.ly/Vsp2T2 and asked: “DOES CLASS SIZE MATTER? READ HAIMSON'S ACCOUNT AND REACH YOUR OWN JUDGMENT.”
Haimson pointed out that:
a. According to report by Will Weissert (2012a) at http://yhoo.it/VxrBCQ, economist Diane Whitmore Schanzenbach http://bit.ly/TVN6hv testified that students in smaller classes “tend to do better on standardized tests and even eventually become better citizens, more likely to own their own homes and save for retirement” and that “study after study shows that smaller classes often mean greater success for students.” Schanzenbach also coathored: (1) “Experimental Evidence on the Effect of Childhood Investments on Postsecondary Attainment and Degree Completion” [Dynarski et al. (2011)] at http://bit.ly/YRF0h7 showing that smaller classes increased the rate of college attendance, especially among poor students, and improved the probability of earning a college degree, especially in high-earning fields such as science, technology, engineering, and mathematics; and (2) “How Does Your Kindergarten Classroom Affect Your Earnings? Evidence From Project Star” [Chetty et al. (2011)] at http://bit.ly/U7oJNn showing that these students were also more likely to own their own home and a 401K more than twenty years later.
b. Whitehurst & Chingos (2011) wrote “Class Size: What Research Says and What it Means for State Policy” [Chingos & Whitehurst (2011)] at http://bit.ly/VXroeA which argued that LOWERING CLASS SIZE WAS A WASTE OF MONEY despite admitting in the report that “very large class-size reductions, on the order of magnitude of 7-10 fewer students per class, can have significant long-term effects on student achievement and other meaningful outcomes.”
c. When Whitehurst was at the US Department of Education from 2002-2008, he headed the Institute of Education Sciences http://ies.ed.gov/, which in a report Identifying and Implementing Educational Practices Supported by Rigorous Evidence: A User Friendly Guide [USDE (2003)] at http://bit.ly/ds1sRS cited CLASS SIZE REDUCTION AS ONLY ONE OF FOUR EXAMPLES OF EDUCATION REFORMS "FOUND TO BE EFFECTIVE in randomized controlled trials – research’s 'gold standard."[Yet] as the lead off witness for the state on Friday Whitehurst argued that contrary to the claims of the plaintiffs, “Texas is doing pretty good” and that these huge budget cuts were immaterial because CLASS SIZE DOESN'T MATTER.
d. Terrence Stutz (2012) at http://bit.ly/VXwA2l reported in the Dallas Morning News: “State attorneys also have been arguing that larger class sizes in Texas - the result of a $5.4 billion funding cut by the Legislature last year - have not hurt students because CLASS SIZES DON'T AFFECT ACHIEVEMENT. Whitehurst testified in support of that position. But again, under cross examination by Dallas lawyer John Turner, Whitehurst had to acknowledge that he wrote an article praising a well-publicized study of lower class sizes in Tennessee that found significant improvement in student achievement. Whitehurst explained that he had changed his mind since writing the article and now has DOUBTS THAT CLASS SIZE HAS MUCH IMPACT ON LEARNING. In later testimony, he said he was being paid $340 an hour by the state to testify in the case, and had already racked up 220 billable hours - for just under $75,000 - before he took the witness stand."
e. Whitehurst racked up 220 billable hours? That means Whitehurst must have worked nearly thirty 8-hour days on it. Wonder what took him so long?
*********************************************
To access the complete 23 kB post please click on http://bit.ly/Ufro6y.
Richard Hake, Emeritus Professor of Physics, Indiana University
Links to Articles: http://bit.ly/a6M5y0
Links to Socratic Dialogue Inducing (SDI) Labs: http://bit.ly/9nGd3M
Academia: http://bit.ly/a8ixxm
Blog: http://bit.ly/9yGsXh
GooglePlus: http://bit.ly/KwZ6mE
Twitter: http://bit.ly/juvd52
“Physics educators have led the way in developing and using objective tests to compare student learning gains in different types of courses, and chemists, biologists, and others are now developing similar instruments. These tests provide convincing evidence that students assimilate new knowledge more effectively in courses including active, inquiry-based, and collaborative learning, assisted by information technology, than in traditional courses.”
- Wood & Gentile (2003)
“In science education, there is almost nothing of proven efficacy.”
- Grover Whitehurst, as quoted by Sharon Begley (2004)
“Well-designed and implemented randomized controlled trials are considered the ‘gold standard’ for evaluaing an intervention's effectiveness, in fields such as medicine, welfare and employment policy, and psychology.”
- USDE (2003)
“Scientifically rigorous studies - particularly, the ‘gold standard’ of Randomized Controlled Trials (RCT’s) - are a mainstay of medicine, providing conclusive evidence of effectiveness for most major medical advances in recent history. In social spending, by contrast, such studies have only a toehold. Where they have been used, however, they have demonstrated the same ability to produce important, credible evidence about what works - and illuminated a path to major progress.”
- Jon Barron (2012)
“In some quarters, particularly medical ones, the randomized experiment is considered the causal 'gold standard.' It is clearly not that in educational contexts, given the difficulties with implementing and maintaining randomly created groups, with the sometimes incomplete implementation of treatment particulars, with the borrowing of some treatment particulars by control group units, and with the limitations to external validity that often follow from how the random assignment is achieved.”
- Thomas Cook and Monique Payne in Evidence Matters [Mosteller &
Boruch (2002)]
“According to the California Class Size Reduction Research Consortium [CCSRRC (2002)], California's attempt to duplicate the vaunted Tennessee RCT study of reduced class size benefits results yielded no conclusive evidence of increased student achievement. One reason appears to be that there were simply not enough teachers in California to support any substantive class size reduction without deterioration of teaching effectiveness.”
- R.R. Hake (2009)
REFERENCES [URL’s shortened by http://bit.ly/ and accessed on 10 Dec 2012.]
Baron, J. 2012. “Applying Evidence to Social Programs.” New York Times, 29 Nov; online at http://nyti.ms/Um9vVI.
Begley, S. 2004. “To Improve Education, We Need Clinical Trials To Show What Works,” Wall Street Journal, 17 December, page B1; online as a 41 kB pdf at http://bit.ly/SSmaym, thanks to David Klahr.
CCSRRC. 2002. “What We Have Learned About Class Size Reduction in California,” California Class Size Reduction Research Consortium [American Institutes for Research (AIR), RAND, Policy Analysis for California Education (PACE), WestEd, and EdSource]; full report online as a 9.5 MB pdf at http://bit.ly/YRD5ZS. A press release is online at http://bit.ly/V923Ms.
Hake, R.R. 2009. “A Response to ‘It's Not All About Class Size’,” AERA-L post of 6 Feb 2009 09:42:04-0800; online on the OPEN! AERA-L archives at http://bit.ly/KBzuXV. Post of 6 Feb 2009 09:42:04-0800 to AERA-L and Net-Gold. The abstract and link to the complete post were also distributed to various discussion lists.
Hake, R.R. 2012. “Grover Whitehurst Testifies Against Class Size Reduction” online on the OPEN! online on the OPEN! AERA-L archives at http://bit.ly/VYtD1l. Post of 9 Dec 2012 18:34:56-0800 to AERA-L and Net-Gold. The abstract and link to the complete post are being transmitted to several discussion lists.
Mosteller, F. & R. Boruch, eds. 2002. Evidence Matters: Randomized Trials in Education Research. Brookings Institution, publisher's information at http://bit.ly/UoX3sA. Amazon.com information at http://amzn.to/n6T0Uo. An expurgated Google book preview is online at http://bit.ly/RX1k3u.
USDE. 2003. U.S. Department of Education, Identifying and Implementing Educational Practices Supported by Rigorous Evidence: A User Friendly Guide. Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, online as a 140 kB pdf at http://bit.ly/ds1sRS.
Wood, W.B., & J.M. Gentile. 2003. “Teaching in a Research Context,” Science 302: 1510; 28 November; online as a 213 kB pdf at http://bit.ly/SyhOvL thanks to Portland State's “Ecoplexity” site.
Showing posts with label Gold Standard. Show all posts
Showing posts with label Gold Standard. Show all posts
Sunday, December 9, 2012
Tuesday, December 4, 2012
Randomized Control Trials - The Tarnished Gold Standard
Some blog followers might be interested in a recent post “Randomized Control Trials - The Tarnished Gold Standard” [Hake (2012a)]. The abstract reads:
************************************************
ABSTRACT: In response to “The Randomistas' War On Global Poverty - Erratum & Addendum” at http://bit.ly/YfMESg, Art Burke of the EvalTalk list pointed to an NYT piece “Applying Evidence to Social Programs” by Jon Baron at http://nyti.ms/Um9vVI.
Baron wrote (slightly edited): “Scientifically rigorous studies - particularly, the 'gold standard' of Randomized Controlled Trials (RCT’s) - are a mainstay of medicine, providing conclusive evidence of effectiveness for most major medical advances in recent history. In social spending, by contrast, such studies have only a toehold. Where they have been used, however, they have demonstrated the same ability to produce important, credible evidence about what works - and illuminated a path to major progress.”
In this post I cite arguments that the “gold standard” RCT studies may not be as lustrous as claimed by Baron:
(1) Ever since the pioneering work of Halloun & Hestenes (1985a) at http://bit.ly/fDdJHm, physicists have been engaged in social science of Physics Education Research (PER) that has made useful, reliable, and nonobvious predictions without resort to RCT’s - e.g. “Why Not Try a Scientific Approach to Science Education?” [Wieman (2007)] at http://bit.ly/anTMfF.
(2) In “A Response to ‘It's Not All About Class Size’ ” [Hake (2009)], I pointed out that according to the California Class Size Reduction Research Consortium [CCSRRC (2002)] at http://bit.ly/V923Ms, California's attempt to duplicate the vaunted Tennessee RCT study of reduced class size benefits results yielded no conclusive evidence of increased student achievement.
(3) In “A Summative Evaluation of RCT Methodology: & An Alternative Approach to Causal Research” [Scriven (2008] at http://bit.ly/93VcWD wrote: “In standard scientific usage, experiments are just carefully constrained explorations, and the RCT is simply a special case of these. To call the RCT the only ‘true experiment’ is part of an attempt at redefinition that distorts the original and continuing usage, and excludes experiments designed to test many simple hypotheses about - or simple efforts to find out - what happens if we do this.”
(4) In “Seventeen Statements by Gold-Standard Skeptics #2” [Hake (2010)] at http://bit.ly/TNpTR9 I cite, among others, the comments of the American Education Research Association; the American Evaluation Association; the National Education Association; the European Evaluation Society; Thomas Cook and Monique Payne, Hugh Burkhardt & Alan Schoenfeld; Margaret Eisenhart & Lisa Towne; Burke Johnson; Annette Lareau & Pamela Barnhouse; Joseph Maxwell; Dennis Phillips; Barbara Schneider, Martin Carnoy, Jeremy Kilpatrick, William Schmidt, & Richard Shavelson; Mack Shelley, Larry Yore, & Brian Hand; Deborah Stipek; and Carol Weiss.
(5) In "Why Most Published Research Findings Are False," John Ioannidis' (2005 at http://1.usa.gov/YxUxkL states ". . . . there is strong evidence that selective outcome reporting, with manipulation of the outcomes and analyses reported, is a common problem even for randomized trails [Chan et al. (2004)] at http://1.usa.gov/X8SB1T.
(6) the present signature quote of Thomas Cook and Monique Payne.
************************************************
To access the complete 18 kB post please click on http://bit.ly/VzVc0K.
Richard Hake, Emeritus Professor of Physics, Indiana University
Links to Articles: http://bit.ly/a6M5y0
Links to Socratic Dialogue Inducing (SDI) Labs: http://bit.ly/9nGd3M
Academia: http://bit.ly/a8ixxm
Blog: http://bit.ly/9yGsXh
GooglePlus: http://bit.ly/KwZ6mE
Twitter: http://bit.ly/juvd52
"In some quarters, particularly medical ones, the randomized experiment is considered the causal 'gold standard.' It is clearly not that in educational contexts, given the difficulties with implementing and maintaining randomly created groups, with the sometimes incomplete implementation of treatment particulars, with the borrowing of some treatment particulars by control group units, and with the limitations to external validity that often follow from how the random assignment is achieved."
- Thomas Cook and Monique Payne in "Evidence Matters" [Mosteller & Boruch (2002)]
REFERENCES [URL shortened by http://bit.ly/ and accessed on 04 Dec 2012.]
Hake, R.R. 2012a. "Randomized Control Trials - The Tarnished Gold Standard" online on the OPEN! AERA-L archives at http://bit.ly/VzVc0K. Post of 4 Dec 2012 19:26:48-0800 to AERA-L and Net-Gold. The abstract and link to the complete post are being transmitted to several discussion lists.
Hake, R.R. 2012b. “The Randomistas' War On Global Poverty - ERRATUM & ADDENDUM,” online on the OPEN! AERA-L archives at http://bit.ly/YfMESg. Post of 30 Nov 2012 12:15:33-0800 to AERA-L and Net-Gold. The abstract and link to the complete post are being transmitted to several discussion lists, and are also on my blog “Hake'sEdStuff” at http://bit.ly/11c5w3e with a provision for comments.
Mosteller, F. & R. Boruch, eds. 2002. Evidence Matters: Randomized Trials in Education Research. Brookings Institution, publisher's information at http://bit.ly/UoX3sA. Amazon.com information at http://amzn.to/n6T0Uo. An expurgated Google book preview is online at http://bit.ly/RX1k3u.
************************************************
ABSTRACT: In response to “The Randomistas' War On Global Poverty - Erratum & Addendum” at http://bit.ly/YfMESg, Art Burke of the EvalTalk list pointed to an NYT piece “Applying Evidence to Social Programs” by Jon Baron at http://nyti.ms/Um9vVI.
Baron wrote (slightly edited): “Scientifically rigorous studies - particularly, the 'gold standard' of Randomized Controlled Trials (RCT’s) - are a mainstay of medicine, providing conclusive evidence of effectiveness for most major medical advances in recent history. In social spending, by contrast, such studies have only a toehold. Where they have been used, however, they have demonstrated the same ability to produce important, credible evidence about what works - and illuminated a path to major progress.”
In this post I cite arguments that the “gold standard” RCT studies may not be as lustrous as claimed by Baron:
(1) Ever since the pioneering work of Halloun & Hestenes (1985a) at http://bit.ly/fDdJHm, physicists have been engaged in social science of Physics Education Research (PER) that has made useful, reliable, and nonobvious predictions without resort to RCT’s - e.g. “Why Not Try a Scientific Approach to Science Education?” [Wieman (2007)] at http://bit.ly/anTMfF.
(2) In “A Response to ‘It's Not All About Class Size’ ” [Hake (2009)], I pointed out that according to the California Class Size Reduction Research Consortium [CCSRRC (2002)] at http://bit.ly/V923Ms, California's attempt to duplicate the vaunted Tennessee RCT study of reduced class size benefits results yielded no conclusive evidence of increased student achievement.
(3) In “A Summative Evaluation of RCT Methodology: & An Alternative Approach to Causal Research” [Scriven (2008] at http://bit.ly/93VcWD wrote: “In standard scientific usage, experiments are just carefully constrained explorations, and the RCT is simply a special case of these. To call the RCT the only ‘true experiment’ is part of an attempt at redefinition that distorts the original and continuing usage, and excludes experiments designed to test many simple hypotheses about - or simple efforts to find out - what happens if we do this.”
(4) In “Seventeen Statements by Gold-Standard Skeptics #2” [Hake (2010)] at http://bit.ly/TNpTR9 I cite, among others, the comments of the American Education Research Association; the American Evaluation Association; the National Education Association; the European Evaluation Society; Thomas Cook and Monique Payne, Hugh Burkhardt & Alan Schoenfeld; Margaret Eisenhart & Lisa Towne; Burke Johnson; Annette Lareau & Pamela Barnhouse; Joseph Maxwell; Dennis Phillips; Barbara Schneider, Martin Carnoy, Jeremy Kilpatrick, William Schmidt, & Richard Shavelson; Mack Shelley, Larry Yore, & Brian Hand; Deborah Stipek; and Carol Weiss.
(5) In "Why Most Published Research Findings Are False," John Ioannidis' (2005 at http://1.usa.gov/YxUxkL states ". . . . there is strong evidence that selective outcome reporting, with manipulation of the outcomes and analyses reported, is a common problem even for randomized trails [Chan et al. (2004)] at http://1.usa.gov/X8SB1T.
(6) the present signature quote of Thomas Cook and Monique Payne.
************************************************
To access the complete 18 kB post please click on http://bit.ly/VzVc0K.
Richard Hake, Emeritus Professor of Physics, Indiana University
Links to Articles: http://bit.ly/a6M5y0
Links to Socratic Dialogue Inducing (SDI) Labs: http://bit.ly/9nGd3M
Academia: http://bit.ly/a8ixxm
Blog: http://bit.ly/9yGsXh
GooglePlus: http://bit.ly/KwZ6mE
Twitter: http://bit.ly/juvd52
"In some quarters, particularly medical ones, the randomized experiment is considered the causal 'gold standard.' It is clearly not that in educational contexts, given the difficulties with implementing and maintaining randomly created groups, with the sometimes incomplete implementation of treatment particulars, with the borrowing of some treatment particulars by control group units, and with the limitations to external validity that often follow from how the random assignment is achieved."
- Thomas Cook and Monique Payne in "Evidence Matters" [Mosteller & Boruch (2002)]
REFERENCES [URL shortened by http://bit.ly/ and accessed on 04 Dec 2012.]
Hake, R.R. 2012a. "Randomized Control Trials - The Tarnished Gold Standard" online on the OPEN! AERA-L archives at http://bit.ly/VzVc0K. Post of 4 Dec 2012 19:26:48-0800 to AERA-L and Net-Gold. The abstract and link to the complete post are being transmitted to several discussion lists.
Hake, R.R. 2012b. “The Randomistas' War On Global Poverty - ERRATUM & ADDENDUM,” online on the OPEN! AERA-L archives at http://bit.ly/YfMESg. Post of 30 Nov 2012 12:15:33-0800 to AERA-L and Net-Gold. The abstract and link to the complete post are being transmitted to several discussion lists, and are also on my blog “Hake'sEdStuff” at http://bit.ly/11c5w3e with a provision for comments.
Mosteller, F. & R. Boruch, eds. 2002. Evidence Matters: Randomized Trials in Education Research. Brookings Institution, publisher's information at http://bit.ly/UoX3sA. Amazon.com information at http://amzn.to/n6T0Uo. An expurgated Google book preview is online at http://bit.ly/RX1k3u.
Monday, July 11, 2011
Re: controlled experiments
Some blog followers might be interested in a discussion-list post “Re: controlled experiments” [Hake (2011)].
The abstract reads:
*********************************************
ABSTRACT: PhysLnrR’s Brian Foley wrote (paraphrasing):
“I would love to be doing controlled experiments much of the time - but they are darn near impossible to pull off. . . . . . Once you have your 40 approved classrooms, you RANDOMLY SELECT 20 TEACHERS and train them on your innovation. . . . . Also your 20 control teachers need to be using their ‘traditional’ teaching (and specifically teach like they have never heard of your innovation. . . . . and if you have designed some good assessments of learning, then you will finally have your result. . . . . .and if your innovation makes a difference YOU JUST MIGHT GET THE MAGICAL p less than 0.05 RESULT.”
Brian seems to have succumbed to the siren calls of the Gold Standardistas and the Statistical Significance Cultists. Modesty forbids mention of these possible antidotes:
a. “Should Randomized Control Trials Be the Gold Standard of Educational Research?” at http://bit.ly/qrUfFz ,
b. “Seventeen Statements by Gold-Standard Skeptics #2” at http://bit.ly/oRGnBp ,
c. “The Cult of Statistical Significance” at http://bit.ly/dkTyXP .
*********************************************
To access the complete 10 kB post please click on http://bit.ly/onA7jk .
Richard Hake, Emeritus Professor of Physics, Indiana University Honorary Member, Curmudgeon Lodge of Deventer, The Netherlands
President, PEdants for Definitive Academic References which Recognize the Invention of the Internet (PEDARRII)
rrhake@earthlink.net
http://www.physics.indiana.edu/~hake
http://www.physics.indiana.edu/~sdi
http://HakesEdStuff.blogspot.com
http://iub.academia.edu/RichardHake
“In some quarters, particularly medical ones, the randomized experiment is considered the causal ‘gold standard.’ IT IS CLEARLY NOT THAT IN EDUCATIONAL CONTEXTS, given the difficulties with implementing and maintaining randomly created groups, with the sometimes incomplete implementation of treatment particulars, with the borrowing of some treatment particulars by control group units, and with the limitations to external validity that often follow from how the random assignment is achieved.”
- Tom Cook & Monique Payne (2002, p. 174)
“After 4 decades of severe criticism, the ritual of null hypothesis significance testing - mechanical dichotomous decisions around a sacred 0.05 criterion - still persists. This article reviews the problems with this practice, including its near-universal misinterpretation of p as the probability that Ho . . . .[[the null hypothesis]]. . . . is false, the misinterpretation that its complement is the probability of successful replication, and the mistaken assumption that if one rejects Ho one thereby affirms the theory that led to the test. Exploratory data analysis and the use of graphic methods, a steady improvement in and a movement toward standardization in measurement, and emphasis on effect sizes using confidence intervals, and the informed use of available statistical methods is suggested. FOR GENERALIZATION, PSYCHOLOGISTS MUST FINALLY RELY, AS HAS BEEN DONE IN THE OLDER SCIENCES, ON REPLICATION.” [My CAPS.]
- Jacob Cohen (1994) in “The earth is round (p less than 0.05)”
REFERENCES [URL’s shortened by http://bit.ly/ and accessed on 11 July 2011.]
Cook, T.D. & M.R. Payne. 2002. “Objecting to the Objections to Using Random Assignment in Educational Research” in Mosteller & Boruch (2002).
Cohen, J. 1994. “The earth is round (p less than 0.05),” American Psychologist 49: 997-1003; online as a 1.2 MB pdf at http://bit.ly/a45I2t thanks to Christopher Green http://www.yorku.ca/christo/.
Hake, R.R. 2011. “Re: controlled experiments,” online on the OPEN! AERA-L archives at http://bit.ly/onA7jk. Post of 11 Jul 2011 11:15:54-0700 to AERA-L, Net-Gold, and PhysLrnR. The abstract and link to the complete post are being transmitted to various discussion lists.
Mosteller, F. & R. Boruch, eds. 2002. Evidence Matters: Randomized Trials in Education Research. Brookings Institution. Amazon.com information at http://amzn.to/n6T0Uo . A searchable expurgated Google Book Preview is online at http://bit.ly/mTcPIE .
The abstract reads:
*********************************************
ABSTRACT: PhysLnrR’s Brian Foley wrote (paraphrasing):
“I would love to be doing controlled experiments much of the time - but they are darn near impossible to pull off. . . . . . Once you have your 40 approved classrooms, you RANDOMLY SELECT 20 TEACHERS and train them on your innovation. . . . . Also your 20 control teachers need to be using their ‘traditional’ teaching (and specifically teach like they have never heard of your innovation. . . . . and if you have designed some good assessments of learning, then you will finally have your result. . . . . .and if your innovation makes a difference YOU JUST MIGHT GET THE MAGICAL p less than 0.05 RESULT.”
Brian seems to have succumbed to the siren calls of the Gold Standardistas and the Statistical Significance Cultists. Modesty forbids mention of these possible antidotes:
a. “Should Randomized Control Trials Be the Gold Standard of Educational Research?” at http://bit.ly/qrUfFz ,
b. “Seventeen Statements by Gold-Standard Skeptics #2” at http://bit.ly/oRGnBp ,
c. “The Cult of Statistical Significance” at http://bit.ly/dkTyXP .
*********************************************
To access the complete 10 kB post please click on http://bit.ly/onA7jk .
Richard Hake, Emeritus Professor of Physics, Indiana University Honorary Member, Curmudgeon Lodge of Deventer, The Netherlands
President, PEdants for Definitive Academic References which Recognize the Invention of the Internet (PEDARRII)
rrhake@earthlink.net
http://www.physics.indiana.edu/~hake
http://www.physics.indiana.edu/~sdi
http://HakesEdStuff.blogspot.com
http://iub.academia.edu/RichardHake
“In some quarters, particularly medical ones, the randomized experiment is considered the causal ‘gold standard.’ IT IS CLEARLY NOT THAT IN EDUCATIONAL CONTEXTS, given the difficulties with implementing and maintaining randomly created groups, with the sometimes incomplete implementation of treatment particulars, with the borrowing of some treatment particulars by control group units, and with the limitations to external validity that often follow from how the random assignment is achieved.”
- Tom Cook & Monique Payne (2002, p. 174)
“After 4 decades of severe criticism, the ritual of null hypothesis significance testing - mechanical dichotomous decisions around a sacred 0.05 criterion - still persists. This article reviews the problems with this practice, including its near-universal misinterpretation of p as the probability that Ho . . . .[[the null hypothesis]]. . . . is false, the misinterpretation that its complement is the probability of successful replication, and the mistaken assumption that if one rejects Ho one thereby affirms the theory that led to the test. Exploratory data analysis and the use of graphic methods, a steady improvement in and a movement toward standardization in measurement, and emphasis on effect sizes using confidence intervals, and the informed use of available statistical methods is suggested. FOR GENERALIZATION, PSYCHOLOGISTS MUST FINALLY RELY, AS HAS BEEN DONE IN THE OLDER SCIENCES, ON REPLICATION.” [My CAPS.]
- Jacob Cohen (1994) in “The earth is round (p less than 0.05)”
REFERENCES [URL’s shortened by http://bit.ly/ and accessed on 11 July 2011.]
Cook, T.D. & M.R. Payne. 2002. “Objecting to the Objections to Using Random Assignment in Educational Research” in Mosteller & Boruch (2002).
Cohen, J. 1994. “The earth is round (p less than 0.05),” American Psychologist 49: 997-1003; online as a 1.2 MB pdf at http://bit.ly/a45I2t thanks to Christopher Green http://www.yorku.ca/christo/.
Hake, R.R. 2011. “Re: controlled experiments,” online on the OPEN! AERA-L archives at http://bit.ly/onA7jk. Post of 11 Jul 2011 11:15:54-0700 to AERA-L, Net-Gold, and PhysLrnR. The abstract and link to the complete post are being transmitted to various discussion lists.
Mosteller, F. & R. Boruch, eds. 2002. Evidence Matters: Randomized Trials in Education Research. Brookings Institution. Amazon.com information at http://amzn.to/n6T0Uo . A searchable expurgated Google Book Preview is online at http://bit.ly/mTcPIE .
Monday, April 26, 2010
Seventeen Statements by Gold-Standard Skeptics #2
Some blog followers might be interested in a post of the above title. The abstract reads:
***************************************
ABSTRACT: Andy Rudd in an EdResMeth post 6 Apr 2010 titled "Cause and Effect" wrote: "Today I dealt with a doctoral student who was adamantly opposed to the idea that causal relationships can be studied using non experimental designs. . . . . . I am curious what others think about the use of non experimental designs to study causal relationships if it is not possible to use an experimental or quasi-experimental design."
This initiated an 18-post thread of diverse comments on the student's opinion, accessible at http://tinyurl.com/y4um3g3 . Rudd's student may have been influenced by the fact that the "Randomized Control Trial" has been enthroned by the U.S. Dept. of Education (USDE, 2008) and Mosteller and Boruch (2002) as the "gold standard" for demonstrating causality in education research.
For consideration by Rudd's student and others, herewith are SEVENTEEN STATEMENTS BY GOLD-STANDARD SKEPTICS:
(1) American Education Research Association;
(2) American Evaluation Association;
(3) Hugh Burkhardt and Alan Schoenfeld;
(4) Tom Cook and Monique Payne;
(5) Margaret Eisenhart and Lisa Towne;
(6) European Evaluation Society;
(7) Richard Hake;
(8) Burke Johnson;
(9) Annette Lareau and Pamela Barnhouse;
(10) Joseph Maxwell;
(11) National Education Association;
(12) Dennis Phillips;
(13) Barbara Schneider, Martin Carnoy, Jeremy Kilpatrick, William Schmidt, and Richard Shavelson;
(14) Michael Scriven;
(15) Mack Shelley, Larry Yore, and Brian Hand;
(16) Deborah Stipek; and
(17) Carol Weiss.
***************************************
To access the complete 85 kB post please click on http://tinyurl.com/3az7tvu .
Richard Hake, Emeritus Professor of Physics, Indiana University
Honorary Member, Curmudgeon Lodge of Deventer, The Netherlands
President, PEdants for Definitive Academic References which Recognize the Invention of the Internet (PEDARRII)
rrhake@earthlink.net
http://www.physics.indiana.edu/~hake
http://www.physics.indiana.edu/~sdi
http://HakesEdStuff.blogspot.com
http://iub.academia.edu/RichardHake
REFERENCES [Tiny URL's courtesy http://tinyurl.com/create.php .]
Hake, R.R. 2010. "Seventeen Statements by Gold-Standard Skeptics," online on the OPEN! AERA-L archives at http://tinyurl.com/3az7tvu . Post of 27 Apr 2010 10:03:58-0700 to AERA-L and Net-Gold. The abstract and link to the complete post are also being transmitted to various discussion lists.
***************************************
ABSTRACT: Andy Rudd in an EdResMeth post 6 Apr 2010 titled "Cause and Effect" wrote: "Today I dealt with a doctoral student who was adamantly opposed to the idea that causal relationships can be studied using non experimental designs. . . . . . I am curious what others think about the use of non experimental designs to study causal relationships if it is not possible to use an experimental or quasi-experimental design."
This initiated an 18-post thread of diverse comments on the student's opinion, accessible at http://tinyurl.com/y4um3g3 . Rudd's student may have been influenced by the fact that the "Randomized Control Trial" has been enthroned by the U.S. Dept. of Education (USDE, 2008) and Mosteller and Boruch (2002) as the "gold standard" for demonstrating causality in education research.
For consideration by Rudd's student and others, herewith are SEVENTEEN STATEMENTS BY GOLD-STANDARD SKEPTICS:
(1) American Education Research Association;
(2) American Evaluation Association;
(3) Hugh Burkhardt and Alan Schoenfeld;
(4) Tom Cook and Monique Payne;
(5) Margaret Eisenhart and Lisa Towne;
(6) European Evaluation Society;
(7) Richard Hake;
(8) Burke Johnson;
(9) Annette Lareau and Pamela Barnhouse;
(10) Joseph Maxwell;
(11) National Education Association;
(12) Dennis Phillips;
(13) Barbara Schneider, Martin Carnoy, Jeremy Kilpatrick, William Schmidt, and Richard Shavelson;
(14) Michael Scriven;
(15) Mack Shelley, Larry Yore, and Brian Hand;
(16) Deborah Stipek; and
(17) Carol Weiss.
***************************************
To access the complete 85 kB post please click on http://tinyurl.com/3az7tvu .
Richard Hake, Emeritus Professor of Physics, Indiana University
Honorary Member, Curmudgeon Lodge of Deventer, The Netherlands
President, PEdants for Definitive Academic References which Recognize the Invention of the Internet (PEDARRII)
rrhake@earthlink.net
http://www.physics.indiana.edu/~hake
http://www.physics.indiana.edu/~sdi
http://HakesEdStuff.blogspot.com
http://iub.academia.edu/RichardHake
REFERENCES [Tiny URL's courtesy http://tinyurl.com/create.php .]
Hake, R.R. 2010. "Seventeen Statements by Gold-Standard Skeptics," online on the OPEN! AERA-L archives at http://tinyurl.com/3az7tvu . Post of 27 Apr 2010 10:03:58-0700 to AERA-L and Net-Gold. The abstract and link to the complete post are also being transmitted to various discussion lists.
Monday, February 22, 2010
Re: Quality Research in Literacy and Science Education: International Perspectives and Gold Standards
Some blog followers may be interested a post [Hake (2010)] of the above title. The abstract reads:
Most of the contributors to SYH are skeptical of the U.S. Dept. of Education's enthronement of Randomized Control Trials (RCT's) as the “Gold Standard” of education research, consistent with the views of most evaluation experts.
On the other hand, the authors contributing to SYH appear to be either dismissive or oblivious of physics education research, inconsistent with the generally positive opinions of most observers.
For example, Millar and Osborne(2009) make the following erroneous claims (paraphrasing): “No standard or commonly agreed outcome measures exist for any major topic. Published assessment tools such as the 'Force Concept Inventory' have not been subjected to the kind of rigorous scrutiny of factorial structure and content validity that would be standard practice for measures of attainment or learning outcome in other subject areas.”
Despite SYH's cavalier dismissal of physics education research, I think some discussion-list subscribers might consider scanning a begged, borrowed, or stolen copy of SYH, before paying Springer's piratical $215
*****************************************
To access the complete 45 kB post please click on http://tinyurl.com/yhhbu72.
Hake, R.R. 2010. “Re: Quality Research in Literacy and Science Education: International Perspectives and Gold Standards,” online on the OPEN! AERA-L archives at http://tinyurl.com/yhhbu72 . Post of 22 Feb 2010 14:04:43-0800 to AERA-L and Net-Gold. The abstract is being sent to various discussion lists.
Millar, R. & J. Osborne. 2009. “Research and Practice: A Complex Relationship?” Chapter 3, pages 41-62, of Shelley et al. (2009a). Surprisingly, the Google book preview of Shelley et al. (2009a) at http://tinyurl.com/yddphh3 contains all of pages 41-62. To see click repeatedly on the “>” at the top of the first page to go to page vi and then click on chapter 3.
Shelley, M.C., L.D. Yore, & B. Hand, eds. 2009a. Quality Research in Literacy and Science Education: International Perspectives and Gold Standards.” Springer, publisher's information at http://www.springerlink.com/content/g2447682464446x2/. Amazon.com information at http://tinyurl.com/yf7efra, note the searchable “Look Inside” feature. Barnes & Noble information at http://tinyurl.com/y8n9pe9. An expurgated (teaser) version is online as a Google “book preview” at http://tinyurl.com/yddphh3.
Subscribe to:
Posts (Atom)
