Showing posts with label normalized gain. Show all posts
Showing posts with label normalized gain. Show all posts

Saturday, March 19, 2011

Re: Interactive Engagement Typically Lowers Student Evaluations of Teaching?

Some blog followers might be interested in discussion-list post “Re: Interactive Engagement Typically Lowers Student Evaluations of Teaching?” [Hake (2011)].


The abstract reads:


**********************************************************

ABSTRACT: PhysLrnR’s Bill Goffe wrote (paraphrasing): “I thought I recalled reading here that interactive engagement typically lowers student evaluations of teaching, but I've not been able any such claims in the literature.”


Goffe's post initiated a 17-post thread (as of 19 March 15:47-0700) accessible at http://bit.ly/i9zBsd to those who take a few minutes to subscribe to PhysLrnR at http://bit.ly/beuikb.


Bill may have overlooked my post “Re: What if students learn better in a course they don't like?” [Hake (2006)]. Therein I wrote (condensing and paraphrasing):


“When I first started teaching an introductory physics course I followed the example of teaching-award-winning faculty and taught in a traditional manner: passive student lectures, lots of exciting demos, algorithmic problem exams, recipe labs, and a relatively easy final exam. I was gratified to receive a Student Evaluation of Teaching (SET) evaluation point average EPA = 3.38 [B plus on a scale of 1 - 4] for ‘overall evaluation of professor.’ Had I continued using traditional methods and giving easy exams I would doubtless have risen to become the U.S. Secretary of Education, or at least President of Indiana University.


Unfortunately for my academic career, I gradually caught on to the fact that students’ conceptual understanding of physics was not substantively increased by traditional pedagogy. I converted to the ‘Arons Advocated Method’ http://bit.ly/boeQQt of ‘interactive engagement.’ This resulted in average normalized gains g(ave) on ‘Force Concept Inventory’ that ranged from 0.54 to 0.65 as compared to the g(ave) of about 0.2 typically obtained in traditional introductory mechanics courses.


But my EPA’s for ‘overall evaluation of professor,’ sometimes dipped to as low as 1.67 (C-), and never returned to the 3.38 high that I had garnered by using traditional ineffective methods. My department chair and his executive committee, convinced by the likes of Peter Cohen (1981, 1990) that SET’s are valid measures of the cognitive impact of introductory courses, took a very dim view of both my teaching and my educational activities.”

**********************************************************


To access the complete 13 kB post please click on http://bit.ly/gKWO1S.


Richard Hake, Emeritus Professor of Physics, Indiana University

Honorary Member, Curmudgeon Lodge of Deventer, The Netherlands

President, PEdants for Definitive Academic References which Recognize theInvention of the Internet (PEDARRII)


rrhake@earthlink.net>

http://www.physics.indiana.edu/~hake

http://www.physics.indiana.edu/~sdi

http://HakesEdStuff.blogspot.com

http://iub.academia.edu/RichardHake


“Few faculty members have any awareness of the expanding knowledge about learning from psychology and cognitive science. Almost no one in the academy has mastered or used this knowledge base. One of my colleagues observed that if doctors used science the way college teachers do, they would still be trying to heal with leeches.”


- James Duderstadt (2000), President Emeritus and University Professor of Science and Engineering at the University of Michigan


REFERENCES [All URL's shortened by http://bit.ly/ and accessed on 19 March 2011.]


Duderstadt, J.J. 2000. A University for the 21st Century. Univ. of Michigan Press, publisher's information at http://bit.ly/cvJ1yI. Amazon.com information at http://amzn.to/fUnbj5, note the “Look Inside” feature.


Hake, R.R. 2011. “Re: Interactive Engagement Typically Lowers Student Evaluations of Teaching?” online on the OPEN! AERA-L archives at http://bit.ly/gKWO1S. Post of 19 Mar 2011 15:51:49-0700 to AERA-L, Net-Gold, and PhysLrnR. The abstract and link to the complete 13 kB post are also being transmitted to various discussion lists.


Saturday, January 22, 2011

How Much Value is Added at Elite Institutions?

Some blog followers might be interested in a recent post “How Much Value is Added at Elite Institutions?” [Hake (2011d)]. The abstract reads:


**********************************

ABSTRACT: In a previous post “Value-Added Inequities: Should Value-Added Measures Be Used to Evaluate Teachers?” [Hake (2010b)] I implied that the less-than-stellar value-added rankings of Boris Korsunsky’s high school and Stuyvesant High School (each with top-tier reputations) were examples of “Value-Added Inequities.” I thank Catherine Johnson for correctly pointing out that those two appraisals were not necessarily inequitable - they could, in fact, be correct.


Two cases in point are the less-than-stellar value-added assessments of instruction at two elite institutions: (1) Eric Mazur's traditional 1990 calculus-based introductory course at Harvard, and (2) traditional introductory courses in electromagnetism a MIT. Both assessments are correct as judged by the value-added assessment provided by the average normalized pre-to-posttest gain on valid tests of students' conceptual understanding. Fortunately, in both cases “interactive engagement” pedagogy greatly improved normalized the pre-to-posttest gains in those courses: (1) Mazur switched to “Peer Instruction,” as is engagingly described by Mazur (2009) in “Confessions of a Converted Lecturer” on UTube at http://bit.ly/dBYsXh; and (2) John Belcher instituted TEAL (Technology Enabled Active Learning), as is cogently reported in the New Your Times by Sarah Rimer (2009) in “At M.I.T., Large Lectures Are Going the Way of the Blackboard” at http://nyti.ms/e3JtYN .


In my opinion, demonstrations that the less-than-stellar value-added assessments of Korsunsky's high school and Stuyvesant High School are inequitable would require meaningful value-added measures such as normalized average pre-to-posttest gains on valid and consistently reliable tests of higher-order learning developed by disciplinary experts, not the value-added measures that characterize “Race to the Top,” and that have been called into question by the many expert panels listed in my previous post “Value-Added Inequities: Should Value-Added Measures Be Used to Evaluate Teachers?”


It is conceivable that if there were “Eric Mazurs” or “John Belchers” at Korsunsky's high-school and the Stuyvesant High School, scenarios similar to those at Harvard and MIT might occur, even though all those institutions are regarded as “elite.”

**********************************


To access the complete 36 kB post please click on http://bit.ly/g25OHd.


Richard Hake, Emeritus Professor of Physics, Indiana University

Honorary Member, Curmudgeon Lodge of Deventer, The Netherlands

President, PEdants for Definitive Academic References which Recognize the Invention of the Internet (PEDARRII)


http://www.physics.indiana.edu/~hake

http://www.physics.indiana.edu/~sdi

http://HakesEdStuff.blogspot.com

http://iub.academia.edu/RichardHake


“I point to the following unwelcome truth: much as we might dislike the implications, research is showing that didactic exposition of abstract ideas and lines of reasoning (however engaging and lucid we might try to make them) to passive listeners yields pathetically thin results in learning and understanding - except in the very small percentage of students who are specially gifted in the field.” - Arnold Arons (1997, p. vii)


REFERENCES [URL’s shortened by http://bit.ly/ and accessed on 22 Jan 2011.]


Arons, A.B. 1997. Teaching Introductory Physics. Wiley. Amazon.com information at http://amzn.to/bBPfop . Note the searchable “Look Inside” feature.


Hake, R.R. 2011a. “The Ceiling Effect #2” online on the OPEN! AERA-L archives at http://bit.ly/hUnHZe . Post of 12 Jan 2011 16:19:49-0800 to AERA-L and Net-Gold. The abstract and link to the complete post were transmitted to various discussion lists are also online on my blog “Hake'sEdStuff” at http://bit.ly/gLWr7W with a provision for comments.


Hake, R.R. 2011b. “Value-Added Inequities: Should Value-Added Measures Be Used to Evaluate Teachers?” online on the OPEN! AERA-L archives at http://bit.ly/fN1HmD. Post of 18 Jan 2011 15:34:47-0800 to AERA-L and Net-Gold. The abstract and link to the complete post were transmitted to various discussion lists are also online on my blog “Hake'sEdStuff” at http://bit.ly/h23shQ with a provision for comments.


Hake, R.R. 2011c. “Value-Added Inequities: Should Value-Added Measures Be Used to Evaluate Teachers?” online on the OPEN! AERA-L archives at http://bit.ly/fAvRpA . Post of 19 Jan 2011 11:36:22 -0800 to AERA-L, EDDRA2, Math-Teach, Net-Gold, and PhysLnR.


Hake, R.R. 2011d. “How Much Value is Added at Elite Institutions?” online on the OPEN! AERA-L archives at http://bit.ly/g25OHd. Post of 22 Jan 2011 14:50:14-0800 to AERA-L and Net-Gold. The abstract and link to the complete post are being transmitted to various discussion lists. See also Hake (2011 a,b,c).

Wednesday, January 12, 2011

The Ceiling Effect #2

Some blog followers might be interested in a recent post “The Ceiling Effect #2” [Hake (2011)]. The abstract reads:


******************************************

ABSTRACT: Sheila Tobias (2010) in a recent APS News “Back Page” editorial “Teachers in the Crosshairs. . .”: wrote:


“In Houston, where a new value-added formula is being used to grade teachers’ skills, ‘N.Y. Times’ education writer Sam Dillon found a high school physics teacher out of the running for high bonuses because, as she reported, ‘My kids come in at a very high level of competence, scoring well before the semester begins.’ After she teaches them for a year, they continue to score well on a state science test but show less measurable ‘gain’ than other classes, so her bonus is small compared with those of other teachers. The teacher has invented a term of her own to characterize this distortion. She calls it the ‘ceiling effect.’ We might call it a saturation effect.”


It's unfortunate that the teacher’s bonus assigners were not aware that it’s the “average normalized gain” [g] = ([%post] – [%pre]) / (100% - [%pre]) [the square brackets [. . .] indicate class averages], often used in physics education research, that can (in some cases) be used as a gauge of relative course effectiveness, NOT the absolute average gain [G] = ([%post] – [%pre]).

******************************************


To access the complete 11 kB post please click on http://bit.ly/hUnHZe .


Richard Hake, Emeritus Professor of Physics, Indiana University

Honorary Member, Curmudgeon Lodge of Deventer, The Netherlands

President, PEdants for Definitive Academic References which Recognize the Invention of the Internet (PEDARRII)


rrhake@earthlink.net>

http://www.physics.indiana.edu/~hake

http://www.physics.indiana.edu/~sdi

http://HakesEdStuff.blogspot.com

http://iub.academia.edu/RichardHake


REFERENCES [URL’s shortened by http://bit.ly/ and accessed on 12 Jan 2010.]


Hake, R.R. 2011. “The Ceiling Effect #2” online on the OPEN! AERA-L archives at http://bit.ly/hUnHZe . Post of 12 Jan 2011 16:19:49-0800 to AERA-L and Net-Gold. The abstract and link to the complete post are being transmitted to various discussion lists.


Tobias, S. 2010. “Teachers in the Crosshairs: The Impact of ‘School Choice,’ ‘Reform,’ and ‘Accountability’,” APS News 20(1), January; online at http://bit.ly/fc6jEj .

Wednesday, January 5, 2011

Learning Outcomes: Face-to-Face vs Online #3

Some blog followers might be interested in a recent post “Learning Outcomes: Face-to-Face vs Online #3” [Hake (2011)]. The abstract reads:


*************************************

ABSTRACT: In an earlier post, “Learning Outcomes: Face-to-Face vs Online,” I responded to a question posed by STLHE-L’s Martin Rosenzweig: “Does anyone know of any published studies comparing online to face-to-face instruction with regards to learning outcomes?” I wrote: “As far as I know the answer is ‘NO.’ The reason is [as pointed out in ‘Can Distance and Classroom Learning Be Increased?’ (Hake, 2008a)] ‘scholars of teaching and learning continue to rely on low-resolution gauges of students’ learning.’ ”


In response, several discussion-list subscribers called attention to “Evaluation of Evidence-Based Practices in Online Learning: A Meta-Analysis and Review of Online Learning Studies” [USDE (2009)] at http://bit.ly/e1VXvA, of which I had been unaware.


In my opinion, the USDE (2009) study is yet another example of reliance on low-resolution gauges of students’ learning. On pages 11-12 it is stated that examples of learning outcome measures included: (a) scores on standardized tests, (b) scores on researcher-created assessments, (c) grades/scores on teacher-created assessments (e.g., assignments, midterm/final exams), and (d) grades or grade point averages.


But among lessons of the physics education research effort [Hake (2002)] are that: (1) “c” and “d” are *invalid* measures of students' *higher-order* learning; and (2) analyses of “a” and “b” are best carried out in terms of the course average *normalized* gain [g], ignored in USDE (2009).


Furthermore, on page 18 of http://bit.ly/e1VXvA the USDE report states: “The mean effect size for all 50 contrasts of online vs face-to-face instruction was +0.20.”


Contrast the above with the effect size d = +2.43 for the superiority of [[g]] (the average of the course average [g]) for 48 face-to-face “interactive engagement” physics courses vs 14 face-to-face “traditional”introductory physics courses [Hake (1998a,b; 2002; 2008b)].


I suspect that similar large effect sizes would be found for the superiority of online “interactive engagement” courses vs online “traditional” courses.


In my opinion it makes little sense to meta-analyze online vs face-to-face instruction without taking into account the relatively large effects on higher-order learning of “interactive-engagement” vs “traditional” instruction.

*************************************


To access the complete 20 kB post please click on http://bit.ly/egC8I3.


Richard Hake, Emeritus Professor of Physics, Indiana University

Honorary Member, Curmudgeon Lodge of Deventer, The

Netherlands

President, PEdants for Definitive Academic References which Recognize the

Invention of the Internet (PEDARRII)


rrhake@earthlink.net>

http://www.physics.indiana.edu/~hake

http://www.physics.indiana.edu/~sdi

http://HakesEdStuff.blogspot.com

http://iub.academia.edu/RichardHake


“Physics educators have led the way in developing and using objective tests to compare student learning gains in different types of courses, and chemists, biologists, and others are now developing similar instruments. These tests provide convincing evidence that students assimilate new knowledge more effectively incourses including active, inquiry-based, and collaborative learning, assisted by information technology, than in traditional courses.”

Wood & Gentile (2003)


REFERENCES [URL’s shortened by http://bit.ly/ and accessed on 05 Jan 2010.


Hake, R.R. 2002. “Lessons from the physics education reform effort,” Ecology and Society 5(2): 28; online at http://bit.ly/aL87VT.


Hake, R.R. 2008a. “Can Distance and Classroom Learning Be Increased?” IJ-SoTL 2(1): January; online at http://bit.ly/98dL0Y.


Hake, R.R. 2011. “Learning Outcomes: Face-to-Face vs Online #3,” online on the OPEN! AERA-L archives at http://bit.ly/egC8I3. Post of 5 Jan 2011 13:54:14 -0800 to AERA-L and Net-Gold. The abstract and link to the complete post are being transmitted to various discussion lists.


Wood, W.B. & J.M. Gentile. 2003. “Teaching in a research context,”
Science 302: 1510; 28 November; online to subscribers at http://bit.ly/9izfFz . A summary is online for all at http://bit.ly/9qGR6m.

Wednesday, June 23, 2010

What's the 'Average Normalized Gain' ? #2

Some blog followers might be interested in a post of the above title. The abstract reads:


*************************************************

ABSTRACT: In response to my post "What's the 'Average Normalized Gain' ?" EvalTalk's Robert Walker made three points to which I respond in turn:


a. "Why "normalized"? There seems to be no relationship to the normal curve." See my discussion of the physics/psychometric language gap in my *complete* post online at http://bit.ly/9GB2R2 .


b. "Why not use Cohen's effect size?" Primarily because, for pre/-post data for a class, it depends on the actual bare (unrenormalized) average <%Gain> = <%post> - <%pre>, whose magnitude, tends to be negatively correlated with the average pretest score <%pre>. On the other hand, in "Lessons from the Physics Education Reform Effort" [Hake (2002) I calculated a relatively large effect size of 2.43 for the average of the average normalized gains for 48 Interactive Engagement courses vs the average of the average normalized gains for 14 Traditional courses.


c. "I have found a meta analysis by Anja Zwingenberger of 36 original studies very useful. They all used effect size." But I doubt that Zwingenberger considered pre/post studies (please correct me if I'm wrong) because, despite their value, they are very rare, in part due to the pre/post paranoia that plagues some PEP's (Psychologists-Education specialists-Psychometricians).

*************************************************


To access the complete 17 kB post please click on http://bit.ly/ckOuJv .


Richard Hake, Emeritus Professor of Physics, Indiana University

Honorary Member, Curmudgeon Lodge of Deventer, The Netherlands

President, PEdants for Definitive Academic References which Recognize the Invention of the Internet (PEDARRII)


rrhake@earthlink.net>

http://www.physics.indiana.edu/~hake

http://www.physics.indiana.edu/~sdi

http://HakesEdStuff.blogspot.com

http://iub.academia.edu/RichardHake


REFERENCES [[URL's shortened by http://bit.ly/ .]


Hake, R.R. 2010. "What's the 'Average Normalized Gain' ? #2" online on the OPEN AERA-L archives at http://bit.ly/ckOuJv. Post of 23 Jun 2010 14:14:45 -0700 to AERA-L & Net-Gold. The abstract and link to the complete post are being transmitted to various discussion lists.

Tuesday, June 22, 2010

What's the "Average Normalized Gain" ?

Some blog followers might be interested in a post of the above title [Hake (2010)]. The abstract reads:


***************************************

ABSTRACT: In response to "Is Psychometrics Pathological Science?" [Hake (2010)], Michael Martin of the EDDRA2 list wrote (paraphrasing): "Hake's post refers to the 'normalized' scores from pretest to posttest. If the meaning is 'standardized' such as the SAT and other Z-score tests, I have a problem." Rather than "having a problem" and then incinerating a straw man, Martin might have considered simply clicking on http://bit.ly/9Fp6QS and then scanning the *entire* post and a few of its references so as to discover that the "average normalized gain" is *defined* as the *actual* gain (<%post> - <%pre>) divided by the *maximum possible gain* (100% - <%pre>), where the angle brackets indicate the class averages. Thus the average normalized gain HAS NOTHING TO DO WITH THE PSYCHOMETRIC "NORMALIZED" SCORES FROM PRETEST TO POSTTEST." That the average normalized gain is a valid and consistently reliable quantitative gauge of the effectiveness of an introductory mechanics course in promoting students' conceptual understanding of Newtonian mechanics is discussed in e.g., "Design-Based Research In Physics Education Research: A Review" [Hake (2008)].

***************************************


To access the complete 16 kB post please click on http://bit.ly/9GB2R2 .


Richard Hake, Emeritus Professor of Physics, Indiana University

Honorary Member, Curmudgeon Lodge of Deventer, The Netherlands

President, PEdants for Definitive Academic References which Recognize the Invention of the Internet (PEDARRII)


rrhake@earthlink.net

http://www.physics.indiana.edu/~hake

http://www.physics.indiana.edu/~sdi

http://HakesEdStuff.blogspot.com

http://iub.academia.edu/RichardHake


REFERENCES [URL's shortened by http://bit.ly/ ]

Hake, R.R. 2010. "What's the 'Average Normalized Gain' ? (was Is Psychometrics Pathological Science?)" online on the OPEN AERA-L archives at http://bit.ly/9GB2R2 . Post of 22 Jun 2010 11:23:38 -0700 to AERA-L, EDDRA2, & Net-Gold. The abstract and URL are being transmitted to various discussion lists.

Monday, December 15, 2008

The Case for Classroom Clickers - A Response to Bugeja

Some blog readers may be interested in a recent report with the above title. The abstract reads (slightly edited):

Michael Bugeja, in a Chronicle of Higher Education  article "Classroom Clickers and the Cost of Technology" states that clickers at Iowa State have been pushed by commercial interests in a way that subverts rather than enhances education, a complaint that deserves to be taken seriously by universities.  But Bugeja then goes on to imply that clickers: (a) were introduced into education by manufacturers, thus ignoring their academic pedigree, and (b) are nearly useless in education, ignoring the evidence for their effectiveness.  Perhaps the most dramatic such evidence has been provided by Eric Mazur, who increased the class average normalized learning gain g(ave) on a standardized test of conceptual understanding of Newtonian mechanics by a factor of about two when he switched from traditional passive-student lectures to clicker-assisted "Peer Instruction" (PI).  In addition, clickers: (1) have contributed to the spread of the PI approach by providing a relatively easy and attractive bridge from traditional lectures to greater interactivity, (2) allow instructors to  obtain real-time student feedback in histogram form, thus "making students' thinking visible and promoting critical listening, evaluation, and argumentation in the class,"  (3) archive student responses so as to improve questions and contribute to education research.  From a broader perspective, clickers may contribute to the spread of "interactive engagement" methods shown to be relatively effective in introductory physics instruction - i.e., methods designed to promote conceptual understanding through the active engagement of students in heads-on (always) and hands-on (usually) activities that yield immediate feedback though discussion with peers and/or instructors.

To access the complete 716 kB report please click on