Wednesday, April 30, 2014

The Continued (and Failed) Quest for Automated Writing Assessment


It continues to surprise me how many are willing to sacrifice effectiveness for efficiency in education. Pearson Education, a well-known textbook publisher and notorious SAT scoring administrator turned education "innovator", has recently teamed up with schools across the country to gather data for their new automated essay scorer as part of the new Common Core Curriculum. "The company is using these student essays to train its robo-grader to replace one of the two human readers grading the essay, although there are no published data on their effectiveness in correcting human readers," says Les Perelman, the now-retired director of Writing Across the Curriculum at MIT. 

And this isn't the first time folks have tried to lay claim to the quality of robo-graders. Mark Shermis and Ben Hamner came fairly close to convincing a lot of people of the validity and significance of their findings regarding the near-human accuracy of their robo-grader – until Perelman blew them out of the water with his critique of their methods and data analysis (Rivard). In his critique, Perelman states, "The study’s methodology used one variable for comparing human readers and a different variable for comparing machine scores, this difference artificially privileging the machines in half the datasets." In his Boston Globe article today, Perelman was quick to point out that these "Robo Graders" are pretty good at counting words and determining vocabulary level, but not at understanding the complexities of thinking and understanding present in an essay. I find it odd that school administrators are willing to pay Pearson for a system that does something any third-grade student can do in Microsoft Word without paying a fee – check word count and check readability by grade level.

Many in our society set out on a quest for automation before fully understanding the phenomena and processes to be automated. Is this because we are waxing nostalgic for the Industrial Revolution? Is it due to of our collective American goal to be the leader in innovation (which to some equates with automation and technologization of just about everything)? Or is it, perhaps more realistically speaking, the human need for fame and fortune for inventing the next great thing? 

"Teaching to the test" is already an issue in our country. Let's hope writing instruction doesn't become an issue of "teaching to the machine" (that slept through fifth-grade composition).

Works Cited

Perelman, Les. "Critique (Ver. 3.4) of Mark D. Shermis & Ben Hammer, “Contrasting State-of-the-Art Automated Scoring of Essays: Analysis”." Ver. 3.4, Mar. 13, 2013. Web. 30 Apr. 2014.

Perelman, Les. "Standardized-test Robo-graders Flunk." BostonGlobe.com. Boston Globe Media LLC, 30 Apr. 2014. Web. 30 Apr. 2014.

Rivard, Ry. "Professors at Odds on Machine-graded Essays." Inside Higher Ed, 15 Mar. 2013. Web. 30 Apr. 2014.

No comments:

Post a Comment