Sample-Size Requirements for Automated Essay Scoring. Research Report. ETS RR-08-32Report as inadecuate

Sample-Size Requirements for Automated Essay Scoring. Research Report. ETS RR-08-32 - Download this document for free, or read online. Document in PDF available to download.

ETS Research Report Series, Jun 2008

Sample-size requirements were considered for automated essay scoring in cases in which the automated essay score estimates the score provided by a human rater. Analysis considered both cases in which an essay prompt is examined in isolation and those in which a family of essay prompts is studied. In typical cases in which content analysis is not employed and in which the only object is to score individual essays to provide feedback to the examinee, it appears that several hundred essays are sufficient. For application of one model to a family of essays, fewer than 100 essays per prompt may often be adequate. The cumulative logit model was explored as a possible replacement of the linear regression model usually employed in automated essay scoring; the cumulative logit model performed somewhat better than did the linear regression model.

Descriptors: Sample Size, Scoring, Essays, Automation, Test Scoring Machines, Regression (Statistics), Prompting

Educational Testing Service. Rosedale Road, MS19-R Princeton, NJ 08541. Tel: 609-921-9000; Fax: 609-734-5410; e-mail: RDweb[at]; Web site:

Author: Haberman, Shelby J.; Sinharay, Sandip


Related documents