The Cost of Agree-Disagree: Satisficing and Sacrificing ReliabilityReport as inadecuate




The Cost of Agree-Disagree: Satisficing and Sacrificing Reliability - Download this document for free, or read online. Document in PDF available to download.



Society for Research on Educational Effectiveness

Of all the approaches to collecting data in the social sciences, the administration of questionnaires to respondents is among the most prevalent. Despite their popularity, there is broad consensus among survey design experts that using these items introduces excessive error into respondents' ratings. The authors attempt to answer the following question: what are the specific impacts on survey data quality if one uses statements and A-D response anchors as opposed to questions and construct-specific (C-S) response anchors? Data were collected from a nationally representative SurveyMonkey panel of parents/primary caregivers for school-aged children. In return for their participation, participants received the chance to win money for a charity of their choosing. The authors found strong evidence that web-pages with only AD items elicited more straight-lining than corresponding pages with only C-S items. To their surprise, it was found that A-D scale scores were frequently less reliable than C-S scale scores even before removing straight-liners. While the authors consider these findings to be conclusive, they believe that more work remains to be done, both theoretical and practical. Tables and scales from the two forms of the survey are appended.

Descriptors: Questionnaires, Surveys, Likert Scales, Test Items, Test Format, Online Surveys, Parents, Child Caregivers, Randomized Controlled Trials, Test Reliability, Statistical Analysis, Correlation

Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; Fax: 202-640-4401; e-mail: inquiries[at]sree.org; Web site: http://www.sree.org





Author: McIntyre, Joe; Gehlbach, Hunter

Source: https://eric.ed.gov/?q=a&ft=on&ff1=dtySince_1992&pg=1507&id=ED562897



DOWNLOAD PDF




Related documents