Title
Modeling Human Coding Of Free Response Data
Keywords
Computational model; Hidden Markov model; Propositional coding; Protocol data analysis
Abstract
Summarization, recall, think-aloud, and question-answering protocol data are examples of free response verbal reports used for the purposes of revealing the structure and content of internal mental representations and processes within the field of discourse processes. Typically, two experienced coders independently semantically annotate a portion of collected protocol data and measures of agreement are used to determine the reliability of the coding. This methodology, however, does not provide an effective method for communicating in an unambiguous manner complex coding procedures to other researchers. To address this problem, an automated methodology called AUTOCODER for coding free response data is evaluated. The AUTOCODER system works by actively interacting with an experienced human coder who semantically annotates key words with "word-concepts" and sequences of word-concepts with "propositions". After training AUTOCODER on a set of 70 segmented and semantically annotated free response verbal reports originally generated by second grade and fifth grade students, AUTOCODER exhibited a good proposition agreement rate of 91% and a kappa agreement score of 65% with respect to an experienced human coder on an additional set of 24 unsegmented free response verbal reports. Limitations and general implications of these findings are also discussed. © 2013 Elsevier Ltd. All rights reserved.
Publication Date
6-24-2013
Publication Title
Computers in Human Behavior
Volume
29
Issue
6
Number of Pages
2394-2403
Document Type
Article
Personal Identifier
scopus
DOI Link
https://doi.org/10.1016/j.chb.2013.05.021
Copyright Status
Unknown
Socpus ID
84879069394 (Scopus)
Source API URL
https://api.elsevier.com/content/abstract/scopus_id/84879069394
STARS Citation
Ghiasinejad, Shahram and Golden, Richard M., "Modeling Human Coding Of Free Response Data" (2013). Scopus Export 2010-2014. 7107.
https://stars.library.ucf.edu/scopus2010/7107