Title

Modeling Human Coding Of Free Response Data

Keywords

Computational model; Hidden Markov model; Propositional coding; Protocol data analysis

Abstract

Summarization, recall, think-aloud, and question-answering protocol data are examples of free response verbal reports used for the purposes of revealing the structure and content of internal mental representations and processes within the field of discourse processes. Typically, two experienced coders independently semantically annotate a portion of collected protocol data and measures of agreement are used to determine the reliability of the coding. This methodology, however, does not provide an effective method for communicating in an unambiguous manner complex coding procedures to other researchers. To address this problem, an automated methodology called AUTOCODER for coding free response data is evaluated. The AUTOCODER system works by actively interacting with an experienced human coder who semantically annotates key words with "word-concepts" and sequences of word-concepts with "propositions". After training AUTOCODER on a set of 70 segmented and semantically annotated free response verbal reports originally generated by second grade and fifth grade students, AUTOCODER exhibited a good proposition agreement rate of 91% and a kappa agreement score of 65% with respect to an experienced human coder on an additional set of 24 unsegmented free response verbal reports. Limitations and general implications of these findings are also discussed. © 2013 Elsevier Ltd. All rights reserved.

Publication Date

6-24-2013

Publication Title

Computers in Human Behavior

Volume

29

Issue

6

Number of Pages

2394-2403

Document Type

Article

Personal Identifier

scopus

DOI Link

https://doi.org/10.1016/j.chb.2013.05.021

Socpus ID

84879069394 (Scopus)

Source API URL

https://api.elsevier.com/content/abstract/scopus_id/84879069394

This document is currently not available here.

Share

COinS