Title

Modeling human coding of free response data

Authors

Authors

S. Ghiasinejad;R. M. Golden

Comments

Authors: contact us about adding a copy of your work at STARS@ucf.edu

Abbreviated Journal Title

Comput. Hum. Behav.

Keywords

Hidden Markov model; Propositional coding; Protocol data analysis; Computational model; LATENT SEMANTIC ANALYSIS; PROTOCOL ANALYSIS; STUDENT ESSAYS; SPEECH; MEMORY; TEXT; RECOGNITION; COMPREHENSION; DISCOURSE; VALIDITY; Psychology, Multidisciplinary; Psychology, Experimental

Abstract

Summarization, recall, think-aloud, and question-answering protocol data are examples of free response verbal reports used for the purposes of revealing the structure and content of internal mental representations and processes within the field of discourse processes. Typically, two experienced coders independently semantically annotate a portion of Collected protocol data and measures of agreement are used to determine the reliability of the coding. This methodology, however, does not provide an effective method for communicating in an unambiguous manner complex coding procedures to other researchers. To address this problem, an automated methodology called AUTOCODER for coding free response data is evaluated. The AUTOCODER system works by actively interacting with an experienced human coder who semantically annotates key words with "word-concepts" and sequences of word-concepts with "propositions". After training AUTOCODER on a set of 70 segmented and semantically annotated free response verbal reports originally generated by second grade and fifth grade students, AUTOCODER exhibited a good proposition agreement rate of 91% and a kappa agreement score of 65% with respect to an experienced human coder on an additional set of 24 unsegmented free response verbal reports. Limitations and general implications of these findings are also discussed. (C) 2013 Elsevier Ltd. All rights reserved.

Journal Title

Computers in Human Behavior

Volume

29

Issue/Number

6

Publication Date

1-1-2013

Document Type

Article

Language

English

First Page

2394

Last Page

2403

WOS Identifier

WOS:000325234600031

ISSN

0747-5632

Share

COinS