Modeling human coding of free response data

    Authors

    S. Ghiasinejad;R. M. Golden

    Comments

    Authors: contact us about adding a copy of your work at STARS@ucf.edu

    Abbreviated Journal Title

    Comput. Hum. Behav.

    Keywords

    Hidden Markov model; Propositional coding; Protocol data analysis; Computational model; LATENT SEMANTIC ANALYSIS; PROTOCOL ANALYSIS; STUDENT ESSAYS; SPEECH; MEMORY; TEXT; RECOGNITION; COMPREHENSION; DISCOURSE; VALIDITY; Psychology, Multidisciplinary; Psychology, Experimental

    Abstract

    Summarization, recall, think-aloud, and question-answering protocol data are examples of free response verbal reports used for the purposes of revealing the structure and content of internal mental representations and processes within the field of discourse processes. Typically, two experienced coders independently semantically annotate a portion of Collected protocol data and measures of agreement are used to determine the reliability of the coding. This methodology, however, does not provide an effective method for communicating in an unambiguous manner complex coding procedures to other researchers. To address this problem, an automated methodology called AUTOCODER for coding free response data is evaluated. The AUTOCODER system works by actively interacting with an experienced human coder who semantically annotates key words with "word-concepts" and sequences of word-concepts with "propositions". After training AUTOCODER on a set of 70 segmented and semantically annotated free response verbal reports originally generated by second grade and fifth grade students, AUTOCODER exhibited a good proposition agreement rate of 91% and a kappa agreement score of 65% with respect to an experienced human coder on an additional set of 24 unsegmented free response verbal reports. Limitations and general implications of these findings are also discussed. (C) 2013 Elsevier Ltd. All rights reserved.

    Journal Title

    Computers in Human Behavior

    Volume

    29

    Issue/Number

    6

    Publication Date

    1-1-2013

    Document Type

    Article

    Language

    English

    First Page

    2394

    Last Page

    2403

    WOS Identifier

    WOS:000325234600031

    ISSN

    0747-5632

    Share

    COinS