The reliability of instructor evaluations of crew performance: Good news and not so good news

    Authors

    M. T. Brannick; C. Prince;E. Salas

    Comments

    Authors: contact us about adding a copy of your work at STARS@ucf.edu

    Abbreviated Journal Title

    Int. J. Aviat. Psychol.

    Keywords

    RESOURCE-MANAGEMENT; TEAMWORK SKILLS; RATING ACCURACY; AVIATION; APPRAISAL; TIME; Psychology, Applied

    Abstract

    dTwo instructors rated the crew performance of helicopter crews (N = 45) who flew a simulated mission in a full-motion simulator. The instructors were given dimension training and behavior-observation training before the ratings. Before making their ratings, instructors completed an observation form as they watched videotapes of the crews and then completed a form that helped link behaviors to dimensions. For their ratings, instructors recorded or evaluated crew behaviors using 3 types of items: (a) specific crew behaviors in response to scenario events (e.g., whether the crew kept out of icing conditions), (b) evaluations of crew responses to scenario events (e.g., overall handling of the icing problem), and (c) crew resource management (CRM) dimensions for the entire scenario (e.g., evaluations of decision making). Results showed (a) both interjudge agreement and internal consistency were high for evaluations of crew responses to scenario events, (b) interjudge agreement was low but internal consistency was high on CRM items and scales, and (c) interjudge agreement was high but internal consistency was low for specific observable behaviors. The results for the evaluations of crew responses to scenario events were very encouraging and showed reliability over time and over crews. Suggestions for improving the reliability for the other 2 item types are provided.

    Journal Title

    International Journal of Aviation Psychology

    Volume

    12

    Issue/Number

    3

    Publication Date

    1-1-2002

    Document Type

    Article

    Language

    English

    First Page

    241

    Last Page

    261

    WOS Identifier

    WOS:000179116700004

    ISSN

    1050-8414

    Share

    COinS