Ill-structured measurement designs in organizational research: Implications for estimating interrater reliability

    Authors

    D. J. Putka; H. Le; R. A. McCloy;T. Diaz

    Comments

    Authors: contact us about adding a copy of your work at STARS@ucf.edu

    Abbreviated Journal Title

    J. Appl. Psychol.

    Keywords

    measurement design; ratings; reliability; JOB-PERFORMANCE RATINGS; PSYCHOLOGICAL-RESEARCH; SITUATIONAL INTERVIEW; SELECTION INTERVIEWS; CORRELATED ERRORS; VALIDITY; COEFFICIENTS; LEVEL; RATER; CONSEQUENCES; Psychology, Applied; Management

    Abstract

    Organizational research and practice involving ratings are rife with what the authors term ill-structured measurement designs (ISMDs)-designs in which raters and ratees are neither fully crossed nor nested. This article explores the implications of ISMDs for estimating interrater reliability. The authors first provide a mock example that illustrates potential problems that ISMDs create for common reliability estimators (e.g., Pearson correlations, intraclass correlations). Next, the authors propose an alternative reliability estimator-G(q,k)-that resolves problems with traditional estimators and is equally appropriate for crossed, nested, and ill-structured designs. By using Monte Carlo simulation, the authors evaluate the accuracy of traditional reliability estimators compared with that of G(q,k) for ratings arising from ISMDs. Regardless of condition, G(q,k) yielded estimates as precise or more precise than those of traditional estimators. The advantage of G(q,k) over the traditional estimators became more pronounced with increases in the (a) overlap between the sets of raters that rated each ratee and (b) ratio of rater main effect variance to true score variance. Discussion focuses on implications of this work for organizational research and practice.

    Journal Title

    Journal of Applied Psychology

    Volume

    93

    Issue/Number

    5

    Publication Date

    1-1-2008

    Document Type

    Article

    Language

    English

    First Page

    959

    Last Page

    981

    WOS Identifier

    WOS:000259416300002

    ISSN

    0021-9010

    Share

    COinS