A Cautionary Note on Modeling Multitrait-Multirater Data Arising From Ill-Structured Measurement Designs

    Authors

    D. J. Putka; C. E. Lance; H. Le;R. A. McCloy

    Comments

    Authors: contact us about adding a copy of your work at [email protected]

    Abbreviated Journal Title

    Organ. Res. Methods

    Keywords

    confirmatory factor analysis; job performance ratings; measurement; design; MULTISOURCE PERFORMANCE RATINGS; COVARIANCE STRUCTURE-ANALYSIS; CROSS-CLASSIFIED DATA; JOB-PERFORMANCE; MEASUREMENT EQUIVALENCE; MULTIMETHOD MATRIX; CONSTRUCT-VALIDITY; RATER; METAANALYSIS; SUBORDINATE; Psychology, Applied; Management

    Abstract

    The authors illustrate a problem with confirmatory factor analysis (CFA)-based strategies to model disaggregated multitrait-multirater (MTMR) data-the potential to find markedly different results with the same sample of ratees simply as a result of how one selects and identifies raters within the data set one has gathered for analysis. Using performance ratings gathered as part of a large criterion-related validation study, the authors show how such differences manifest themselves in several ways including variation in (a) covariance matrices that serve as input for the modeling effort, (b) model convergence, (c) admissibility of solutions, (d) overall model fit, (e) model parameter estimates, and (f) model selection. Implications of this study for past research and recommendations for future CFA-based MTMR modeling efforts are discussed.

    Journal Title

    Organizational Research Methods

    Volume

    14

    Issue/Number

    3

    Publication Date

    1-1-2011

    Document Type

    Article

    Language

    English

    First Page

    503

    Last Page

    529

    WOS Identifier

    WOS:000291030900005

    ISSN

    1094-4281

    Share

    COinS
     

    Accessibility Statement

    This item was created or digitized prior to April 24, 2027, or is a reproduction of legacy media created before that date. It is preserved in its original, unmodified state specifically for research, reference, or historical recordkeeping. In accordance with the ADA Title II Final Rule, the University Libraries provides accessible versions of archival materials upon request. To request an accommodation for this item, please submit an accessibility request form.