Crowdsourcing and personality measurement equivalence: A warning about countries whose primary language is not English

    Authors

    J. Feitosa; D. L. Joseph;D. A. Newman

    Comments

    Authors: contact us about adding a copy of your work at STARS@ucf.edu

    Abbreviated Journal Title

    Pers. Individ. Differ.

    Keywords

    Crowdsourcing; Measurement; Data collection techniques; Survey methods; Invariance testing; ITEM RESPONSE THEORY; MEASUREMENT INVARIANCE; MECHANICAL TURK; FIT; INDEXES; CULTURES; METHODOLOGIES; TESTS; Psychology, Social

    Abstract

    In the search to find cheaper, faster approaches for data collection, crowdsourcing methods (i.e., online labor portals that allow independent workers to complete surveys for compensation) have risen in popularity as a tool for personality researchers, despite a lack of evidence regarding the equivalence of crowdsourcing with traditional data collection methods. The purpose of this study was to evaluate crowdsourcing as a data collection tool by examining the measurement equivalence of crowdsourced data (i.e., from Amazon.com's MTurk) with more traditional samples (i.e., an undergraduate sample and a sample of organizational employees). Our results (using a popular measure of Big Five personality) provided evidence of measurement equivalence across all three samples, with one important exception: crowdsourced data (from MTurk) only exhibited measurement invariance with traditional data collection methods when responses were restricted to participants from native-English speaking countries. Although MTurk appears to be an easy, cost-effective data collection tool, our results suggest that MTurk data are similar to traditionally-collected data only when the MTurk sample is restricted to IP addresses from English-speaking countries. (c) 2014 Elsevier Ltd. All rights reserved.

    Journal Title

    Personality and Individual Differences

    Volume

    75

    Publication Date

    1-1-2015

    Document Type

    Article

    Language

    English

    First Page

    47

    Last Page

    52

    WOS Identifier

    WOS:000348270900009

    ISSN

    0191-8869

    Share

    COinS