Using Bayesian Correspondence Criteria To Compare Results From A Randomized Experiment And A Quasi-Experiment Allowing Self-Selection

Keywords

Bayesian; nonrandomized experiment; propensity score; within-study comparison

Abstract

Background: Randomized experiments yield unbiased estimates of treatment effect, but such experiments are not always feasible. So researchers have searched for conditions under which randomized and nonrandomized experiments can yield the same answer. This search requires well-justified and informative correspondence criteria, that is, criteria by which we can judge if the results from an appropriately adjusted nonrandomized experiment well-approximate results from randomized experiments. Past criteria have relied exclusively on frequentist statistics, using criteria such as whether results agree in sign or statistical significance or whether results differ significantly from each other. Objectives: In this article, we show how Bayesian correspondence criteria offer more varied, nuanced, and informative answers than those from frequentist approaches. Research design: We describe the conceptual bases of Bayesian correspondence criteria and then illustrate many possibilities using an example that compares results from a randomized experiment to results from a parallel nonequivalent comparison group experiment in which participants could choose their condition. Results: Results suggest that, in this case, the quasi-experiment reasonably approximated the randomized experiment. Conclusions: We conclude with a discussion of the advantages (computation of relevant quantities, interpretation, and estimation of quantities of interest for policy), disadvantages, and limitations of Bayesian correspondence criteria. We believe that in most circumstances, the advantages of Bayesian approaches far outweigh the disadvantages.

Publication Date

4-1-2018

Publication Title

Evaluation Review

Volume

42

Issue

2

Number of Pages

248-280

Document Type

Article

Personal Identifier

scopus

DOI Link

https://doi.org/10.1177/0193841X18789532

Socpus ID

85052402906 (Scopus)

Source API URL

https://api.elsevier.com/content/abstract/scopus_id/85052402906

This document is currently not available here.

Share

COinS