Towards more credible conceptual replications under heteroscedasticity and unbalanced designs

Pedro Mateu, Brooks Applegate, Chris L. Coryn

Research output: Contribution to journalArticle in a journalpeer-review

Abstract

Theory cannot be fully validated unless original results have been replicated, resulting in conclusion consistency. Replications are the strongest source of evidence to verify research findings and knowledge claims. In the social sciences, replication studies often fail and thus a continuing need for replication studies to confirm tentative facts, expand knowledge to gain new understanding, and verify hypotheses. Failure to replicate in the social and behavioral sciences sometimes arises due to dissimilarity between hypotheses formulated in original and replication studies. Alternatively, failure to replicate also occurs when the same hypothesis is tested; but done so in the absence of knowledge from previous investigations, as when original study effect sizes are not considered in replication studies. To increase replicability of research findings, this paper demonstrates that the application of two one-sided tests to evaluate a replication question provides a superior means for conducting replications, assuming all other methodological procedures remained as similar as possible. Furthermore, this paper sought to explore the impact of heteroscedasticity and unbalanced designs in replication studies in four paired conditions of variance and sample size. Two Monte Carlo simulations, each with two stages, were conducted to investigate conclusion consistency among different replication procedures to determine the repeatability of an observed effect. Overall, the proposed approach yielded a higher proportion of successful replications than the conventional approach (testing the original null hypothesis of no effect). Thus, findings can be confirmed by replications and in the absence of confirmation, there cannot be a final statement about any theory.

Original languageEnglish
Pages (from-to)723-751
Number of pages29
JournalQuality and Quantity
Volume58
Issue number1
Early online date18 Apr 2023
DOIs
StatePublished - Feb 2024

Bibliographical note

Publisher Copyright:
© 2023, The Author(s), under exclusive licence to Springer Nature B.V.

Keywords

  • Conceptual replications
  • Heteroscedasticity
  • Two-one sided tests
  • Unbalanced designs

Fingerprint

Dive into the research topics of 'Towards more credible conceptual replications under heteroscedasticity and unbalanced designs'. Together they form a unique fingerprint.

Cite this