article · The Journal of Finance
While standard errors capture uncertainty arising from sampling in a data-generating process, variation across researchers generates additional uncertainty termed nonstandard errors. When 164 independent teams analysed the same dataset to test identical hypotheses, their varying analytical choices produced sizable nonstandard errors. This demonstrates that scientific conclusions can vary widely even when evaluating identical data. The magnitude of these errors is notably smaller for research that is more reproducible or attains higher evaluation ratings. Furthermore, introducing peer-review stages helps to reduce nonstandard errors across teams. Despite the substantial impact of this researcher variation on scientific results, participating researchers routinely underestimate the scale of nonstandard errors present in their work.
Scientific results often appear definitive, yet different teams examining identical data can reach varying conclusions due to nonstandard errors. Recognising that this uncertainty exists, and that researchers routinely underestimate it, highlights the critical necessity of peer-review stages and reproducibility standards to ensure that evidence-based conclusions are genuinely dependable.
The abstract does not indicate an application pathway or commercialisation potential for this research.
AI-generated from the published abstract. Always read the original work before citing.
ABSTRACT In statistics, samples are drawn from a population in a data‐generating process (DGP). Standard errors measure the uncertainty in estimates of population parameters. In science, evidence is generated to test hypotheses in an evidence‐generating process (EGP). We claim that EGP variation across researchers adds uncertainty—nonstandard errors (NSEs). We study NSEs by letting 164 teams test the same hypotheses on the same data. NSEs turn out to be sizable, but smaller for more reproducible or higher rated research. Adding peer‐review stages reduces NSEs. We further find that this type of uncertainty is underestimated by participants.
This page summarises published work. The authoritative version sits with the publisher.
DOI: 10.1111/jofi.13337
Is something wrong with this record? Report it or request removal.
Discussion
Have you built on this work, tried to replicate it, or seen it applied in practice? Share what you know. Verified researchers and MARATTO™ domain experts can open a discussion, and any member can reply. Contributions are reviewed before they appear.
No discussion yet. Open the first thread.
New to MARATTO™? Create a free account.
