Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Correspondence
  • Published:

Moving beyond P values: data analysis with estimation graphics

Nature Methods volume 16, pages 565–566 (2019)Cite this article

Subjects

Access through your institution
Buy or subscribe

To the Editor — For at least 77 years, the limitations of null-hypothesis significance testing (NHST) have been discussed, without agreement on a suitable alternative1,2. Estimation methods that estimate effect sizes and their uncertainty have great potential to shift the current data-analysis culture away from dichotomous thinking and toward quantitative reasoning3,4. Although NHST limits the analyst to the ill-conceived question of “Does it?”5, estimation instead draws the analyst’s attention to the question of “How much?”—the very topic that defines quantitative research. Here we describe the estimation graphic, a plot that displays an experimental dataset’s complete statistical information. We also introduce software that makes high-quality estimation graphics available to all.

Although visualizing the null distribution is an improvement, this picture nevertheless illustrates NHST’s flawed logic: to ‘prove’ that the null hypothesis is false, the analyst must invoke the existence of something (the tail segment) that the hypothesis itself predicts1. Even the premise of NHST is unrealistic: any intervention to a system will produce some (at least infinitesimal) effect; thus a hypothesized effect size of precisely zero is inevitably false3.

This is a preview of subscription content, access via your institution

Access options

Access through your institution

Buy this article

  • Purchase on SpringerLink
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: The evolution of two-group data graphics.

Data availability

The Matlab, Python and R packages are available on GitHub (https://github.com/ACCLAB/DABEST-python) and are licensed under the BSD 3-Clause Clear License.

References

  1. Berkson, J. J. Am. Stat. Assoc. 37, 325–335 (1942).

    Article  Google Scholar 

  2. Benjamin, D. J. et al. Nat. Hum. Behav. 2, 6–10 (2017).

    Article  Google Scholar 

  3. Cohen, J. Am. Psychol. 49, 997–1003 (1994).

    Article  Google Scholar 

  4. Cumming, G. & Calin-Jageman, R. Introduction to the New Statistics: Estimation, Open Science, and Beyond (Routledge, 2016).

  5. McCloskey, D. The Secret Sins of Economics (Prickly Paradigm Press, 2002).

  6. Wilkinson, L. Am. Stat. 53, 276–281 (1999).

    Google Scholar 

  7. Gardner, M. J. & Altman, D. G. Br. Med. J. 292, 746–750 (1986).

    Article  CAS  Google Scholar 

  8. Altman, D., Machin, D., Bryant, T. & Gardner, S. Statistics with Confidence: Confidence Interval and Statistical Guidelines (BMJ Books, 2000).

  9. Efron, B. & Tibshirani, R. J. An Introduction to the Bootstrap (CRC Press, 1994).

Download references

Acknowledgements

We thank M. Rahman for help and advice, and H. Nguyen for developing the web app front end.

Author information

Authors and Affiliations

  1. Institute for Molecular and Cell Biology, A*STAR, Singapore, Singapore

    Joses Ho, Tayfun Tumkaya, Sameer Aryal, Hyungwon Choi & Adam Claridge-Chang

  2. Department of Physiology, National University of Singapore, Singapore, Singapore

    Tayfun Tumkaya & Adam Claridge-Chang

  3. Center for Neural Science, New York University, New York, NY, USA

    Sameer Aryal

  4. Department of Medicine, Yong Loo Lin School of Medicine, National University of Singapore, Singapore, Singapore

    Hyungwon Choi

  5. Program in Neuroscience and Behavioral Disorders, Duke-NUS Medical School, Singapore, Singapore

    Adam Claridge-Chang

Authors
  1. Joses Ho
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Tayfun Tumkaya
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Sameer Aryal
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Hyungwon Choi
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Adam Claridge-Chang
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Adam Claridge-Chang.

Ethics declarations

Competing interests

The authors declare no competing interests.

Supplementary information

Supplementary Information

Supplementary Notes 1–4 and Supplementary Figs. 1 and 2.

Reporting Summary

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ho, J., Tumkaya, T., Aryal, S. et al. Moving beyond P values: data analysis with estimation graphics. Nat Methods 16, 565–566 (2019). https://doi.org/10.1038/s41592-019-0470-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41592-019-0470-3

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing