Attrition distorts results in experimental biomedical research
The misreporting of animal numbers appears to be a common and regular occurrence in basic and preclinical research studies. Misreporting reduces the robustness of results and represents a threat to the validity and reproducibility of research studies. Prof. Dr. Ulrich Dirnagl, Head of the Department of Experimental Neurology at Charité – Universitätsmedizin Berlin, and his team arrived at this conclusion after analyzing hundreds of published preclinical stroke and cancer studies. Their findings, which have been published in the journal PLOS Biology*, suggest that a lack of transparent reporting and a careless approach to data analysis are far from the exception.
The loss of animals (attrition) can dramatically alter results in experiments with a small sample size; however, there remains a paucity of data on the effects of random and biased attrition. In an effort to remedy the situation, Prof. Dirnagl and his team analyzed a large number of previously-published studies. Their results were sobering. The majority of published studies provided insufficient information on sample sizes, i.e. the number of animals required to meet research objectives. Many of the studies also bore evidence of animals 'disappearing' during the course of the research, a practice that can result in a major distortion of research results.
For Prof. Dirnagl, the study's lead investigator, this prompted a number of questions: “Where did all the animals go? Were there reasonable grounds for their exclusion? Did the researchers remove them for a specific reason? Or did severe side effects render them too ill to be included in the analysis?” He goes on to add that “the information provided in the majority of the studies analyzed is so unclear, it is impossible to answer these questions.” The research team from Charité used sophisticated software to simulate the effects of animals being removed from studies. Attrition was shown to have a significant impact on the robustness of research findings, with a decrease in robustness linked to an increase in the number of animals lost from the study. If random and unintentional, attrition can result in a minor distortion of research results. However, if it is done in a targeted manner, such as when animals are specifically selected because their outcomes do not support the research hypothesis, attrition distorts results and produces false positives.
It is possible for the odd animal to be lost during the course of a biomedical research study, or for an animal to require exclusion from an experiment. While this is a normal occurrence, any such losses have to be properly documented if results are to remain valid. “The deciding factor here is that the number of animals used, as well as any potential losses over the course of the study, have to be documented. This is the only way to ensure that the research data can be used to predict whether a particular treatment might also be effective in humans,” explains the study's first author, Constance Holman. The researchers point out that a large proportion of previously-published studies are based on either an overestimation of effect sizes or completely distorted research results. In the majority of cases, this is not due to research fraud but a result of selection bias, i.e. the unintentional and subconscious bias displayed by researchers who want to see their hypothesis confirmed. Lack of transparency in reporting only adds to the level of difficulty in reproducing research results.
If this analysis of preclinical stroke and cancer studies is representative of preclinical research as a whole, a significant number of biomedical research studies owe their conclusions to the loss or exclusion of specific animals. “For the sake of good research practice and the credibility of research results, it is essential that details of experiments be documented and verifiable.”
*Holman C, Piper SK, Grittner U, Diamantaras AA, Kimmelman J, Siegerink B, et al. Where Have All the Rodents Gone? The Effects of Attrition in Experimental Research on Cancer and Stroke. PLoS Biol 14(1): e1002331. Jan. 2016. doi:10.1371/journal.pbio.1002331
Links
Charité Experimental Neurology
NeuroCure
Contact
Prof. Dr. Ulrich Dirnagl
Head of the Department of Experimental Neurology
Klinik und Poliklinik für Neurologie
Charité – Universitätsmedizin Berlin
Tel: +49 (30) 450 560 122
Back to Overview