‘ECR Asks’ is a series of Q&A sessions where I speak with experienced researchers to explore their journey in, and perspectives on, open science and transparent research. With the goal of supporting early career researchers like myself, this series aims to answer big questions and share practical insights on navigating the ever evolving landscape of open science and academia.

I would like to thank Joel for sharing his time and insights on such a large, complex, topic.

Q&A

Each of our contributions to the understanding of the natural world — to paraphrase Newton — stands on the shoulders of giants by building upon the research that came before. But what if we couldn’t trust this foundation? What if the findings we take as fact were, actually, only a moment in time, or artifacts of sampling bias, or a cherry-picked p-value with inflated effect size? When half of research in the social sciences and more than half of cancer biology lab research is not able to be replicated, there is little reason to assume any other field would be immune. Can we really move forward in ecology and evolutionary biology without first asking ourselves the question — how bad is our replication crisis?

For this edition of ‘ECR Asks’, I spoke with Dr Joel Pick, postdoctoral researcher at the University of Edinburgh, UK. His primary research focuses on understanding how ecological and evolutionary processes interact to affect short term population dynamics but he has also written and spoken extensively across challenges in open science. As chair of the SORTEE Advocacy committee, he has led and collaborated on SORTEE publications, as well as leading research into the replication crisis himself. In this interview we discuss two of his recent releases, Replication studies: a win-win for early-career training and behavioral ecology as well as preprint What do ecology and evolutionary biology journal websites communicate about their policies and preferences regarding replication studies?

Joel joined me on June 8th, 2026, to explain the importance of replication and how ECRs can play their part in addressing the replication crisis.

This interview has been edited for length and clarity.

Q&A

The Replication Crisis

ECR: For those of us new to research, would you be able to describe the replication crisis? What is the scale of this challenge in ecology and evolutionary biology?

JP: In short, much of the research currently in the literature doesn’t produce comparable results when we repeat the experiment. In ecology and evolution, it’s hard to know the scale of the issue because there are so few replication studies conducted in the first place with which to compare. Most of the research into the replication crisis comes from fields like psychology, cancer biology, and social sciences, where they’ve done large-scale replication projects, replicating hundreds of studies. In these fields, the take-away after replication is that nearly all of the effects reduce in size, and a lot go from being significant to non-significant. We have all the signs that would indicate there is a similar replication crisis within ecology.

In ecology and evolutionary biology, due to sampling variation, the effect sizes we can estimate vary a lot just by chance, and we’re often in a situation where our studies have quite a low statistical power. You can imagine a situation where you’re looking at some phenomenon where there is no true effect. When you’re sampling this phenomenon, particularly in a study with a low sample size, sometimes you’ll find a positive effect, sometimes you’ll find a negative one, and a certain proportion of each will happen to be significant — just through chance and sampling variation alone. In order to reach statistical significance, these will naturally be the results that are more extreme though… Journals are biased towards these results that are statistically significant with big effects, however. And, thus, although the experimental samples would actually be suggesting effects all over the place, centering on zero, journals end up publishing mostly the results from the extreme ends of the scale (that is, those that are significantly positive and significantly negative). Thus, we end up with inflated effect sizes and a distorted picture of the phenomenon in the literature.

This bias is then exacerbated because people are aware that journals only want to publish significant findings and they start biasing themselves towards getting those significant results. Questionable research practices like p-hacking (where you’re cherry-picking statistically significant results) or HARK-ing (hypothesizing after result known) start to increase, even subconsciously.

ECR: In your preprint What do ecology and evolutionary biology journal websites communicate about their policies and preferences regarding replication studies? you and your co-authors find that of the 226 journals in ecology and evolutionary biology, only 4 encouraged replication research, with the remainder not mentioning or implicitly discouraging replication. Is this mainly because replications receive less interest and citations than novel research? Is the replication crisis therefore mainly an extension of publication bias?

JP: These two ideas are connected but there is more to it. Published replication studies are a way of diagnosing the replication crisis, but the underlying crisis can be there with or without them. We could have no replication studies but no crisis if each of those studies did happen to replicate upon testing. Similarly, we could have an abundance of replication papers but, if they didn’t find comparable results, we would still have the replication crisis. However, in that case, we probably would have discovered the crisis earlier and done more to address it… Currently, without replication studies to serve as clear diagnostics, we have to estimate and argue that the crisis even exists in the first place.

As to whether replications are of less interest than novel research, I don’t think so. It’s a common conception that replication studies get fewer citations but, as far as I’m aware, there isn’t any evidence to support this. Why would they be less cited? If the replication shows support for the original research, it would be cited to show support, if it didn’t show support, it would be cited as opposing evidence — either way, it’s of interest, and to not cite it would be ignoring important science.

I don’t think it’s necessarily intrinsic that we see replication as low status. Instead, I think there has been a vicious cycle. Journals think replications are uninteresting so won’t publish them; journals aren’t publishing them so grants don’t fund them; grants aren’t funding them, so universities don’t support them; and then, because universities don’t support them, grants don’t fund them, and journals won’t publish them, researchers view them poorly. This perception is the major barrier.

What are replication studies

ECR: Are we conducting replications to “catch” questionable research practices?

JP: No, conducting a replication is not about showing that the original finding was wrong, or anything like that. You’re simply asking, does this replicate? If you find consistent results or inconsistent results, either way, that’s important, and it doesn’t necessarily mean that the original study did anything “wrong”. This is clear to see when we consider longitudinal studies with long time scales. An interesting question is ‘What does knowledge mean in a changing world?’ 20-years ago, everything was different to how it is now. If our ideas are based on studies conducted 20 or more years ago, we actually need to redo them, because everything’s changed, and we need to see what effect this change has had to move our knowledge-base forward. Besides, even if nothing had changed, and everything was done in exactly the same way in both the original and the replication, we don’t expect exactly the same result anyway. Because of sampling, it would actually be really unlikely to get the same result twice. Anyone who has done multiple replications on their own system already knows that there is quite a bit of variation that naturally arises, and the “real answer” will be somewhere within the range of sampled answers.

ECR: Are we meant to be deeply reinvestigating the paper’s methodology, or assuming it was correct?

JP: Replication is about getting a new dataset on the same phenomenon and, with direct replication specifically, trying to get it collected the same way as the first one in order to directly compare them. For this, we do tend to assume that everything was fine with the original study. Obviously, save your time and don’t go out and replicate something you think there is a fundamental flaw with, though. That being said, in some cases, if you think there is a big flaw, then it could be interesting to repeat an experiment with methodological improvements and see what happens.

ECRs and Incentivising Replications

ECR: How do you change the current negative perception of replication studies and encourage researchers to undertake more replication research?

JP: There is already, and will always be, a set of people (for example, those of us in SORTEE), who think all of this is very important and should be actioned right away. However, more broadly, academics are very busy, fighting their own fires, and often struggling to get their own research off the ground, without necessarily having the capacity to be up against these bigger structural challenges. They can be aware of the replication crisis but think: “I don’t know what you expect me to do about it? Unless you can give me a big pot of money, I need to get on with my own work.” There is a silent majority who don’t have the resources to fight the establishment and will go along with whatever is set as the status-quo. Which is why I think there needs to be incentivisation, and that it should be top-down.

All it takes is for there to be a big grant for replication research, and all of a sudden, universities would be excited by replication studies (and the prospect of getting money). Similarly, if journals became more open to publishing replications, then everyone would be happy.

ECR: In your paper Replication studies: a win-win for early-career training and behavioral ecology, you suggest incorporating replication studies into standard research training. While this uptake would be great news for science, until these studies are not considered as equal to original research, these publications may not translate to grants and promotions. In your words: “At best, close replications seem to be widely regarded as a public good that individuals do not benefit from conducting”. How do you convince postgraduate students (at the least secure stage of their career, within their limited candidature timeframe) to take this risk?

JP: The important thing about that quote is that it’s about perception only. Besides, it’s also worth considering that as researchers progress through their career, they do less and less of the hands-on research… The majority of a supervisor’s research comes from their students and postdocs. So, while the Early Career Researcher might be taking a risk undertaking replication studies, the supervisor is taking that risk too. It isn’t one-sided.

In this paper we propose replication as early career training, which actually results in less risk for the new academic. Instead of starting the research journey spending several months overwhelmed by the literature, not knowing the system, and probably making some big mistakes, starting with a replication gives more structure. It’s almost like doing research with guardrails on, providing a defined structure in which to learn about the research method and academic system. In theory, you would choose a study that would later lead on to the other research you intend on doing, and, in the process of replicating it, you learn the methodology and analysis and literature, without having to make the major decisions. Even better, with journals that offer dedicated publication of replication studies (for example, Behavioral Ecology), there is a defined outlet for this research, and publication should be quick. The graduate student therefore immediately gets experience within the entire system, compared to if they were starting out on something completely new from scratch.

It’s also just a very good way for new researchers to get first-hand experience with the fundamental statistical ideas of sampling. Most of the time in ecology and evolutionary biology, a study is only conducted once and then we move on from it. The one answer that study generated is treated like “The Answer” rather than just a sample from the range of possible answers. By getting research graduates to work on a replication themselves, they will quickly learn about variation and stochasticity.

Besides, this is not an all or nothing. We’re in science to find out new stuff, right? No one is suggesting that early researchers only do replication studies. Rather, it’s just that it is very, very important to make sure the stuff that we have already “found out” is verifiable. There just needs to be a balance.

Resources

To learn more, see Joel’s recent webinar, What role do journals play in fighting the replication crisis?.