3 Solving Rigor Problems
DEMO VERSION (NOT FOR PUBLIC DISTRIBUTION)
Summary
Being rigorous is a continual process. Rigor champions weren’t born that way, but started by identifying problems, fixing them, and then sharing those fixes with others. You can do the same by finding areas in your research work that are troublesome, and improving them to be better.
Goals
Realize that struggles in research can be a source of rigor issues.
Find pathways to improving as an individual, and as a group.
Getting started
You have explored various ideological and professional reasons why rigor is a key aspect of pursuing science, but what does that actually mean for your day-to-day work?
A good place to start is to think of a recent problem or struggle in your lab—it could be an opportunity for improving your rigor and your research!
1. Find an issue that is causing you trouble
Do any of these sound familiar?
Have you been receiving emails from colleagues or lab members with questions about the details of a past study?
Have you lost track of scripts you later needed, or do you spend time tracking down old code for features or bug fixes?
Do you struggle with confounding factors or too much variability across samples?
2. Consider a systematic solution
Although there is a straightforward fix for these issues (e.g. answer all the questions by email, fix the code that wasn’t working, start the experiment over), that only resolves the problem once.
A systematic solution could prevent these problems in the first place. Taking a bit of time to improve your rigor practices early could save you a lot of time in the long run!
| Problem | One-off Solution | Systematic Solution |
|---|---|---|
| Many emails about past study details |
Answer all emails |
Review reproducibility checklists |
| Lost code |
Rewrite code from memory |
Proper version control |
| High variability |
Repeat experiment |
Design with proper controls and exclusion criteria |
3. Take the time to investigate solutions
Other researchers have probably encountered the same problem, and they might not only have devised a fix, but if you’re lucky, they’ve documented and shared it with the broader research community! The first step is to engage with a relevant community to see if others have encountered the same problem and have a solution.
| Rigor Resources |
Rigor Communities |
Published Research Advice |
|---|---|---|
| C4R Website | The Turing Way Slack | Ten Simple Rules |
| NINDS Website | ReproducibiliTea journal clubs | Practical Data Science for Stats |
To help ease the search for such places, we’ve assembled collections of tools and communities. Each entry in either collection comes with a landing page designed to help guide you forward, such as a tool tutorial or a community onboarding activity. We encourage you to check out these pages and contribute to the collections!
4. Review your solution with others
When you do find a solution, it’s crucial that you take the extra step of sharing your experience with other lab members, and maybe even update lab guides so that everyone else can avoid that issue in the future.
Don’t worry about getting everything correct immediately—even small, incremental changes to your research practices build over time, establishing good rigor habits and defining the culture of your local environment.
5. Work on identifying rigor issues together
Sometimes, the solution isn’t clear after looking around, or your problem involves more than one person (such that you can’t just make an individual change). An alternative to going alone is to discuss, as a group, which issues are collectively most important.
Give this a try below.
A Rigor Champion isn’t someone who knows every last thing about rigor—it’s someone that cares about proper research methodology and experimental design, and commits to being better, one day at a time.
Takeaways
Common struggles and obstacles in research can be indicative of rigor issues.
Improving our research practices can both make our lives easier and support better science.
There are many resources and groups to help us improve!