Reporting Selection Bias
Let’s revise a manuscript to better communicate about selection bias with readers.
You’ll work from a real published study of memory and thinking after Covid-19. We’ve removed the sections that appropriately report on selection bias. Your task is to rebuild the manuscript to practice reporting on selection bias.
Your Goal
Your goal is to revise the manuscript until a reader can assess selection bias, review how it was prevented or mitigated, and find it clearly discussed. Additionally, take note of where in a paper each piece of that reporting belongs.
What You’re About To Do
- Read the incomplete draft and identify what a reader would need.
- Help ensure that the Methods and Results sections are complete.
- Select the best description of how selection bias was mitigated.
- Compare your choices with C4R’s choices.
Read the incomplete draft
Below is an adapted draft of the manuscript with its selection-bias reporting removed. After reading it, identify what a reader would need in order to judge whether selection bias is present.
Details on the study
Cognition and Memory after Covid-19 in a Large Community Sample (Hampshire et al., 2024, New England Journal of Medicine). Researchers asked whether adults who had Covid-19 show objectively measurable cognitive deficits and if so, then for how long any deficits last. Adults from REACT, a large community study in England, were invited to complete an online survey and eight cognitive tasks. Scores were compared across groups defined by illness duration, variant period, and hospitalization.
How thorough is your reporting?
Readers cannot sufficiently judge selection bias based on this draft. What is missing? Select all that apply.
Round 1: Make the report complete
Revise select passages in the Methods and Results. Every option improves the draft. Select one to see what it does well and what any trade-offs are. Keep the revision you would write; your choice is written into the shared manuscript with track changes. Note that you can switch until you move on.
Details on the study
Cognition and Memory after Covid-19 in a Large Community Sample (Hampshire et al., 2024, New England Journal of Medicine). Researchers asked whether adults who had Covid-19 show objectively measurable cognitive deficits and if so, then for how long any deficits last. Adults from REACT, a large community study in England, were invited to complete an online survey and eight cognitive tasks. Scores were compared across groups defined by illness duration, variant period, and hospitalization.
How thorough is your reporting?
Round 2: Show how bias was mitigated and discussed
Your manuscript now shows that selection bias is likely present. Revise additional passages to report and discuss how the authors checked for and addressed bias. As before, every option is an improvement. Keep the one you would write.
Details on the study
Cognition and Memory after Covid-19 in a Large Community Sample (Hampshire et al., 2024, New England Journal of Medicine). Researchers asked whether adults who had Covid-19 show objectively measurable cognitive deficits and if so, then for how long any deficits last. Adults from REACT, a large community study in England, were invited to complete an online survey and eight cognitive tasks. Scores were compared across groups defined by illness duration, variant period, and hospitalization.
How thorough is your reporting?
Compare your choices with C4R’s
Every C4R choice is text derived from what the authors published (Hampshire et al., 2024). Where your choice differs, a note says what the published version adds.
Selection-bias reporting checklist
Methods: Detail who you sampled, who you invited, and the exact steps taken to check and adjust for sample bias.
Results: Map out exactly how many people dropped out at each stage, and contrast participant demographics directly against those invited.
Discussion: Be completely transparent about what your design successfully captured, what it missed, and how those gaps shift your final findings.
Congratulations, you’ve completed the activity!
You revised a manuscript to help readers assess a study in regards to selection bias.
Selection bias must be transparently reported across who you sampled, any participants who left the study, why they left, how these groups compare, and what you say about it.
The next time you read a paper (or write one!) check for: complete reporting, prevention/mitigation, and discussion of selection bias.