I’ve been in my current position as a librarian for an evidence synthesis team for nine months now, and I’m finally feeling settled and confident in this position. Over the past two months or so, I’ve been given more opportunities to be involved in larger projects like our rapid review products and as part of this, I’ve been brushing up on my quality assessment skills.

Prior to this job, I didn’t have much experience with quality assessment. I knew some basic information, like where to find the Cochrane Risk of Bias tool, and that different study types needed to be assessed differently, but I didn’t have any practical experience. It’s exciting for me to be able to add another skill to my librarian toolbox and to be more involved in my team’s projects.

In our rapid review products, we include just about all study types: randomized controlled trials (RCTs), cohort studies, case studies, etc. When it comes to assessing bias, we sort the included studies into either RCTs or non-RCTs. For RCTs, we use the gold standard Cochrane Risk of Bias tool and for non-RCTs, we use the Robins-I tool.

The easiest way to keep track of your different bias assessments is to create a table where each row is a different study and each column is a different bias domain. Using the tool, you rate each bias domain on a scale of low risk, some concerns, or high risk, and then provide an overall rating of the study (this is a very broad strokes description of the process, fyi). It sounds easy enough, but I found when putting it into practice that there’s a lot of skill and experience that goes into providing a quality assessment of quality (ha!).

A good rule of thumb when rating a study is to first imagine what the perfect version of this study would look like. How would it be blinded? How would the procedure be carried out? How would it be measured? Once you have the perfect study in mind, compare it to the actual study; any shortcomings between the actual study and the perfect study are areas at risk for bias.

Now, for rapid reviews, there is no one right way to conduct the review and there is a lot of gray area around whether quality assessment is included or not; Haby et al. (2016) found that one of the most common shortcuts in rapid review methodology is to eliminate the quality assessment stage. It’s basically up to the review authors to decide, and there is reasoning to both include and exclude this part of the review process. Since the purpose of a rapid review is to synthesize evidence more rapidly than a traditional systematic review, the quality assessment stage is often eliminated to save time; the trade-off here is a lower quality review.

If you’re doing a rapid review and trying to decide if you should include a quality assessment, here are some considerations:

• How many people are on your team? If you have the manpower, doing a one- or two-person quality assessment may be feasible.
• How many studies are you including in your synthesis? If you’re dealing with a large volume of literature, you may want to skip quality assessment to save time. Or, limit the quality assessment to only RCTs or some other study type.
• What is the purpose of your rapid review? If you’re balking at skipping the quality assessment stage, maybe you need to re-evaluate the reason why you’re conducting this rapid review and consider a different review type.

Here are some great resources to check out if you’re interested in brushing up on your quality assessment skills:

• Assessing risk of bias in a randomized trial (Cochrane)
• Assessing risk of bias in a non-randomized study (Cochrane)
• ROBIS: A Risk of Bias Assessment Tool for Systematic Reviews (McMaster University)
• Tools for assessing risk of reporting biases in studies and syntheses of studies: a systematic review (Page et al., 2018)