PICO Portal (PP) prioritizes articles for review using natural language processing and active learning to monitor user decisions on eligibility during title/abstract screening. To predict eligibility, PP uses several algorithms including both decision tree (e.g., SVM, Random Forest Naïve Bayes) and deep learning (e.g., Bidirectional Encoder Representations (BERT)) models.

We tested the system on eight completed conventional SRs (i.e., two independent screeners for title/abstract and full-text eligibility) with a total of 55,812 records (range: 4,204 to 14,188), and on a range of topics from social to biomedical sciences. For each case, we simulated the screening using 100 articles to train and build predictions for eligibility, re-ranking the next 100 articles, and continuing through the records comparing the predicted eligibility with the actual results. We found that with the active AI predictions, reviewers needed to screen only 9-40% of title/abstracts to capture 95% of eligible articles.

Using five SRs as case studies, we show that 40-60% of screening effort can be saved using PICO Portal, an AI-assisted, web-based, SR platform. In one SR, reviewers screened 3600 records to achieve 95% of the predicted abstract includes, but when compared to the predictions and the adjudicated decisions, they had identified 95% of included articles after screening only 600 title/abstracts (16%). Future research should examine the impact of missing the final 5% of articles on review conclusions and assess the resource-benefit ratio.

Our findings and future recommendations are from the researcher and funder perspective. Our conclusions directly impact the amount of time reviewers need to complete a SR.