# PICO Portal Presents at ISPOR 2026 PICO Portal is proud to present its latest research at ISPOR 2026 in Philadelphia. Stop by our poster session (details below). Poster Session 1 - Monday, May 18 10:30am - 1:30pm AI-Assisted Risk Of Bias Assessment In Observational Studies: Isolating Judgment Using A Passage-Based Validation Approach (MSR29) (LINK TO POSTER) AUTHORS: Eitan Agai, Karen A. Source: https://picoportal.org/blog/pico-portal-at-ispor-2026/

PICO Portal is proud to present its latest research at [ISPOR 2026](https://www.ispor.org/conferences-education/conferences/upcoming-conferences/ispor-2026) in Philadelphia. Stop by our poster session (details below). ### [Poster Session 1 – Monday, May 18 10:30am – 1:30pm](https://www.ispor.org/docs/default-source/cti-meeting-21463-documents/69c1ff67-647e-4aa7-8af9-04b9500558e7.pdf?sfvrsn=72e3fcf2_0) [AI-Assisted Risk Of Bias Assessment In Observational Studies: Isolating Judgment Using A Passage-Based Validation Approach (MSR29)](https://www.ispor.org/docs/default-source/cti-meeting-21463-documents/69c1ff67-647e-4aa7-8af9-04b9500558e7.pdf?sfvrsn=72e3fcf2_0) [(LINK TO POSTER)](https://www.ispor.org/docs/default-source/cti-meeting-21463-documents/69c1ff67-647e-4aa7-8af9-04b9500558e7.pdf?sfvrsn=72e3fcf2_0) AUTHORS: Eitan Agai, Karen A. Robinson, Alon Agai OBJECTIVES: To evaluate whether a large language model (LLM) can reproduce expert risk of bias (RoB) judgments when the evidence is held constant, by isolating and validating the assessment step rather than evidence identification. METHODS: We conducted a passage-based validation study using 128 observational studies from a systematic review of per- and polyfluoroalkyl substances (PFAS) and health outcomes. Two experienced reviewers assessed RoB across nine Navigation Guide domains and identified the text passages (evidence) supporting each judgment. These human-selected passages, together with the corresponding Navigation Guide domain questions and guidance, were provided to an LLM. The model was not tasked with locating evidence or reviewing full-text articles. For each passage -domain pair, the LLM generated a structured RoB rating on a five-level scale. Model outputs were compared with human-adjudicated ratings using exact agreement, acceptable agreement, percent agreement, and weighted Cohen’s kappa, overall and by domain. RESULTS: Agreement between LLM-generated and human-adjudicated RoB ratings varied by domain. Concordance was highest in domains relying on explicit reporting, with agreement ranging from 95.1% to 100%, and lower in domains requiring contextual judgement, with agreement ranging from 76.3 to 89%. Most discrepancies reflected partial or acceptable agreement rather than opposing assessments, indicating partial alignment in judgment when evidence was shared. CONCLUSIONS: Separating evidence identification from assessment enables targeted evaluation of AI judgment quality and provides a clearer foundation for responsible integration of AI into evidence synthesis. When restricted to human-curated evidence passages within observational studies, LLMs demonstrated reasoning and assessment similar to that of a human RoB assessor. [

](https://www.ispor.org/heor-resources/presentations-database/presentation-cti/ispor-2026/poster-session-1-4/ai-assisted-risk-of-bias-assessment-in-observational-studies-isolating-judgment-using-a-passage-based-validation-approach) Curious about how to enhance your organization’s review workflows? [Schedule a time with Eitan.](https://calendly.com/eagai/30-minute-meeting)