Large Language Model Agents for Evidence Extraction
Abstract
Data extraction is one of the most time-consuming stages of a systematic review, and some of the information it requires is highly contextual. The numbers of participants assessed, randomised and lost to follow-up can be spread across text, tables and flow diagrams, and in pharmacovigilance case reports the drug suspected of causing a reaction is not always named as such. This oral presentation evaluates whether specialised large language model (LLM) agents can extract this kind of information as structured outputs that reviewers can check against the source.
Artur Nowak will present the work on 8 December 2026, from 12:10 to 12:25, at the Cochrane Colloquium in Kraków, in the session Responsible AI in evidence synthesis 1 (ICE Kraków, S2 Theatre). Session details are available in the official Colloquium programme.
Two extraction tasks
The study examined two use cases with different demands. For patient flow, the agents extracted the number of participants assessed for eligibility, randomised and lost to follow-up, by study arm where applicable. For pharmacovigilance, they identified the drugs suspected of causing adverse reactions in published reports. Together, the tasks test both numerical accuracy and the interpretation of causal language.
How the evaluation was designed
Trained reviewers prepared annotation guidelines and reference datasets in Laser AI: 160 clinical studies for patient flow and 71 publications for pharmacovigilance. Training sets were used to develop prompts and configure task-specific agents. Each agent ran as a structured extraction pipeline that produced one record per study, with a supporting quotation, the location in the document and a brief rationale for each value. This makes each output auditable, because a reviewer can see where a value came from and why it was selected. Performance was assessed with F1 scores on held-out test data, together with a qualitative analysis of errors.
What the study found
On text-only inputs, patient-flow extraction achieved a macro-averaged F1 score of 90% across the three variables, and the outputs were generally consistent and traceable. Identification of suspected drugs reached an F1 score of 80% on 50 held-out publications. Most errors occurred in case series describing several patients, where some patient–drug pairs were missed. Other errors involved combination therapies and reports in which suspected causality was implied rather than stated.
The error analysis is as informative as the scores. It shows where contextual judgement remains difficult, and the review of outputs has already helped refine the annotation guidance for future work.
Limitations and ongoing work
The reported scores apply to text-only evaluations. Much patient-flow information appears in tables and CONSORT flow diagrams, and the team has since extended the work to these formats. That work is ongoing, and updated results are planned for the presentation at the Colloquium.
Implications
The study suggests that specialised agents can move beyond narrative summaries towards structured, source-linked extraction across different review tasks. For evidence users and decision makers, this approach could reduce manual workload and support more scalable and consistent evidence pipelines, while keeping each extracted value open to verification.
Read the accepted abstract or view the official programme.
