Cochrane Colloquium 2026 – Oral Presentation

Give Me Instructions: AI-Assisted Screening

December 2026
Ewa Borowiack, Monika Opalek, Artur Nowak, Ewelina Sadowska

Abstract

Large language models are increasingly used to support title and abstract screening, but the instructions they receive are rarely evaluated as part of the method. Most comparisons focus on which model is used. This oral presentation asks a different question: if the eligibility criteria stay the same, how much does it matter how they are written for the model?

Artur Nowak will present the work on 8 December 2026, from 11:55 to 12:10, at the Cochrane Colloquium in Kraków, in the session Responsible AI in evidence synthesis 1 (ICE Kraków, S2 Theatre). Session details are available in the official Colloquium programme.

Why the formulation of criteria matters

Published eligibility criteria are written for human reviewers, who share methodological background and can ask the review team how to interpret an ambiguous rule. A model sees only the text it is given. Criteria that look clear to a reviewer can leave room for different readings, for example about which populations, comparators or study designs qualify. In practice, review teams often turn their criteria into more explicit screening guidance for human reviewers as well. The study tests whether doing the same for a model makes AI-assisted screening more reliable.

What the study compared

The team evaluated seven published reviews spanning different review types, clinical domains and study designs. Each review was screened with three versions of the same criteria:

  • the published eligibility criteria used verbatim, with the model asked whether a title and abstract met them;
  • operational decision rules written manually by the study team;
  • decision rules generated automatically by GPT-5.1 from the published criteria.

Screening was implemented as structured decision trees in Laser AI (Laser Tags), in which the model answered predefined inclusion and exclusion questions and its answers were mapped to include, exclude or unclear decisions. Each formulation was tested with GPT-5.1 and GPT-5-mini. Performance was measured against the studies included in the original reviews, using sensitivity (the share of included studies that the model retained) and precision (the share of retained records that were included in the review).

What the study found

Screening performance depended to a large extent on how the criteria were formulated, and the pattern differed by model. With GPT-5.1, both types of decision rules improved median sensitivity compared with verbatim criteria. Manually written rules performed best overall, and automatically generated rules came close to them, while precision was broadly similar across the three conditions.

With GPT-5-mini, verbatim criteria gave the highest sensitivity but very low precision, which means that many irrelevant records would still need human screening. Both decision-rule approaches substantially improved precision while keeping sensitivity high overall. Results varied between reviews, so the effect of a formulation or a model cannot be judged from a single review.

Implications for review teams

The findings suggest that instructions are part of the screening method rather than a technical detail. When a team evaluates or reports AI-assisted screening, the way the criteria were expressed matters alongside the choice of model. Generating decision rules automatically from published criteria may capture much of the benefit of manual reformulation, which could make the approach practical at scale. However, because performance varied between reviews, teams should still validate the instructions on their own review and consider both the studies missed and the number of records left for human screening.

Read the accepted abstract or view the official programme.

Blank white image with no visible content or details