Methodological Limitations of LLMs in Systematic Literature Reviews: A Synthesized Analysis

Framing the Problem

Systematic literature reviews derive their authority from reproducibility, comprehensiveness, and methodological transparency. Large language models have structural properties that are in direct tension with each of these values. Understanding what kind of limitation each issue represents — not just that limitations exist — determines which mitigations are genuinely effective versus cosmetically reassuring. The honest starting point is that the evidence base for LLM use in SLRs is itself immature; most guidance reflects a priori reasoning rather than robust comparative research.

---

Core Limitations and Targeted Mitigations

1. Hallucination and Confabulation

The limitation: LLMs optimize for textual plausibility, not factual accuracy. These are correlated but distinct objectives. A model trained on research literature learns what research sounds like, enabling it to produce convincing citations with plausible author names, realistic journal titles, and reasonable publication years — that never existed. This is not occasional error; it is a structural feature of how these models generate text.

Why standard quality checks may fail: Fabricated citations often pass superficial scrutiny. The fabrication may only become apparent upon attempting actual retrieval.

Mitigations:

• Treat zero LLM-generated references as pre-verified. Every citation requires independent verification against primary databases (PubMed, Scopus, Web of Science).
• Never ask an LLM what a study finds — only ask it to analyze text you have already retrieved and provided.
• Use Retrieval-Augmented Generation (RAG) pipelines where the model is constrained to a corpus you control. Pair this with explicit negative constraints: "If the information is not in the provided text, state 'Not Found.' Do not use external knowledge."
• Acknowledge that this limitation is partially non-mitigable: some fabrication risk persists even with careful prompting. This should be stated explicitly in published reviews.

---

2. Reproducibility and Stochastic Inconsistency

The limitation: LLMs produce different outputs for identical or near-identical inputs across sessions, prompt framings, and API version updates. This violates a foundational principle of systematic reviews: that documented procedures can be replicated by independent researchers.

The deeper issue: A "method" that cannot be fully specified cannot be fully replicated. Even fixing temperature to zero reduces but does not eliminate variability, because model weights themselves change across versions.

Mitigations:

• Set temperature to zero (or minimum available setting) for all screening and extraction tasks.
• Document and archive the exact model version, API timestamp, system prompt, and all user prompts. Publish these as supplementary materials.
• Run duplicate screening passes and calculate inter-session agreement statistics — treat the LLM as you would a second human rater and assess its reliability formally.
• Use "chain-of-thought" prompting with logging: require the model to explain its reasoning step-by-step, and store these intermediate steps as part of the audit trail.
• Pre-register the LLM-integration protocol where possible, analogous to PROSPERO registration for traditional SLRs.

---

3. Knowledge Cutoffs and Coverage Gaps

The limitation: LLMs have fixed training cutoffs and cannot access recently published literature. More subtly, even within their training window, coverage is structurally uneven: English-language, open-access, and high-citation work is overrepresented relative to gray literature, non-English sources, and specialized regional databases. This creates structural bias, not merely a gap.

Mitigations:

• Never use an LLM as a search source. Use traditional, time-stamped databases (PubMed, Scopus, EMBASE, Web of Science) as the primary retrieval mechanism. LLMs should assist only with processing a corpus already retrieved through systematic methods.
• Document the LLM's training cutoff explicitly in the review's limitations section.
• For recency, integrate real-time database APIs (Semantic Scholar, arXiv) through tools like LangChain or Rayyan, while recognizing that even these require human verification.
• Gray literature searches must be conducted through traditional means — LLMs cannot remedy geographic or publication-type coverage gaps.

---

4. Prompt Sensitivity and Construct Validity

The limitation: LLM outputs vary substantially with prompt wording, framing, and presentation order. This means the inclusion criteria as operationalized in a prompt may not faithfully represent the inclusion criteria as intended by the researcher — and this gap is difficult to detect without deliberate testing.

In traditional screening, operationalization happens through human judgment, documented examples, calibration sessions, and training. With LLMs, the operationalization happens through natural language that may be interpreted differently than intended.

Mitigations:

• Pilot-test prompts against a gold-standard set of articles with known inclusion/exclusion status before full deployment. Calculate sensitivity and specificity.
• Use multiple prompt formulations for the same task and check for consistency across them.
• Have subject matter experts review LLM decisions on a random sample throughout the process — not just at the end.
• Be aware that "optimizing" prompts on a pilot set may overfit to that set; the gold standard sample should be held out from prompt development.

---

5. Selection Bias and Bias Amplification

The limitation: LLMs absorb the biases present in their training data — publication bias toward positive results, geographic concentration in the Global North, English-language dominance, and methodological preferences of high-impact journals. When used for screening or synthesis, they may systematically favor certain types of evidence in ways that are difficult to detect using conventional review tools.

Why this is insidious: Traditional SLRs have explicit mechanisms for addressing publication bias (funnel plots, gray literature searches, GRADE assessments). LLM-mediated reviews may introduce new bias vectors without comparable detection tools.

Mitigations:

• Audit LLM decisions for systematic patterns: does the model disproportionately exclude certain study designs, geographic regions, or non-English sources?
• Explicitly prompt for diversity when using LLMs for any exploratory tasks: request attention to studies from underrepresented regions, preprints, and negative results.
• Recognize that this limitation warrants explicit acknowledgment in the published review — not just in internal documentation.
• Cross-validate screening decisions across multiple LLM systems (e.g., GPT-4, Claude, Gemini) to detect model-specific biases through disagreement.

---

6. Inadequate Critical Appraisal and Domain Judgment

The limitation: LLMs lack genuine domain expertise. They can pattern-match to disciplinary conventions but may fail to recognize methodological subtleties that affect quality assessment, context-dependent relevance, inconsistent field-specific terminology, or implicit assumptions embedded in research designs. They cannot reliably distinguish high-quality from low-quality evidence when the distinction requires substantive judgment rather than pattern recognition.

Mitigations:

• Reserve quality appraisal tasks for human domain experts. This is not currently a task that should be delegated to LLMs under any framing.
• Use structured quality appraisal tools (GRADE, RoB 2, CASP, AMSTAR) administered by humans — LLMs may assist with data extraction but should not render quality judgments.
• Use a "devil's advocate" prompting strategy — deploy a second LLM instance specifically tasked with identifying contradictions, methodological flaws, and outliers — to surface what a consensus-seeking model might smooth over. But treat this as hypothesis-generation for human review, not as appraisal itself.

---

A Framework for Task-Appropriate LLM Use

Not all SLR tasks carry equal risk. The following reflects a principled allocation of LLM involvement:

The appropriate mental model is the LLM as a high-throughput first-pass filter — not as a synthesizer, not as a quality assessor, and never as a source. Its outputs at every stage require human verification against primary sources.

---

Transparency and Reporting Requirements

Any SLR that integrates LLMs should report:

• The specific model, version, and API timestamp used
• All prompts, temperature settings, and parameter choices (published as supplementary materials)
• Which tasks involved LLM assistance and which did not
• The validation procedure applied to LLM outputs and its results
• An explicit acknowledgment that full independent replication may not be achievable

The PRISMA framework does not yet have a fully validated AI extension, but researchers should apply its spirit: document every decision point at which an LLM was involved so that a reader can assess where methodological risk concentrates.

---

An Honest Conclusion

The greatest risk is not that researchers will use LLMs in obviously careless ways. It is that they will use them in seemingly careful ways that nonetheless introduce systematic distortions not detectable through conventional review processes — hallucinations that survive into print, biases that scale existing problems, or prompt-operationalized criteria that diverge silently from the researcher's intent.

LLMs can reduce the burden of tedious, high-volume tasks at the margins of SLR methodology. They cannot substitute for the methodological rigor that makes systematic reviews worth conducting in the first place. Used as a turbocharged assistant under strict human oversight, they offer genuine efficiency gains. Used as an authority, they are a mechanism for laundering poor evidence with the appearance of systematic rigor.

The standard should be: if you would not trust this process for a clinical guideline affecting patient care, revise it until you would — or be transparent that your review falls short of that bar.