Background Artificial intelligence (AI) tools are increasingly used to support antimicrobial prescribing, but most published literature focuses on accuracy rather than reproducibility across identical inputs. Reproducibility is a key requirement for safe clinical decision support. Methods We conducted a comparative experimental study assessing the reproducibility of outputs and the clinical correctness of antibiotic and treatment-duration recommendations generated by a general-purpose large language model (ChatGPT-5.2) and a literature-grounded clinical decision-support system (OpenEvidence). Twelve standardised infectious disease cases (four Gram-positive, four Gram-negative non-fermenters, and four Gram-negative fermenters) were submitted 100 times to each tool using official web interfaces under default settings. Variability was quantified using Shannon entropy, normalised entropy, and the Gini–Simpson index with bootstrap confidence intervals. Clinical correctness was assessed by three blinded infectious diseases specialists. Results Across 2400 independent model interactions, corresponding to 12 clinical scenarios submitted 100 times to each of the two AI tools, substantial variability in response reproducibility was observed. For antibiotic selection, entropy ranged from complete agreement (H = 0) to high heterogeneity (H up to 2.53). ChatGPT showed complete agreement in five of 12 cases, compared with one of 12 for OpenEvidence. In some scenarios, the probability that two responses were different reached 81%. Despite lower reproducibility, OpenEvidence generated more correct antibiotic recommendations (54.8% vs 35.7%; P < 0.001) and more fully correct combined responses (33.4% vs 19.0%; P < 0.001). In mixed-effects analysis, OpenEvidence had higher odds of correct antibiotic recommendations (OR 12.4, 95% CI 8.1–18.7). Conclusion Substantial variability in outputs generated from identical inputs was observed, and reproducibility did not align with clinical correctness. Such variability may lead to inconsistent therapeutic decisions, raising concerns about the use of these tools for direct clinical decision-making. Reproducibility should be considered a key requirement, alongside accuracy, in the evaluation and governance of AI-based clinical decision-support tools.
Reproducibility of AI-generated antibiotic recommendations in standardised clinical scenarios: A proof-of-concept experimental study
Russo, Antonio;
2026-01-01
Abstract
Background Artificial intelligence (AI) tools are increasingly used to support antimicrobial prescribing, but most published literature focuses on accuracy rather than reproducibility across identical inputs. Reproducibility is a key requirement for safe clinical decision support. Methods We conducted a comparative experimental study assessing the reproducibility of outputs and the clinical correctness of antibiotic and treatment-duration recommendations generated by a general-purpose large language model (ChatGPT-5.2) and a literature-grounded clinical decision-support system (OpenEvidence). Twelve standardised infectious disease cases (four Gram-positive, four Gram-negative non-fermenters, and four Gram-negative fermenters) were submitted 100 times to each tool using official web interfaces under default settings. Variability was quantified using Shannon entropy, normalised entropy, and the Gini–Simpson index with bootstrap confidence intervals. Clinical correctness was assessed by three blinded infectious diseases specialists. Results Across 2400 independent model interactions, corresponding to 12 clinical scenarios submitted 100 times to each of the two AI tools, substantial variability in response reproducibility was observed. For antibiotic selection, entropy ranged from complete agreement (H = 0) to high heterogeneity (H up to 2.53). ChatGPT showed complete agreement in five of 12 cases, compared with one of 12 for OpenEvidence. In some scenarios, the probability that two responses were different reached 81%. Despite lower reproducibility, OpenEvidence generated more correct antibiotic recommendations (54.8% vs 35.7%; P < 0.001) and more fully correct combined responses (33.4% vs 19.0%; P < 0.001). In mixed-effects analysis, OpenEvidence had higher odds of correct antibiotic recommendations (OR 12.4, 95% CI 8.1–18.7). Conclusion Substantial variability in outputs generated from identical inputs was observed, and reproducibility did not align with clinical correctness. Such variability may lead to inconsistent therapeutic decisions, raising concerns about the use of these tools for direct clinical decision-making. Reproducibility should be considered a key requirement, alongside accuracy, in the evaluation and governance of AI-based clinical decision-support tools.I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.
