Ruibin Agent versus mainstream large language models: A comparative study on medical literature comprehension with esophageal cancer as a case study_Chinese Journal of Clinical Thoracic and Cardiovascular Surgery

Authors：

WEN Pinghua ¹ , JIANG Zhijie ¹ , JIANG Huan ¹ , YUAN Xianglei ¹ , ZHOU Yu ² , MA Hu ³ , LU Chao ³ ,  HU Bing ¹

1. Department of Gastroenterology and Hepatology, Digestive Endoscopy Medical Engineering Research Laboratory, West China Hospital, Sichuan University, Chengdu, 610041, P. R. China;
2. Shanghai Huihao Yisheng Information Technology Co., Ltd, Shanghai, 200080, P. R. China;
3. Huawei Technologies Co., Ltd, Chengdu, 610000, P. R. China;

Corresponding?author：

HU Bing, Email: hubing@wchscu.edu.cn

Keywords：

Artificial intelligence; large language models; interpretation of medical literature; esophageal neoplasms

DOI：

10.7507/1007-4848.202506081

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Objective To explore the application value of artificial intelligence in medical research assistance, and analyze the key paths to achieve precise execution of model instructions, improvement of model interpretation completeness, and control of hallucinations. Methods Taking esophageal cancer research as the scenario, five types of literature including research articles, case reports, reviews, editorials, and guidelines were selected for model interpretation tests. The model performance was systematically evaluated from five dimensions: recognition accuracy, format accuracy, instruction execution accuracy, content reliability rate, and content completeness index. The performance differences of Ruibin Agent, GPT-4o, Claude 3.7 Sonnet, DeepSeek V3, and DouBao-pro models in medical literature interpretation tasks were compared. Results A total of 15 studies were included, with 3 studies of each type. The five models collectively conducted 1 875 tests. Due to the poor recognition accuracy of the editorial type, the overall recognition accuracy of Ruibin Agent was significantly lower than other models (92.0% vs. 100.0%, P＜0.001). In terms of format accuracy, Ruibin Agent was significantly better than Claude 3.7 Sonnet (98.7% vs. 92.0%, P=0.002) and GPT-4o (98.7% vs. 78.9%, P＜0.001). In terms of instruction execution accuracy, Ruibin Agent was better than GPT-4o (97.3% vs. 80.0%, P＜0.001). In terms of content reliability rate, Ruibin Agent was significantly lower than Claude 3.7 Sonnet (84.0% vs. 92.0%, P=0.010) and DeepSeek V3 (84.0% vs. 94.7%, P＜0.001). In terms of content completeness index, the median scores of Ruibin Agent, GPT-4o, Claude 3.7 Sonnet, DeepSeek V3, and DouBao-pro were 0.71, 0.60, 0.85, 0.74, and 0.77, respectively. Conclusion Ruibin Agent has significant advantages in terms of formatted interpretation of medical literature and instruction execution accuracy. In the future, it is necessary to focus on optimizing the recognition ability of editorial types, strengthening the coverage ability of core elements of various types of literature to improve interpretation completeness, and improving content reliability through optimizing the confidence mechanism to ensure the rigor of medical literature interpretation.

Citation： WEN Pinghua, JIANG Zhijie, JIANG Huan, YUAN Xianglei, ZHOU Yu, MA Hu, LU Chao, HU Bing. Ruibin Agent versus mainstream large language models: A comparative study on medical literature comprehension with esophageal cancer as a case study. Chinese Journal of Clinical Thoracic and Cardiovascular Surgery, 2025, 32(10): 1404-1410. doi: 10.7507/1007-4848.202506081 Copy

1.	Wang C, Wang F, Li S, et al. Patient triage and guidance in emergency departments using large language models: multimetric study. J Med Internet Res, 2025, 27: e71613.
2.	Yuan XL, Liu W, Lin YX, et al. Effect of an artificial intelligence-assisted system on endoscopic diagnosis of superficial oesophageal squamous cell carcinoma and precancerous lesions: a multicentre, tandem, double-blind, randomised controlled trial. Lancet Gastroenterol Hepatol, 2024, 9(1): 34-44.
3.	McDuff D, Schaekermann M, Tu T, et al. Towards accurate differential diagnosis with large language models. Nature, 2025, 642(8067): 451-457.
4.	Ye J, Huang C, Chen Z, et al. A multi-dimensional constraint framework for evaluating and improving instruction following in large language models. arXiv:2505.07591.
5.	Tang L, Sun Z, Idnay B, et al. Evaluating large language models on medical evidence summarization. NPJ Digit Med, 2023, 6(1): 158.
6.	Zhang Y, Li Y, Cui L, et al. Siren's song in the AI ocean: a survey on hallucination in large language models. arXiv:2309.01219.
7.	石銳, 鄭兵, 姚巡, 等. "華西黌醫"大模型構建與應用. 中國胸心血管外科臨床雜志, 2025, 32(5): 587-593.Shi R, Zheng B, Yao X, et al. Construction and application of the "Huaxi Hongyi" large medical model. Chin J Clin Thorac Cardiovasc Surg, 2025, 32(5): 587-593.
8.	周小芹, 劉慧珍, 王婷, 等. 基于大語言模型的臨床預測模型研究報告指南(TRIPOD-LLM)解讀. 中國胸心血管外科臨床雜志, 2025, 32(7): 940-946.Zhou XQ, Liu HZ, Wang T, et al. Interpretation of the TRIPOD-LLM reporting guideline for studies using large language models. Chin J Clin Thorac Cardiovasc Surg, 2025, 32(7): 940-946.
9.	Liang H, Fan JH, Qiao YL. Epidemiology, etiology, and prevention of esophageal squamous cell carcinoma in China. Cancer Biol Med, 2017, 14(1): 33-41.
10.	Horie Y, Yoshio T, Aoyama K, et al. Diagnostic outcomes of esophageal cancer by artificial intelligence using convolutional neural networks. Gastrointest Endosc, 2019, 89(1): 25-32.
11.	Azad TD, Chaudhuri AA, Fang P, et al. Circulating tumor DNA analysis for detection of minimal residual disease after chemoradiotherapy for localized esophageal cancer. Gastroenterology, 2020, 158(3): 494-505.e6.
12.	Ishihara R, Arima M, Iizuka T, et al. Endoscopic submucosal dissection/endoscopic mucosal resection guidelines for esophageal cancer. Dig Endosc, 2020, 32(4): 452-493.
13.	Shah MA, Kennedy EB, Catenacci DV, et al. Treatment of locally advanced esophageal carcinoma: ASCO guideline. J Clin Oncol, 2020, 38(23): 2677-2694.
14.	Zhou Q, Wei Y, Zhai H, et al. Comorbid early esophageal cancer and Gongylonema pulchrum infection: a case report. BMC Gastroenterol, 2021, 21(1): 305.
15.	Kitagawa Y, Ishihara R, Ishikawa H, et al. Esophageal cancer practice guidelines 2022 edited by the Japan esophageal society: part 1. Esophagus, 2023, 20(3): 343-372.
16.	Liu CQ, Ma YL, Qin Q, et al. Epidemiology of esophageal cancer in 2020 and projections to 2030 and 2040. Thorac Cancer, 2023, 14(1): 3-11.
17.	Deboever N, Jones CM, Yamashita K, et al. Advances in diagnosis and management of cancer of the esophagus. BMJ, 2024, 385: e074962.
18.	Kato K, Machida R, Ito Y, et al. Doublet chemotherapy, triplet chemotherapy, or doublet chemotherapy combined with radiotherapy as neoadjuvant treatment for locally advanced oesophageal cancer (JCOG1109 NExT): a randomised, controlled, open-label, phase 3 trial. Lancet, 2024, 404(10447): 55-66.
19.	Lin YX, Liu W, Yuan XL, et al. Endoscopic submucosal dissection for the treatment of advanced esophageal cancer subsequent to chemotherapy. Gastrointest Endosc, 2024, 99(6): 1048-1049.
20.	Maan ADI, Koch AD. Artificial intelligence assistance during upper endoscopy: a game changer in detection of esophageal squamous cell carcinoma? Chin Clin Oncol, 2024, 13(6): 89.
21.	Mahawongkajit P, Bunprachoen K. Innovative push percutaneous endoscopic gastrostomy by means of gastropexy and Foley catheter in a patient with advanced head and neck and esophageal cancer. Endoscopy, 2024, 56(S01): E428-E429.
22.	Pubu S, Zhang JW, Yang J. Early diagnosis of esophageal cancer: how to put "early detection" into effect? World J Gastrointest Oncol, 2024, 16(8): 3386-3392.
23.	Wang FM, Mo P, Yan X, et al. Present situation and prospect of immunotherapy for unresectable locally advanced esophageal cancer during peri-radiotherapy. World J Gastrointest Oncol, 2024, 16(1): 1-7.
24.	劉軍, 趙文哲. 人工智能技術在臨床醫學領域的應用與實踐. 中華醫學信息導報, 2025, 40(9): 14.Liu J, Zhao WZ. Application and practice of artificial intelligence technology in clinical medicine. China Med News, 2025, 40(9): 14.
25.	Zhang K, Yang X, Wang Y, et al. Artificial intelligence in drug development. Nat Med, 2025, 31(1): 45-59.
26.	Wang YJ, Yang K, Wen Y, et al. Screening and diagnosis of cardiovascular disease using artificial intelligence-enabled cardiac magnetic resonance imaging. Nat Med, 2024, 30(5): 1471-1480.
27.	韓序, 劉亮, 樓文暉. 生成式人工智能大型語言模型在消化道癌癥領域輔助科研創作的現狀分析: 基于2024年美國臨床腫瘤學會中國學者數據. 中國實用外科雜志, 2024, 44(8): 894-899.Han X, Liu L, Lou WH. A comprehensive analysis of large language models in generative artificial intelligence-assisted research writing: insights from 2024 ASCO gastrointestinal oncology data by Chinese scholars. Chin J Pract Surg, 2024, 44(8): 894-899.
28.	Bao T, Zhang H, Zhang C. Enhancing abstractive summarization of scientific papers using structure information. Expert Syst Appl, 2025, 261: 125529.
29.	Gu Y, Tinn R, Cheng H, et al. Domain-specific language model pretraining for biomedical natural language processing. ACM Trans Comput Healthc, 2021, 3(1): 2.
30.	Moradi M, Samwald M. Improving the robustness and accuracy of biomedical language models through adversarial training. J Biomed Inform, 2022, 132: 104114.
31.	Liang X, Song S, Zheng Z, et al. Internal consistency and self-feedback in large language models: a survey. arXiv:2407.14507.
32.	劉澤垣, 王鵬江, 宋曉斌, 等. 大語言模型的幻覺問題研究綜述. 軟件學報, 2025, 36(3): 1152-1185.Liu ZY, Wang PJ, Song XB, et al. Survey on hallucinations in large language models. J Softw, 2025, 36(3): 1152-1185.
33.	Hu M, He B, Wang Y, et al. Mitigating large language model hallucination with faithful finetuning. arXiv:2406.11267.
34.	Xu N, Ma X. DecoPrompt: decoding prompts reduces hallucinations when large language models meet false premises. arXiv:2411.07457.
35.	Fadeeva E, Rubashevskii A, Shelmanov A, et al. Fact-checking the output of large language models via token-level uncertainty quantification. arXiv:2403.04696.

1. Wang C, Wang F, Li S, et al. Patient triage and guidance in emergency departments using large language models: multimetric study. J Med Internet Res, 2025, 27: e71613.
2. Yuan XL, Liu W, Lin YX, et al. Effect of an artificial intelligence-assisted system on endoscopic diagnosis of superficial oesophageal squamous cell carcinoma and precancerous lesions: a multicentre, tandem, double-blind, randomised controlled trial. Lancet Gastroenterol Hepatol, 2024, 9(1): 34-44.
3. McDuff D, Schaekermann M, Tu T, et al. Towards accurate differential diagnosis with large language models. Nature, 2025, 642(8067): 451-457.
4. Ye J, Huang C, Chen Z, et al. A multi-dimensional constraint framework for evaluating and improving instruction following in large language models. arXiv:2505.07591.
5. Tang L, Sun Z, Idnay B, et al. Evaluating large language models on medical evidence summarization. NPJ Digit Med, 2023, 6(1): 158.
6. Zhang Y, Li Y, Cui L, et al. Siren's song in the AI ocean: a survey on hallucination in large language models. arXiv:2309.01219.
7. 石銳, 鄭兵, 姚巡, 等. "華西黌醫"大模型構建與應用. 中國胸心血管外科臨床雜志, 2025, 32(5): 587-593.Shi R, Zheng B, Yao X, et al. Construction and application of the "Huaxi Hongyi" large medical model. Chin J Clin Thorac Cardiovasc Surg, 2025, 32(5): 587-593.
8. 周小芹, 劉慧珍, 王婷, 等. 基于大語言模型的臨床預測模型研究報告指南(TRIPOD-LLM)解讀. 中國胸心血管外科臨床雜志, 2025, 32(7): 940-946.Zhou XQ, Liu HZ, Wang T, et al. Interpretation of the TRIPOD-LLM reporting guideline for studies using large language models. Chin J Clin Thorac Cardiovasc Surg, 2025, 32(7): 940-946.
9. Liang H, Fan JH, Qiao YL. Epidemiology, etiology, and prevention of esophageal squamous cell carcinoma in China. Cancer Biol Med, 2017, 14(1): 33-41.
10. Horie Y, Yoshio T, Aoyama K, et al. Diagnostic outcomes of esophageal cancer by artificial intelligence using convolutional neural networks. Gastrointest Endosc, 2019, 89(1): 25-32.
11. Azad TD, Chaudhuri AA, Fang P, et al. Circulating tumor DNA analysis for detection of minimal residual disease after chemoradiotherapy for localized esophageal cancer. Gastroenterology, 2020, 158(3): 494-505.e6.
12. Ishihara R, Arima M, Iizuka T, et al. Endoscopic submucosal dissection/endoscopic mucosal resection guidelines for esophageal cancer. Dig Endosc, 2020, 32(4): 452-493.
13. Shah MA, Kennedy EB, Catenacci DV, et al. Treatment of locally advanced esophageal carcinoma: ASCO guideline. J Clin Oncol, 2020, 38(23): 2677-2694.
14. Zhou Q, Wei Y, Zhai H, et al. Comorbid early esophageal cancer and Gongylonema pulchrum infection: a case report. BMC Gastroenterol, 2021, 21(1): 305.
15. Kitagawa Y, Ishihara R, Ishikawa H, et al. Esophageal cancer practice guidelines 2022 edited by the Japan esophageal society: part 1. Esophagus, 2023, 20(3): 343-372.
16. Liu CQ, Ma YL, Qin Q, et al. Epidemiology of esophageal cancer in 2020 and projections to 2030 and 2040. Thorac Cancer, 2023, 14(1): 3-11.
17. Deboever N, Jones CM, Yamashita K, et al. Advances in diagnosis and management of cancer of the esophagus. BMJ, 2024, 385: e074962.
18. Kato K, Machida R, Ito Y, et al. Doublet chemotherapy, triplet chemotherapy, or doublet chemotherapy combined with radiotherapy as neoadjuvant treatment for locally advanced oesophageal cancer (JCOG1109 NExT): a randomised, controlled, open-label, phase 3 trial. Lancet, 2024, 404(10447): 55-66.
19. Lin YX, Liu W, Yuan XL, et al. Endoscopic submucosal dissection for the treatment of advanced esophageal cancer subsequent to chemotherapy. Gastrointest Endosc, 2024, 99(6): 1048-1049.
20. Maan ADI, Koch AD. Artificial intelligence assistance during upper endoscopy: a game changer in detection of esophageal squamous cell carcinoma? Chin Clin Oncol, 2024, 13(6): 89.
21. Mahawongkajit P, Bunprachoen K. Innovative push percutaneous endoscopic gastrostomy by means of gastropexy and Foley catheter in a patient with advanced head and neck and esophageal cancer. Endoscopy, 2024, 56(S01): E428-E429.
22. Pubu S, Zhang JW, Yang J. Early diagnosis of esophageal cancer: how to put "early detection" into effect? World J Gastrointest Oncol, 2024, 16(8): 3386-3392.
23. Wang FM, Mo P, Yan X, et al. Present situation and prospect of immunotherapy for unresectable locally advanced esophageal cancer during peri-radiotherapy. World J Gastrointest Oncol, 2024, 16(1): 1-7.
24. 劉軍, 趙文哲. 人工智能技術在臨床醫學領域的應用與實踐. 中華醫學信息導報, 2025, 40(9): 14.Liu J, Zhao WZ. Application and practice of artificial intelligence technology in clinical medicine. China Med News, 2025, 40(9): 14.
25. Zhang K, Yang X, Wang Y, et al. Artificial intelligence in drug development. Nat Med, 2025, 31(1): 45-59.
26. Wang YJ, Yang K, Wen Y, et al. Screening and diagnosis of cardiovascular disease using artificial intelligence-enabled cardiac magnetic resonance imaging. Nat Med, 2024, 30(5): 1471-1480.
27. 韓序, 劉亮, 樓文暉. 生成式人工智能大型語言模型在消化道癌癥領域輔助科研創作的現狀分析: 基于2024年美國臨床腫瘤學會中國學者數據. 中國實用外科雜志, 2024, 44(8): 894-899.Han X, Liu L, Lou WH. A comprehensive analysis of large language models in generative artificial intelligence-assisted research writing: insights from 2024 ASCO gastrointestinal oncology data by Chinese scholars. Chin J Pract Surg, 2024, 44(8): 894-899.
28. Bao T, Zhang H, Zhang C. Enhancing abstractive summarization of scientific papers using structure information. Expert Syst Appl, 2025, 261: 125529.
29. Gu Y, Tinn R, Cheng H, et al. Domain-specific language model pretraining for biomedical natural language processing. ACM Trans Comput Healthc, 2021, 3(1): 2.
30. Moradi M, Samwald M. Improving the robustness and accuracy of biomedical language models through adversarial training. J Biomed Inform, 2022, 132: 104114.
31. Liang X, Song S, Zheng Z, et al. Internal consistency and self-feedback in large language models: a survey. arXiv:2407.14507.
32. 劉澤垣, 王鵬江, 宋曉斌, 等. 大語言模型的幻覺問題研究綜述. 軟件學報, 2025, 36(3): 1152-1185.Liu ZY, Wang PJ, Song XB, et al. Survey on hallucinations in large language models. J Softw, 2025, 36(3): 1152-1185.
33. Hu M, He B, Wang Y, et al. Mitigating large language model hallucination with faithful finetuning. arXiv:2406.11267.
34. Xu N, Ma X. DecoPrompt: decoding prompts reduces hallucinations when large language models meet false premises. arXiv:2411.07457.
35. Fadeeva E, Rubashevskii A, Shelmanov A, et al. Fact-checking the output of large language models via token-level uncertainty quantification. arXiv:2403.04696.

Chinese Journal of Clinical Thoracic and Cardiovascular Surgery

Ruibin Agent versus mainstream large language models: A comparative study on medical literature comprehension with esophageal cancer as a case study

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content