Background: Artificial intelligence (AI)-based chatbots have shown promise in providing counseling to patients with metabolic dysfunction-associated steatotic liver disease (MASLD). While ChatGPT3.5 has demonstrated the ability to comprehensively answer MASLD-related questions in English, its accuracy remains suboptimal. Whether language influences these results is unclear. This study aims to assess ChatGPT’s performance as a counseling tool for Italian MASLD patients. Methods: Thirteen Italian experts rated the accuracy, completeness and comprehensibility of ChatGPT3.5 in answering 15 MASLD-related questions in Italian using a six-point accuracy, three-point completeness and three-point comprehensibility Likert’s scale. Results: Mean scores for accuracy, completeness and comprehensibility were 4.57 ± 0.42, 2.14 ± 0.31 and 2.91 ± 0.07, respectively. The physical activity domain achieved the highest mean scores for accuracy and completeness, whereas the specialist referral domain achieved the lowest. Overall, Fleiss’s coefficient of concordance for accuracy, completeness and comprehensibility across all 15 questions was 0.016, 0.075 and −0.010, respectively. Age and academic role of the evaluators did not influence the scores. The results were not significantly different from our previous study focusing on English. Conclusion: Language does not appear to affect ChatGPT’s ability to provide comprehensible and complete counseling to MASLD patients, but accuracy remains suboptimal in certain domains.

Evaluation of ChatGPT as a Counselling Tool for Italian-Speaking MASLD Patients: Assessment of Accuracy, Completeness and Comprehensibility / N. Pugliese, D. Polverini, R. Lombardi, G. Pennisi, F. Ravaioli, A. Armandi, E. Buzzetti, A. Dalbeni, A. Liguori, A. Mantovani, R. Villani, I. Gardini, C. Hassan, L. Valenti, L. Miele, S. Petta, G. Sebastiani, A. Aghemo. - In: JOURNAL OF PERSONALIZED MEDICINE. - ISSN 2075-4426. - 14:6(2024 Jun), pp. 568.1-568.11. [10.3390/jpm14060568]

Evaluation of ChatGPT as a Counselling Tool for Italian-Speaking MASLD Patients: Assessment of Accuracy, Completeness and Comprehensibility

R. Lombardi;L. Valenti;
2024

Abstract

Background: Artificial intelligence (AI)-based chatbots have shown promise in providing counseling to patients with metabolic dysfunction-associated steatotic liver disease (MASLD). While ChatGPT3.5 has demonstrated the ability to comprehensively answer MASLD-related questions in English, its accuracy remains suboptimal. Whether language influences these results is unclear. This study aims to assess ChatGPT’s performance as a counseling tool for Italian MASLD patients. Methods: Thirteen Italian experts rated the accuracy, completeness and comprehensibility of ChatGPT3.5 in answering 15 MASLD-related questions in Italian using a six-point accuracy, three-point completeness and three-point comprehensibility Likert’s scale. Results: Mean scores for accuracy, completeness and comprehensibility were 4.57 ± 0.42, 2.14 ± 0.31 and 2.91 ± 0.07, respectively. The physical activity domain achieved the highest mean scores for accuracy and completeness, whereas the specialist referral domain achieved the lowest. Overall, Fleiss’s coefficient of concordance for accuracy, completeness and comprehensibility across all 15 questions was 0.016, 0.075 and −0.010, respectively. Age and academic role of the evaluators did not influence the scores. The results were not significantly different from our previous study focusing on English. Conclusion: Language does not appear to affect ChatGPT’s ability to provide comprehensible and complete counseling to MASLD patients, but accuracy remains suboptimal in certain domains.
artificial intelligence; chatbot; counselling; diet; MASLD; physical activity; steatosis;
Settore MEDS-05/A - Medicina interna
giu-2024
26-mag-2024
Article (author)
File in questo prodotto:
File Dimensione Formato  
jpm-14-00568.pdf

accesso aperto

Tipologia: Publisher's version/PDF
Dimensione 1.3 MB
Formato Adobe PDF
1.3 MB Adobe PDF Visualizza/Apri
Pubblicazioni consigliate

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/2434/1108370
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 2
  • ???jsp.display-item.citation.isi??? 1
social impact