Is ChatGPT better than me? Analyzing the applicability of Large Language Models to the syllabus of a university course

This paper investigates and compares the use of three Large Language Models (LLMs), i.e., ChatGPT4, Google Gemini, and Claude 3.5, as decision support systems to plan the syllabus of a university course. The experiment was conducted in the context of the bachelor’s degree in Music Informatics of the University of Milan. The course under the exam focuses on the MIDI protocol, a very technical subject in the field of sound and music computing. The responses provided by LLMs have been evaluated by the author, a domain expert who has been teaching the subject for more than 10 years. From here emerges the provocative question in the title: can an AI-based chatbot prove to be more effective than an experienced teacher in defining educational objectives, materials, and the lesson plan for a university course? The results of the experimentation show that all three LLMs enable the formulation of a fairly comprehensive syllabus, allowing for the structuring of a university course. Their responses present interesting points of convergence in terms of both structure and content, while also highlighting some specificities. At present, biases and limitations still exist that make chatbots excellent co-pilots but do not replace the role of the teacher. Finally, a generalization is proposed to outline potential benefits and risks in the applicability of LLMs to the planning of educational activities.