From Annotation to Detection: Evaluating LLMs for Hate Speech and Stereotype Identification in Italian

Manual data annotation is often slow, expensive, and difficult to scale-especially for tasks that are subjective and context-dependent, like detecting hate speech and stereotypes. With recent progress in Large Language Models (LLMs), there is growing potential to automate this process. In this study, first, we explore the use of a committee of LLMs (GPT-4omini, Gemini-1.5-flash, and DeepSeek-R1) to generate annotations for Italian social media content. Then, the quality of LLM-generated labels is evaluated by following a teacher-student approach, where we trained a smaller student model (phi3.5-miniinstruct) on them and testing its performance against a humanlabeled dataset. The results indicate that fine-tuning the student model over LLM-generated labels, with careful dataset balancing and hyperparameter tuning, can yield a significant improvement over the baseline (approximately 17% improvement), suggesting that committee-based LLM annotation can provide high-quality labels. Therefore, this work shows the proposed approach can be a reliable and scalable alternative to manual labeling by properly addressing challenges like class imbalance to be sure about the fairness and accuracy of the fine-tuned models.