Who and How: Using Sentence-Level NLP to Evaluate Idea Completeness

Real-time feedback is very important, yet challenging to provide for free-text learner contributions in Technology-Enhanced Learning. We study whether a generic NLP pipeline can identify completeness features of learner ideas during security training. We apply PoS Tag- ging and Dependency Parsing on contextualised short texts, collected within a dedicated learning environment and we compare the results to an expert-annotated ground truth. We scan these contributions for the absence of responsible stakeholder (who) or featured action (how ). A total of 1174 contributions in two security domains were analysed. We report precision on who (P P V = 0.929) and on how (P P V = 0.691). We consider the first result to be sufficient to provide real-time formative feedback for the case of absent who. Our results suggest that for the purposes of providing feedback in free input problem-solving exercises, generic transformer pipelines without fine-tuning can achieve good performance on stakeholder identification.