Who Should Do It? Automatic Identification of Responsible Stakeholder in Writings During Training

In online Problem-Based Learning (PBL), being able to provide immediate feedback to learners is invaluable, yet difficult to achieve. We examine how well an off-the-shelf Natural Language Processing (NLP) framework is able to detect the absence of an identified responsible stakeholder in ideas generated during security training. Part-of-Speech Tagging and Dependency Parsing are applied on contextualised written learner contributions, collected from a PBL environment and compare the results to an assessment performed by experts. Using grammatical analysis, we aim to detect the absence of an identified responsible stakeholder in collected contributions (n=1174) from two security domains. Four heuristics are compared, resulting in a precision of (PPV=0.929) on the best of these, sufficient to provide immediate feedback to learners. Our results suggest that for the purposes of scaffolding open-ended PBL exercises, off-the-shelf NLP frameworks can achieve good performance on responsible stakeholder identification.