Human Expert Labeling Process (HELP): Towards a Reliable Higher-Order User State Labeling by Human Experts



In our longitudinal research, we have been working towards an adaptive learning system automatically detecting student engagement as a higher-order user state in real-time. The labeled data necessary for supervised learning can be obtained through labeling conducted by human experts. Using multiple labelers to label collected data and obtaining agreement among different labelers on same samples of data is critical to train final engagement model accurately. Addressing these challenges, we developed a rigorous labeling process (HELP) specific to educational context with multi-faceted labels and multiple expert labelers....