Abstract

Accurately representing changes in mental states over time is crucial for understanding their complex dynamics. However, there is little methodological research on the validity and reliability of human-produced continuous-time annotation of these states. We present a psychometric perspective on valid and reliable construct assessment, examine the robustness of interval-scale (e.g., values between zero and one) continuous-time annotation, and identify three major threats to validity and reliability in current approaches. We then propose a novel ground truth generation pipeline that combines emerging techniques for improving validity and robustness. We demonstrate its effectiveness in a case study involving crowd-sourced annotation of perceived violence in movies, where our pipeline achieves a .95 Spearman correlation in summarized ratings compared to a .15 baseline. These results suggest that highly …

Metadata

publication

Behavior Research Methods 56 (8), 8784-8800, 2024

year

2024

publication date

2024/12

authors

Brandon M Booth, Shrikanth S Narayanan

link

https://link.springer.com/article/10.3758/s13428-024-02503-3

resource_link

https://link.springer.com/article/10.3758/s13428-024-02503-3

journal

Behavior Research Methods

volume

56

issue

8

pages

8784-8800

publisher

Springer US