Abstract

Neural dialogue state tracking (DST) models employ various strategies to further push state-of-the-art results in DST benchmarks. However, it is difficult to gauge whether these results, which are on test sets distributionally similar to those seen during training, translate to better real-world performance as these models tend to fail when faced with realistic natural language variations. Qualitative analysis that go beyond comparing benchmark metrics have been sparse, limiting our understanding of comparative strengths of different approaches to DST and subsequently impeding opportunities to combine them. Therefore, a comprehensive and consolidated comparison of DST performance is crucial. To this end, we design CheckDST, a framework with a collection of metrics that formally quantifies robustness to perturbations and facilitates qualitative comparisons by measuring commonly known challenges and problematic behaviors. Using CheckDST, we discover that the robustness and reaction to perturbations of top-performing DST models vary significantly despite comparable joint goal accuracy results (JGA). Although span-based classification models achieve slightly better JGA than generation models, they are significantly less robust to distribution shift and give up entirely on their predictions while generation models continue to attempt at a prediction for the correct slots. Secondly, we observe that while stopping training early, eg, at the first epoch, hurts JGA, the resulting models are significantly more robust to distribution shift. Lastly, guided by weaknesses that CheckDST expose, we propose PrefineDST, a simple generation model pretrained …

Date

2021

Authors

Hyundong Cho, Chinnadhurai Sankar, Christopher Lin, Kaushik Ram Sadagopan, Shahin Shayandeh, Asli Celikyilmaz, Jonathan May, Ahmad Beirami

Journal

arXiv preprint arXiv:2112.08321