Probabilistic Predictions of People Perusing: Evaluating Metrics of Language Model Performance for Psycholinguistic Modeling

Proceedings of the 10th Workshop on Cognitive Modeling and Computational Linguistics (CMCL 2020)

Venue: CMCL
Type: Workshop
Interpretability
Psycholinguistics
Authors
Affiliation

Sophie Hao

Yale University

Simon Mendelsohn

Yale University

Rachel Sterneck

Yale University

Randi Martinez

Yale University

Robert Frank

Yale University

Published

November 19, 2020

Abstract
By positing a relationship between naturalistic reading times and information-theoretic surprisal, surprisal theory (Hale, 2001; Levy, 2008) provides a natural interface between language models and psycholinguistic models. This paper re-evaluates a claim due to Goodkind and Bicknell (2018) that a language model’s ability to model reading times is a linear function of its perplexity. By extending Goodkind and Bicknell’s analysis to modern neural architectures, we show that the proposed relation does not always hold for Long Short-Term Memory networks, Transformers, and pre-trained models. We introduce an alternate measure of language modeling performance called predictability norm correlation based on Cloze probabilities measured from human subjects. Our new metric yields a more robust relationship between language model quality and psycholinguistic modeling performance that allows for comparison between models with different training configurations.