On Relative Clause Attachment Preferences in the L2 LSTM LM

다국어 초록 (Multilingual Abstract)

A well-known evaluation technique of neural language models (NLMs) is how models correctly assign probabilities to valid versus invalid syntactic constructions. This methodology suggests a grammatical sentence is more probable than an ungrammatical sentence. In this study, we use ambiguous relative clause attachment to extend such evaluation to cases of multiple simultaneous valid interpretations except for grammaticality differences. We compare model performance in English and Korean L2 learners of English to probe the biases of the L2 LM for ambiguous relative clause attachments. As an initial research step, we implement the L2 Long-Short Term Memory (LSTM) model using the K-English Textbook corpus. We then test the attachment preferences of the L2 LM using the stimuli from Davis (2022). In so doing, we confirm that the L2 LSTM LM prefers a LOW attachment in all test sentences, as shown by the L1 LMs. In addition, the L2 LM’s knowledge of implicit causality is not as robust as that of humans. We find a mismatch between human attachment preferences and NLMs. Through several experiments, we provide additional compelling evidence that there is a broader gap between NLMs and humans during comprehension tasks.

참고문헌 (Reference)

1 Bernardy, J. P., "Using Deep Neural Networks to Learn Syntactic Agreement" 15 : 1-15, 2017

2 Wolf, T., "Transformers: State-of-the-art Natural Language Processing" 38-45, 2020

3 Frank, S. L., "The Interaction between Structure and Meaning in Sentence Comprehension: Recurrent Neural Networks and Reading Times"

4 Smith, N. J., "The Effect of Word Predictability on Reading Time Is Logarithmic" 128 : 302-319, 2013

5 Frank, S. L., "The ERP Response to the Amount of Information Conveyed by Words in Sentences" 140 : 1-11, 2015

6 최선주 ; 박명관, "Syntactic Priming in the L2 Neural Language Model" 103 : 81-104, 2022

7 최선주 ; 박명관, "Syntactic Priming by L2 LSTM Language Models" 22 : 547-562, 2022

8 Ravfogel, S., "Studying the Inductive Biases of RNNs with Synthetic Variations of Natural Languages"

9 Carreiras, M., "Relative Clause Interpretation Preferences in Spanish and English" 36 : 353-372, 1993

10 Davis, F., "Recurrent Neural Network Language Models always Learn English-like Relative Clause Attachment"

11 Van Schijndel, M., "Quantity Doesn’t Buy Quality Syntax with Neural Language Models"

12 Kehler, A., "Pronominal Reference and Pragmatic Enrichment : A Bayesian Account" 1063-1068, 2015

13 Kehler, A., "Prominence and Coherence in a Bayesian Theory of Pronoun Interpretation" 154 : 63-78, 2019

14 Merity, S., "Pointer Sentinel Mixture Models"

15 Davis, F. L, "On the Limitations of Data: Mismatches between Neural Models of Language and Humans" Cornell University 2022

16 Euhee Kim ; 최선주, "On The L2 LSTM LM’s Knowledge of Implicit Causality" 39 : 107-124, 2023

17 Van Schijndel, M, "Modeling Garden Path Effects without Explicit Hierarchical Syntax" 2018

18 Radford, A., "Language Models are Unsupervised Multitask Learners"

19 Liu, X., "Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding"

20 최선주 ; 박명관 ; 김유희, "How are Korean Neural Language Models ‘surprised’ Layerwisely?" 28 (28): 301-317, 2021

21 Lau, J. H., "Grammaticality, Acceptability, and Probability : A Probabilistic View of Linguistic Knowledge" 41 : 1202-1241, 2017

22 Enguehard, E., "Exploring the Syntactic Abilities of RNNs with Multi-Task Learning"

23 Levy, R., "Expectation-Based Syntactic Comprehension" 106 : 1126-1177, 2008

24 Futrell, R., "Do RNNs Learn Human-like Abstract Word Order Preferences?"

25 Peters, M., "Deep Contextualized Word Representations" 1 : 2227-2237, 2018

26 Cuetos, F., "Cross-Linguistic Differences in Parsing : Restrictions on the Use of the Late Closure Strategy in Spanish" 30 : 73-105, 1988

27 Frazier, L., "Construal" MIT Press 1996

28 Gulordava, K., "Colorless Green Recurrent Networks Dream Hierarchically"

29 Fern ndez, E. M, "Bilingual Sentence á Processing" 1-312, 2003

30 Devlin, J., "Bert: Pre-Training of Deep Bidirectional Transformers for Language Understanding"

31 Warstadt, A., "BLiMP : The Benchmark of Linguistic Minimal Pairs for English" 8 : 377-392, 2020

32 Linzen, T., "Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies" 4 : 521-535, 2016

33 Gilboy, E., "Argument Structure and Association Preferences in Spanish and English Complex NPs" 54 : 131-167, 1995

34 Rohde, H., "Anticipating Explanations in Relative Clause Processing" 118 : 339-358, 2011

35 Carreiras, M., "Another Word on Parsing Relative Clauses : Eyetracking Evidence from Spanish and English" 27 : 826-833, 1999

36 최선주 ; 박명관, "An L2 Neural Language Model of Adaptation to Dative Alternation in English" 40 : 143-159, 2022

37 Hale, J, "A Probabilistic Earley Parser as a Psycholinguistic Model" 2001

38 Shannon, C. E., "A Mathematical Theory of Communication" 27 : 379-423, 1948