Extrapolation in NLP
Computation and Language
2018-05-18 v1
Abstract
We argue that extrapolation to examples outside the training space will often be easier for models that capture global structures, rather than just maximise their local fit to the training data. We show that this is true for two popular models: the Decomposable Attention Model and word2vec.
Keywords
Cite
@article{arxiv.1805.06648,
title = {Extrapolation in NLP},
author = {Jeff Mitchell and Pasquale Minervini and Pontus Stenetorp and Sebastian Riedel},
journal= {arXiv preprint arXiv:1805.06648},
year = {2018}
}
Related papers
View all related →
Machine Learning · Computer Science
Location Attention for Extrapolation to Longer Sequences
Yann Dubois, Gautier Dagan, Dieuwke Hupkes, Elia Bruni
2020-04-23
Computer Vision and Pattern Recognition · Computer Science
Learning Representations that Support Extrapolation
Taylor W. Webb, Zachary Dulberg, Steven M. Frankland, Alexander A. Petrov +2
2023-09-08
Computation and Language · Computer Science
Reverse Transfer Learning: Can Word Embeddings Trained for Different NLP Tasks Improve Neural Language Models?
Lyan Verwimp, Jerome R. Bellegarda
2019-09-11
Machine Learning · Computer Science
On Logical Extrapolation for Mazes with Recurrent and Implicit Networks
Brandon Knutson, Amandin Chyba Rabeendran, Michael Ivanitskiy, Jordan Pettyjohn +3
2025-07-21
Machine Learning · Computer Science
The Extrapolation Power of Implicit Models
Juliette Decugis, Alicia Y. Tsai, Max Emerling, Ashwin Ganesh +1
2024-07-22
Computation and Language · Computer Science
LSG Attention: Extrapolation of pretrained Transformers to long sequences
Charles Condevaux, Sébastien Harispe
2022-10-28
Computation and Language · Computer Science
Understanding the RoPE Extensions of Long-Context LLMs: An Attention Perspective
Meizhi Zhong, Chen Zhang, Yikun Lei, Xikai Liu +4
2024-12-13
Machine Learning · Computer Science
On the Implicit Bias of Gradient Descent for Temporal Extrapolation
Edo Cohen-Karlik, Avichai Ben David, Nadav Cohen, Amir Globerson
2022-03-25
Computation and Language · Computer Science
A Novel Perspective to Look At Attention: Bi-level Attention-based Explainable Topic Modeling for News Classification
Dairui Liu, Derek Greene, Ruihai Dong
2023-10-09
Computation and Language · Computer Science
Memorization vs. Generalization: Quantifying Data Leakage in NLP Performance Evaluation
Aparna Elangovan, Jiayuan He, Karin Verspoor
2021-02-04
Computation and Language · Computer Science
A Length-Extrapolatable Transformer
Yutao Sun, Li Dong, Barun Patra, Shuming Ma +5
2022-12-21
Computation and Language · Computer Science
How language models extrapolate outside the training data: A case study in Textualized Gridworld
Doyoung Kim, Jongwon Lee, Jinho Park, Minjoon Seo
2024-12-09
Machine Learning · Computer Science
World Models
David Ha, Jürgen Schmidhuber
2018-05-10
Computation and Language · Computer Science
Rule Extrapolation in Language Models: A Study of Compositional Generalization on OOD Prompts
Anna Mészáros, Szilvia Ujváry, Wieland Brendel, Patrik Reizinger +1
2024-10-25
Computer Vision and Pattern Recognition · Computer Science
Beyond Self-attention: External Attention using Two Linear Layers for Visual Tasks
Meng-Hao Guo, Zheng-Ning Liu, Tai-Jiang Mu, Shi-Min Hu
2021-06-01
Machine Learning · Computer Science
Learning to Extrapolate: A Transductive Approach
Aviv Netanyahu, Abhishek Gupta, Max Simchowitz, Kaiqing Zhang +1
2023-04-28
Computation and Language · Computer Science
Effective Attention Modeling for Neural Relation Extraction
Tapas Nayak, Hwee Tou Ng
2019-12-10
Computation and Language · Computer Science
Exploring Context Window of Large Language Models via Decomposed Positional Vectors
Zican Dong, Junyi Li, Xin Men, Wayne Xin Zhao +4
2024-11-19
Artificial Intelligence · Computer Science
Concept Extrapolation: A Conceptual Primer
Matija Franklin, Rebecca Gorman, Hal Ashton, Stuart Armstrong
2023-06-21
Computation and Language · Computer Science
Exploring Length Generalization in Large Language Models
Cem Anil, Yuhuai Wu, Anders Andreassen, Aitor Lewkowycz +6
2022-11-15
Machine Learning · Computer Science
Towards Open-World Feature Extrapolation: An Inductive Graph Learning Approach
Qitian Wu, Chenxiao Yang, Junchi Yan
2023-06-14
Information Retrieval · Computer Science
Evaluating Interpolation and Extrapolation Performance of Neural Retrieval Models
Jingtao Zhan, Xiaohui Xie, Jiaxin Mao, Yiqun Liu +3
2022-08-05
Computation and Language · Computer Science
Deep Multi-Task Learning with Shared Memory
Pengfei Liu, Xipeng Qiu, Xuanjing Huang
2016-09-26
Computation and Language · Computer Science
Simple Local Attentions Remain Competitive for Long-Context Tasks
Wenhan Xiong, Barlas Oğuz, Anchit Gupta, Xilun Chen +4
2022-05-05
Computation and Language · Computer Science
What do Toothbrushes do in the Kitchen? How Transformers Think our World is Structured
Alexander Henlein, Alexander Mehler
2022-04-13