Polynomial Regression as a Task for Understanding In-context Learning Through Finetuning and Alignment
Machine Learning
2024-07-30 v1 Computation and Language
Abstract
Simple function classes have emerged as toy problems to better understand in-context-learning in transformer-based architectures used for large language models. But previously proposed simple function classes like linear regression or multi-layer-perceptrons lack the structure required to explore things like prompting and alignment within models capable of in-context-learning. We propose univariate polynomial regression as a function class that is just rich enough to study prompting and alignment, while allowing us to visualize and understand what is going on clearly.
Cite
@article{arxiv.2407.19346,
title = {Polynomial Regression as a Task for Understanding In-context Learning Through Finetuning and Alignment},
author = {Max Wilcoxson and Morten Svendgård and Ria Doshi and Dylan Davis and Reya Vir and Anant Sahai},
journal= {arXiv preprint arXiv:2407.19346},
year = {2024}
}
Comments
ICML Workshop on In-Context Learning