Accurate predictive models of the visual cortex neural response to natural visual stimuli remain a challenge in computational neuroscience. In this work, we introduce V1T, a novel Vision Transformer based architecture that learns a shared visual and behavioral representation across animals. We evaluate our model on two large datasets recorded from mouse primary visual cortex and outperform previous convolution-based models by more than 12.7% in prediction performance. Moreover, we show that the self-attention weights learned by the Transformer correlate with the population receptive fields. Our model thus sets a new benchmark for neural response prediction and can be used jointly with behavioral and neural recordings to reveal meaningful characteristic features of the visual cortex.
@article{arxiv.2302.03023,
title = {V1T: large-scale mouse V1 response prediction using a Vision Transformer},
author = {Bryan M. Li and Isabel M. Cornacchia and Nathalie L. Rochefort and Arno Onken},
journal= {arXiv preprint arXiv:2302.03023},
year = {2023}
}
Comments
updated references and added link to code repository; add analysis on generalization and visualize aRFs; updated with TMLR publication