This article explores the adaptive relationship between Encoder Layers and Decoder Layers using the SOTA model Helsinki-NLP/opus-mt-de-en, which translates German to English. The specific method involves introducing a bias-free fully connected layer between the Encoder and Decoder, with different initializations of the layer's weights, and observing the outcomes of fine-tuning versus retraining. Four experiments were conducted in total. The results suggest that directly modifying the pre-trained model structure for fine-tuning yields suboptimal performance. However, upon observing the outcomes of the experiments with retraining, this structural adjustment shows significant potential.
@article{arxiv.2405.08570,
title = {Rethinking the adaptive relationship between Encoder Layers and Decoder Layers},
author = {Yubo Song},
journal= {arXiv preprint arXiv:2405.08570},
year = {2024}
}