English

Preference-Consistent Knowledge Distillation for Recommender System

Information Retrieval 2025-01-14 v2

Abstract

Feature-based knowledge distillation has been applied to compress modern recommendation models, usually with projectors that align student (small) recommendation models' dimensions with teacher dimensions. However, existing studies have only focused on making the projected features (i.e., student features after projectors) similar to teacher features, overlooking investigating whether the user preference can be transferred to student features (i.e., student features before projectors) in this manner. In this paper, we find that due to the lack of restrictions on projectors, the process of transferring user preferences will likely be interfered with. We refer to this phenomenon as preference inconsistency. It greatly wastes the power of feature-based knowledge distillation. To mitigate preference inconsistency, we propose PCKD, which consists of two regularization terms for projectors. We also propose a hybrid method that combines the two regularization terms. We focus on items with high preference scores and significantly mitigate preference inconsistency, improving the performance of feature-based knowledge distillation. Extensive experiments on three public datasets and three backbones demonstrate the effectiveness of PCKD. The code of our method is provided in https://github.com/woriazzc/KDs.

Keywords

Cite

@article{arxiv.2311.04549,
  title  = {Preference-Consistent Knowledge Distillation for Recommender System},
  author = {Zhangchi Zhu and Wei Zhang},
  journal= {arXiv preprint arXiv:2311.04549},
  year   = {2025}
}

Comments

TKDE 2024 Accepted

R2 v1 2026-06-28T13:14:55.085Z