Sense Knowledge Distillation for Decoder Models

A new study introduces an innovative approach to improve the capabilities of large language models (LLMs) based on decoder architectures. The method, called Decoder-based Sense Knowledge Distillation (DSKD), focuses on integrating structured lexical knowledge, such as word senses and their relationships, directly into the training process.

The main goal is to overcome a common limitation in LLMs: the tendency to overlook structured lexical knowledge, despite their ability to learn contextual embeddings rich in semantic information. DSKD allows integrating lexical resources during training, without introducing the need to consult dictionaries during the inference phase, thus maintaining the efficiency of the model.

Experimental results demonstrate that DSKD significantly enhances the performance of decoder models, allowing them to inherit structured semantics and improve their understanding of natural language.