vLLM Hook: A New Tool for Internal Model Programmability

vLLM Hook, an open-source plug-in designed to extend the programming capabilities of the internal states of large language models (LLMs) deployed via vLLM, has been released. vLLM is an open-source library for model serving and inference.

The plug-in aims to bridge the gaps in the current vLLM implementation, which limits the ability to program the internal states of deployed models. This limitation hinders the use of advanced model alignment and enhancement techniques.

Key Features

vLLM Hook offers two programming modes:

  • Passive Programming: Allows monitoring of the model's internal states for subsequent analysis, without altering its generation.
  • Active Programming: Allows active intervention in the model's generation by modifying the internal states.

The plug-in integrates with vLLM via a configuration file that specifies which internal states to capture. Version 0 of vLLM Hook includes usage demonstrations for prompt injection detection, enhanced retrieval-augmented generation (RAG), and activation steering.

The project invites the community to contribute to the improvement of vLLM Hook via the dedicated GitHub repository: https://github.com/ibm/vllm-hook.