Uncertainty Quantification in Generative Models

Uncertainty Quantification (UQ) is critical for making generative models more reliable and robust. A new study introduces Directional Concentration Uncertainty (DCU), an innovative framework for UQ that offers greater flexibility and superior performance compared to existing heuristic methods.

Directional Concentration Uncertainty (DCU)

DCU is a statistical procedure that quantifies the concentration of embeddings based on the von Mises-Fisher (vMF) distribution. The method measures the geometric dispersion of multiple outputs generated by a language model, using continuous embeddings of the outputs themselves, without resorting to task-specific heuristics.

Performance and Generalization

Experiments demonstrate that DCU matches or exceeds the calibration levels of previous approaches, such as semantic entropy. Furthermore, DCU effectively generalizes to more complex tasks in multimodal domains, opening new perspectives for integration into UQ frameworks for multimodal and agentic systems.

For those evaluating on-premise deployments, there are trade-offs to consider carefully. AI-RADAR offers analytical frameworks on /llm-onpremise to support these evaluations.