Choosing an LLM Model for RTX 5090

A user on the LocalLLaMA forum reported winning an RTX 5090 graphics card at NVIDIA's GTC, signed by Jensen Huang. The user, excited about the win, asks the community which language model is best suited for use with this new GPU.

The question implies a local (on-premise) use of the card, opening up interesting scenarios for those who want to run large language models without relying on cloud resources. The choice of model will depend on the specifications of the RTX 5090, such as the amount of VRAM and computing power, information not yet publicly available. It will also be crucial to consider the level of quantization supported by the GPU (FP16, INT8, etc.) to optimize performance and reduce memory usage.

For those considering on-premise deployments, there are trade-offs to consider carefully. AI-RADAR offers analytical frameworks on /llm-onpremise to evaluate these aspects.