A user from the LocalLLaMA Reddit community has initiated a benchmarking project to evaluate the performance of secondhand Tesla GPUs, characterized by a high amount of VRAM, in running LLMs locally.

Benchmark Objective

The main objective is to compare the effectiveness of these GPUs, which are cheaper but have high VRAM, against more recent cards when used in parallel. Many backends for LLMs can take advantage of multi-GPU configurations within a single server, making this comparison relevant.

Methodology

To quantify performance, the user has developed a GPU server benchmarking suite, published on esologic.com. This suite will allow for measuring and comparing the performance of different hardware configurations objectively.

Context

The availability of used Tesla GPUs at affordable prices opens up new possibilities for those wishing to run LLMs locally, maintaining complete control over data and infrastructure. For those evaluating on-premise deployments, there are trade-offs to consider, such as higher initial costs compared to the cloud, but potentially lower costs in the long term, as discussed in AI-RADAR's analytical frameworks on /llm-onpremise.