SanityBoard, a platform for evaluating large language models (LLMs), has recently added new benchmark results and features.

New Models and Agents

The update includes results from 27 new evaluations, including models such as Qwen3.5 Plus, GLM 5, Gemini 3.1 Pro, and Sonnet 4.6. Three new open source agents focused on code generation have also been integrated: kilocode cli, cline cli, and pi.

Performance Analysis

The author highlights how GPT-codex models tend to perform better in these benchmarks due to their propensity for iteration. In contrast, Claude models, which iterate less, may be disadvantaged in this type of evaluation. However, it is emphasized that Claude models may be more suitable in interactive coding scenarios.

Importance of Infrastructure

A crucial aspect highlighted is the significant impact of the infrastructure used on model performance. The speed and quality of the infrastructure can greatly influence benchmark results. The author has tried to mitigate this effect through multiple retries and manual checks, but acknowledges that z.ai's infrastructure has presented problems, making evaluation via their API difficult.

For those evaluating on-premise deployments, there are trade-offs to consider carefully. AI-RADAR offers analytical frameworks on /llm-onpremise to support these evaluations.