A recent test demonstrated the possibility of running the Qwen 3.5–9B language model locally on a MacBook Air (M4, 16 GB) thanks to the implementation of Google's TurboQuant compression algorithm.

Implementation Details

The experiment involved patching llama.cpp with the TurboQuant method. Subsequently, the Qwen 3.5–9B model was run with a context window of 20000 tokens. Previously, handling prompts of this size on such a device was considered impractical.

Implications

This development suggests that running open-source language models on consumer devices such as MacBook Air or Mac Mini could become a reality. Although current performance is still limited, advances in hardware promise to further improve inference speed.

Availability

A macOS application implementing this technology is available open source.