Two semiconductor heavyweights are rewriting the rules for running Large Language Models locally. While Qualcomm is pushing ahead with its Dragonfly platform, designed for on-device inference and low-power processing, MediaTek is not sitting idle: its ties with TPU suppliers and in-house ASIC expertise keep it firmly in the race.

This isn’t just about raw speed. Moving LLM inference from the cloud to the edge means lower latency, no reliance on constant connectivity, and, crucially, keeping data under direct control. For regulated industries or businesses that cannot send sensitive prompts to external data centers, edge execution becomes the only viable path to adopting generative AI without compromising data sovereignty.

Qualcomm has built its offering on tight integration between CPU, GPU, and neural processing units, with the clear goal of running quantized models—at INT8 or FP16 precision—within a tight thermal envelope of just a few watts. Dragonfly is the commercial face of this effort: an ecosystem of tools and libraries aimed at smoothing the path for developers building LLM-based applications, from lightweight fine-tuning to on-device inference. Exact throughput and latency figures are still under wraps, but the direction is unmistakable: bring text-based reasoning capabilities, once the preserve of discrete GPUs with hundreds of GB of VRAM, to the network’s edge.

MediaTek is taking a different route. The Taiwanese company has invested in dedicated APU (AI Processing Unit) cores and, according to DIGITIMES sources, is forging strategic agreements to embed custom TPUs and ASIC accelerators tailored for AI workloads. This hybrid approach—mixing proprietary IP with external partnerships—allows it to offer modular solutions that can scale from mobile to industrial IoT devices, all while keeping a competitive cost profile. That matters: for an organization evaluating an on-premise deployment spread across dozens or hundreds of nodes, the TCO of inference hardware becomes the decisive factor.

Beyond technical specs, software ecosystem maturity is key. Qualcomm brings years of experience with the Snapdragon framework and a developer community accustomed to its DSPs. MediaTek, meanwhile, is ramping up support for ONNX Runtime and TensorFlow Lite, working to close the gap in deployment tooling. For teams building on-premise AI pipelines, the quality of development tools and ease of integration with existing orchestration systems count as much as advertised TOPS numbers.

In this landscape, what’s at stake goes beyond vendor rivalry. The arrival of chips ever more capable of running LLMs locally expands the on-premise computing frontier, opening up scenarios once unthinkable: fully air-gapped voice assistants, edge servers for sensitive document analysis, industrial gateways processing natural language without ever leaving the factory network. For architecture decision-makers, weighing compute power against energy constraints and compliance demands requires robust analytical frameworks. On AI-RADAR, deep dives into self-hosting toolchains offer a compass through these choices, helping avoid the trap of passing fads.