A Reddit user (u/toxicdog) managed to run a Gemma 4 LLM directly inside the Godot 4.7 game engine, using only GDScript and Vulkan compute shaders. No llama.cpp, no Python, no server, and no native extensions: everything runs locally. The model's computations are handled by Vulkan compute shaders, while the script handles GGUF loading, tokenization, sampling, and the user interface. The result is an experiment that supports a single model — gemma-4-E2B-it in Q4_K_M quantization — and runs at roughly one-tenth the speed of llama.cpp with CUDA.

Why context matters

This attempt is far from production-ready, but it signals how the technical barrier for running local inference with large language models (LLMs) is lowering. A game engine designed for rendering and physics is repurposed to a completely different task, leveraging GPU access through Vulkan. And it does so without any external dependencies: the model and all the logic reside on the user's machine.

This has implications for interactive application developers. If an LLM can run inside Godot, then a video game — or a simulation, a training environment, an embedded assistant — can offer dynamically generated dialogue, unscripted narrative behaviors, and language interactions without calling cloud APIs. All while staying on-device, with private data and zero network latency. Data sovereignty is ensured by design, without even worrying about GDPR compliance or API costs.

But the flip side is in the numbers. Being ten times slower than a dedicated CUDA optimization with llama.cpp is no small detail. It means that on mid-range gaming hardware, a model with several billion parameters can take seconds to produce a single sentence. The experiment works because the model is small (Gemma 4 ranges from 2 to 27 billion parameters depending on the variant, and Q4_K_M quantization further reduces VRAM footprint), but the performance gap highlights the cost of generality: Vulkan is a cross-platform graphics and compute API, but transformer-specific optimizations — like fused CUDA kernels — are not easily replicated without significant engineering investment.

Who gains from this scenario

The indie developer and modding community. Godot is an open-source engine with growing adoption, and being able to experiment with an LLM in GDScript — a Python-like language integrated into the engine — without wrestling with native compilations opens the door to rapid prototyping. GPU hardware providers also win: every new local workload that stays off a cloud server increases demand for cards capable of general computation, not just rasterization.

Conversely, cloud inference services (like those from big vendors) see these experiments as a niche, but the long-term trend toward on-device processing erodes their grip on certain application categories. If tomorrow game engines integrated optimized inference runtimes, the need for API calls for every dialogue would fade, saving operational cost (TCO) for developers and providing a more responsive user experience.

Structurally, what it tells us

This experiment shows that local LLM inference is no longer confined to specialized tools. The ecosystem is expanding sideways: from the classic toolkit (llama.cpp, Ollama, vLLM) to game engines, productivity apps, even embedded environments. The GGUF format and quantization allow models to be carried into unexpected contexts, while consumer GPU compute APIs (Vulkan, DirectML, Metal) become progressively more capable of executing neural networks without too much hassle.

Yet the gap between "it works" and "it works well" remains wide. The challenge is to close it with deeper integrations: if Godot or Unity included an optimized inference backend (perhaps using llama.cpp as a library), performance could approach native speeds, making local AI a standard component of game development. Meanwhile, those choosing on-premise LLM deployment today can view these experiments as a maturity indicator: the technology isn't ready for the masses yet, but the direction is clear.

The project, open source on GitHub, is a taste of what could become a fundamental building block for future game design: language models no longer as external services, but as resources integrated into the game architecture, loaded like textures and scripts, running on the same silicon that draws the polygons.