The starting point is not a datacenter with the latest GPUs, but a batch of FPGAs originally built for mining. The llm.vhdl project documents a concrete experiment: running a 9-billion-parameter LLM quantized to INT4 on two SQRL FK33 boards, each with 8 GB of HBM2 and roughly 400 GB/s of bandwidth, purchased at 280 dollars apiece. At 75 MHz the system produces about 2 tokens/s during generation. The number does not challenge commercial platforms, but it is enough to signal something more important: the entry cost for experimenting with on-premise inference on programmable logic is dropping below the hundreds-of-dollars threshold.

The project uses a distributed pipeline across two FPGAs for Qwen3.5-9B: prefill reaches about 6 tokens/s on a 256-token prompt, dropping to 5.5 tokens/s on 2,300 tokens; generation goes from 3.2 tokens/s in short contexts to 2.4 tokens/s between 2,000 and 3,000 tokens. Verification is done layer by layer against llama.cpp. The bottleneck is not HBM2 bandwidth, but RTL efficiency and clock frequency, still stuck at 75 MHz. Raising the clock requires logic optimizations and higher core voltage. For those evaluating self-hosted deployment, this is the central trade-off: the FPGA offers full control over the pipeline and memory architecture, but pays with an RTL development curve and performance per watt far from an ASIC integration.

The jump to 27B and memory limits

The project extends the same architecture to Qwen3.8-27B INT4, but only through estimates derived from the per-operation profile of the 9B model. On a Jungle Cat, which combines two XCVU35P dies and costs 375 dollars as an ex-mining board, the current design would deliver roughly 1.1 tokens/s in generation at short context and 0.5 tokens/s at 16k. Resizing the RTL for the larger die and raising the clock to 200 MHz brings the estimate to about 8 tokens/s in generation, or 15 tokens/s with tensor parallelism across the two dies. However, four VU35P dies are needed to cover the full 262k context window, because with two dies the KV cache cannot fit alongside the roughly 14.5 GB of weights beyond 45k tokens. At that point the estimate is about 25 tokens/s in prefill and generation at short context, 10 tokens/s at 16k, and 1 token/s at full context.

The ASIC shadow and the on-premise lesson

To set a theoretical boundary for the project, the author asked Opus 5.5 to estimate what would happen by moving the same RTL to a 2023 TSMC N3 process with six HBM3 stacks at 4.9 TB/s. At 2 GHz, the 27B model would reach roughly 294 tokens/s at short context, 106 tokens/s at 16k, and 10 tokens/s at 262k, with estimated consumption between 125 and 340 W. It is a frontier estimate that does not make ASIC a realistic deployment choice for a single experimenter, but it signals the distance between reconfigurable hardware and a specialized implementation. The structural novelty is different: the project does not treat the FPGA as a generic accelerator, but as a test bench to understand where bottlenecks form in an on-premise LLM, including KV cache management and tensor parallelism load. For those evaluating on-premise deployment, AI-RADAR publishes analytical frameworks at /llm-onpremise to assess these trade-offs. The repository is released under the MIT license. The two BC-250 boards purchased for 60 and 75 dollars, used to program the Jungle Cat, complete a lab whose total cost remains under 500 dollars.