DFlash Speculative Decoding on Apple Silicio: Up to 3.3x Performance Boost with MLX

DFlash Speculative Decoding on Apple Silicio: Up to 3.3x Performance Boost with MLX

Recent advancements in Large Language Model (LLM) inference on local hardware have shown significant progress. A developer has implemented DFlash speculative decoding, an advanced technique for accelerating text generation, specifically for the Apple Silicio architecture. This native implementation, built using Apple's MLX framework, promises to unlock new capabilities for professionals working with on-premise and self-hosted deployments.

Initial tests, conducted on an M5 Max system equipped with 64GB of unified memory, revealed a notable increase in performance. For models like Qwen3.5-9B in bf16 format, generation speed reached 85 tokens per second, representing a 3.3x improvement over the baseline without DFlash. These results underscore the potential of software optimizations tailored to fully leverage specific hardware characteristics.

Technical Details and Performance Benchmarks

The DFlash implementation on Apple Silicio relies on a smaller "draft" model that generates a block of 16 tokens in parallel. These tokens are then verified by the "target" model in a single forward pass. This process ensures an output identical to the standard greedy method but with superior efficiency. Measurements exclude prefill time, focusing solely on the generation phase, with token acceptance rates between 80% and 87% across all models.

Detailed benchmarks show varying performance depending on the model and context length:
* Qwen3.5-9B (bf16): Up to 85 tok/s (3.3x speedup) for 1024 tokens and 80 tok/s (3.1x) for 2048 tokens.
* Qwen3.5-4B (bf16): Achieved 109 tok/s (2.7x) for 1024 tokens and 133 tok/s (3.2x) for 2048 tokens, showing an interesting peculiarity: speed increases with longer contexts, as the model is small enough to maintain an optimal balance between draft and verification.
* Qwen3.5-27B (quantized): With 8-bit quantization, 35 tok/s (2.5x) were recorded for 1024 tokens. With 4-bit quantization, performance increased to 44 tok/s (1.9x) for 1024 tokens. It's noteworthy that 8-bit quantization provided a better speedup ratio than 4-bit, as the latter makes the verification phase so fast that the bottleneck shifts to the bf16 draft model.

Key optimizations enabling these results include a patch to support head_dim=256 in MLX's attention, the elimination of one GPU-CPU sync per cycle, and packed QKV projections to reduce the number of kernel dispatches.

Lessons from Apple Silicio Architecture

The development experience revealed crucial aspects of Apple Silicio's unified memory architecture. On these systems, operations are predominantly bandwidth-bound, a factor that alters the dynamics of speculative decoding. Contrary to expectations, implementing custom Metal kernels for operations like batched-GEMV or fused gated SiLU proved slower (0.5-0.8x) than standard MLX GEMM implementations, leading to the decision to revert to baseline solutions.

Another important observation concerns the cost of the verification phase, which remains almost constant (57ms vs 59ms) when moving from 4 to 16 tokens. This suggests that model weight loading dominates the cost, rather than the number of tokens to verify. Consequently, strategies like "verifying fewer tokens when confidence is low" might not yield the expected benefits in this context. For quantized models, the optimization landscape flips: the draft model (bf16) becomes slower than the verified target model (int4/int8), a structural limitation of speculative decoding on bandwidth-bound hardware with quantized targets.

Future Prospects and Implications for On-Premise Deployment

The project is still under development, with ongoing work in several areas. These include draft model compression or distillation for Qwen3.5-27B, aimed at mitigating the bf16 draft bottleneck on quantized targets. Work is also progressing on long context stability, where speedup tends to degrade beyond 2048 tokens due to KV cache growth. The exploration of MoE (Mixture of Experts) models is another priority, with the goal of combining the verification cost of a small model with the quality of a large one.

These developments are particularly relevant for organizations considering LLM deployment on-premise or in air-gapped environments. The ability to achieve high performance on local hardware, such as Apple Silicio, offers concrete alternatives to cloud-based solutions, with positive implications for data sovereignty and TCO. For those evaluating analytical frameworks to compare the trade-offs between on-premise and cloud deployments, AI-RADAR offers resources and insights at /llm-onpremise. The author has stated the intention to open source the code once it is ready, which could further accelerate the adoption of these techniques.