Community tests and vendor announcements show large language models running locally on accessible GPUs, workstations, and edge devices, from AMD Radeon/W7900 to Intel D-Series and even a jailbroken PS5. VRAM, context length, and efficient serving are now central to self-hosted AI deployment.
A local test with EmbeddingGemma2 shows semantic image search on a consumer GPU, indexing 3,003 screenshots in 510 seconds with a 330 MB Q8 model and 768-dimensional vectors. But retrieval quality depends heavily on query wording and corpus character...
A homelab, not a vendor, documented a 27-billion-parameter LLM with 262,000 tokens of context on a single Radeon RX 9060 XT with 16 GB of VRAM, using quantization, KV cache streaming, and software fixes. The signal: the entry cost for high-context se...
A homelab with a single 16GB Radeon RX 9060 XT and 24GB of DDR5 runs a 27-billion-parameter LLM at 262,000 tokens of context. The setup uses IQ4_XS quantization and KV cache streaming, reporting 42.7 tokens/s generation at 20k context.
The open source project Openspline adds an avatar layer to existing voice AI pipelines: from an image, it generates a talking face with synchronized lips, executed locally on a GPU. The lite model exceeds 90 FPS on an NVIDIA RTX 4090 and integrates w...
The ORL (On-Policy Residual Learning) project aims to speed up diffusion LLMs without sacrificing quality. On a single RTX 5090 the team reports 2,788 tokens/s with LLaDA-2.1-mini in NVFP4; the open-source dlmrt inference engine uses CUDA graphs and ...
Intel software engineers have released LLM-Scaler-vLLM PV 1.5, a containerized Docker stack designed to simplify LLM inference on Intel Arc hardware. The release signals a software investment aimed at making Intel's discrete GPUs more competitive in ...
A real-world test shows affinity handles four Radeon AI PRO R9700 cards without degenerate output. The 2.875 bpw quant keeps 11,008 experts in VRAM, reaching 79 tok/s greedy decoding with no host pool allocated. The earlier bug was an artifact of sim...
A 24-billion-parameter LLM tested on an AMD W7900 with a 128,000-token context and sustained speed around 95 tokens per second signals that on-premise inference is maturing beyond NVIDIA's ecosystem. AI-Radar explores trade-offs, data sovereignty, an...
A user reported a positive experience with LiquidAI's LFM2.5-24-A2B model (24B parameters), highlighting high performance (~95 tps) on an AMD W7900 GPU. The model, requiring 30GB of VRAM for Q8 quantization with a 128k token context, proves reliable ...
A jailbroken PS5 performs inference with Qwen 27B at around 20 tokens per second on its GPU. The Reddit report shows how consumer hardware with unified memory can support self-hosted inference for a 27B-parameter model, lowering the barrier for on-pr...
Intel has turned Crescent Island into a commercial lineup: the D-Series. The D270 card combines 32 Xe3P cores, 256 XMX engines, up to 480 GB of LPDDR5X and 1.5 TB/s of bandwidth. Rated performance reaches 1.3 PetaFLOPS FP8, with support from FP4 to F...
Intel has announced the GPU D Series, the successor to products known as Crescent Island and aimed at enterprise inference. The announcement lacks VRAM or performance details, but the renaming signals a more direct positioning in the market for on-pr...
Intel has named its Crescent Island data center inference card D-Series and revealed D270 specs: 1.3 PFLOPS dense FP8 and 1.5 TB/s memory bandwidth. A signal for on-prem LLM serving economics.
