📁 LLM

A test on a 16GB RTX 4060 Ti shows Qwen 3.8 27B at 30-35 tokens/s with Q3_xxs quantization, strong coding and logic, but conversational slips. The Q4 floor is not automatic for local deployments.

On a 32GB RTX 5090, a new GGUF NVFP4 quant for Qwen3.8 27B reaches 6,250 tokens/s in prefill with 2048-token prompts, 50% faster than a Q4_0 of the same memory footprint and 4-7% faster than other NVFP4 quants. It includes a quantized MTP draft head and settings to make MTP up to 15% faster.

A Reddit experiment merges Qwen3.8 and Qwen3.6 into a single GGUF file. It passes only a smoke test, but shows how the modularity of self-hosted LLMs is changing incentives, governance, and validation costs.

AntLing has published six checkpoints for Ling-3.0: two sizes, three stages each, all MIT-licensed and without post-training. Material for continued pretraining, fine-tuning and research, not a ready chat model. The stage map is context, not validation; no stage is indicated as best.

An experiment merging Qwen3.8-27B and Qwen3.6-27B, starting from a GGUF file with Q6_K_XL quantization, produced QwenMix-3.7. The author highlights the structural compatibility between the two models, which differ in training by only seven tokens. No testing beyond a smoke test: the case is useful for those evaluating LLM modularity in self-hosted contexts.

Hands-on tests and offline benchmarks suggest Qwen3.8-27B performs worse than Qwen3.6 on factual recall when no external tools are used. For air-gapped deployments relying on model weights alone, the regression is significant.

LongNovel is a multi-scale, bilingual Chinese-English benchmark for detecting hallucinations in long novel summaries. Built on 29 Chinese novels from 16k to 100k tokens and BookSum data, it identifies eight hallucination types. The test set was manually revised. The project signals a shift in perspective: fidelity evaluation becomes an operational criterion for self-hosted deployments on complex documents.

ECASQ jointly optimizes adaptive quantization and lossless compression by minimizing MSE under an entropy budget and an unbiasedness constraint. The optimal dynamic program runs in O(sd^2) time and O(d^2) space. A GPU-friendly approximate version reduces space to O(d) while guaranteeing MSE no larger than the optimal solution using one fewer bit of entropy per entry. Iterative refinement yields near-optimal results.

A developer applied depth pruning to Qwen3.8-27B, bringing it to roughly 22.7 billion parameters without fine-tuning. The model, available in bf16, q8, and q4 on MLX, handles coding, agentic use, and multi-turn conversations with limited degradation, but struggles on edge cases and underspecified prompts. The author published no benchmarks and recommends testing before use.

Unsloth has published new Qwen3.8-27B GGUF files based on Dynamic v3.0. The company reports more than 10% higher accuracy at the same size and a 1-bit quantization retaining 77% accuracy while running on 8GB of RAM. It clarifies the update is not a fix, and releases its imatrix file for community testing and fine-tuning.

Three new Ornith 1.5 models—9B, 35B-A3B, and 397B—have appeared on Hugging Face, each with GGUF versions. The immediate availability of quantized formats signals a direct focus on local and self-hosted deployment, prompting reflection on the trade-offs between size, hardware, and TCO for those evaluating on-premise scenarios.

A new framework aligns brain and text embeddings in a shared semantic space for retrieval-based decoding, separating neural signal from LLM reconstruction. MD-SigLIP uses duplicate-aware contrastive learning and a listwise margin term to enforce ranking constraints between positive and negative clusters. Tests show state-of-the-art retrieval performance on full-vocabulary and subset evaluations. The approach reduces dependence on generative inference and opens the way to local pipelines for sensitive neural data.

The recent release of Qwen 2.4T Max's open weights, despite requiring extreme hardware like B200 clusters, marks a crucial step for the local AI community. While on-premise deployment is complex for the largest version, the initiative paves the way for Quantization options that could bring frontier intelligence to consumer hardware, strengthening data sovereignty and infrastructure control.

Hugging Face has passed three million models published on the Hub. The number includes quantized versions, fine-tunes and conversions, rather than distinct base models. For teams managing local stacks, the milestone shifts the bottleneck from model availability to selection, license verification and production reproducibility. Open distribution is growing, but solid evaluation infrastructure is needed before bringing a model on-premise.

A Princeton-led study put Claude Opus 4.8 agents to work on unpublished NeurIPS 2026 research questions. The agents handled engineering tasks but lacked the judgment and creativity needed for open-ended research, and both papers were rejected. The finding cuts against short recursive self-improvement timelines and reframes hardware and local deployment planning.

HarmProfile collects more than 80,000 validated harmful artifacts from 23 frontier LLMs across 13 model families, organized into 15 harm categories and 57 subcategories. The dataset shifts safety analysis from binary attack outcomes to the distribution of dangerous content. Results show that more capable models not only produce harmful outputs at scale but also display broader, more diverse risk profiles.

FPO adapts LLMs without a backward pass through the model body, reaching 2.7–3.2x the throughput of standard fine-tuning and about 40% less peak training memory. On OLMo-2-7B, Qwen3-8B, and Falcon3-7B, it improves in-domain perplexity while leaving MMLU, ARC-Challenge, HellaSwag, and Winogrande within seed-noise of baseline—something full-network fine-tuning does not reliably reproduce.

A controlled clinical benchmark on gpt-4.1-nano shows 93.5% accuracy but imperfect calibration: confidence rises with evidence distance from the diagnostic boundary and falls with missing information, yet remains too high in moderate, conflicting errors. The result shifts evaluation criteria from accuracy alone to confidence quality in medical deployments.

Fine-tuning on SWE-bench does not transfer capability to suites like Django or LiveCodeBench. The benchmark becomes a training target and stops measuring general ability. For self-hosted LLM operators, an inflated score distorts VRAM, quantization, and TCO calculations. Multi-task evaluation and continuous benchmark maintenance are needed, not a single number.

BCMT separates local token interaction from global context propagation. In tests up to 1024 tokens, it achieves validation performance comparable to Dense Transformers, with higher training throughput and lower memory consumption. The exponential causal memory mechanism is parallelizable and compatible with standard self-attention. A relevant signal for self-hosted deployment, because it eases VRAM constraints without requiring specialized kernels.