China has unveiled LineShine, a 1.54-exaflop supercomputer based exclusively on CPUs, equipped with 2.4 million Huawei-designed Armv9 cores. This CPU-only architecture represents a strategic response to US GPU restrictions, highlighting an alternative path to achieve high computing capabilities and strengthen technological sovereignty in critical sectors like HPC and AI.
A new llama.cpp fork addresses a long-standing issue with tensor parallelism, enabling the use of quantized KV caches on dual GPU setups. This leads to over a 40% performance increase for LLM inference, demonstrated with a 27B Qwen model on consumer hardware. The solution is crucial for those seeking on-premise efficiency and optimized TCO.
Adlink is focusing on physical AI, integrating AI directly into tangible systems for critical sectors like robotics, healthcare, and semiconductors. This approach demands edge and on-premise solutions to ensure low latency, data sovereignty, and reliability, presenting new challenges and opportunities for hardware infrastructure and deployment.
New benchmarks on AMD Strix Halo hardware explore llama.cpp performance with Qwen3.6 LLMs, comparing standard and MTP versions. Results highlight significant improvements in token generation for both models, with the 27B-MTP showing substantial overall acceleration, especially in long-context chat scenarios. The 35B-MTP model, however, presents a more nuanced picture, with increased generation but slightly higher total time in some tests.
A recent test demonstrated the capability of an RTX 5090 GPU, connected via an eGPU dock to an M-series MacBook, to handle extremely intensive graphical workloads. The experiment, which saw the system run Cyberpunk 2077 at over 100 FPS with max settings and frame generation, highlights the potential of eGPU solutions to extend the computational capabilities of unconventional platforms. This approach offers interesting insights for on-premise deployment scenarios requiring flexibility and computational power.
AMD has released ROCm 7.13, the latest preview of its Core SDK, introducing support for Instinct MI350P GPUs and an expanded range of Ryzen AI APUs. This update is crucial for developers and enterprises utilizing AMD hardware for artificial intelligence workloads, strengthening the software ecosystem in anticipation of the upcoming ROCm 8.0 release and facilitating on-premise deployments.
A detailed analysis explores the energy efficiency of an on-premise setup featuring four NVIDIA RTX 3090 GPUs for Large Language Model inference. Tests reveal a peak efficiency point at 220W per GPU, balancing throughput and power consumption, a crucial insight for those managing local infrastructures and aiming to optimize TCO.
A meeting between former President Trump and President Xi Jinping touched upon 'AI guardrails,' though no formal agreements were reached. Concurrently, deliveries of NVIDIA H200 GPUs to Chinese buyers remain blocked. This scenario highlights the geopolitical complexities influencing the availability of critical hardware for Large Language Models, a crucial factor for on-premise deployment strategies and data sovereignty.
The latest revision of the Vulkan specification, version 1.4.352, includes an important proprietary NVIDIA extension: VK_NV_cooperative_matrix_decode_vector. This new feature aims to optimize matrix operations, which are fundamental for artificial intelligence workloads, including Large Language Model Inference and training. The extension promises performance improvements on NVIDIA hardware, offering new opportunities for on-premise deployments that demand efficiency and control.
xAI's Colossus 1 supercomputer, initially intended for Grok's training, has been reallocated for inference workloads by Anthropic due to its inefficient mixed-architecture design. Meanwhile, Elon Musk is preparing Colossus 2, a new infrastructure based exclusively on Blackwell architecture, designed for frontier model training and with potential implications for future corporate strategies.
The deployment of Artificial Intelligence models, including Large Language Models (LLMs), is no longer confined to cloud data centers. There is growing interest in running AI workloads on local or edge hardware, driven by data sovereignty, low latency, and TCO optimization needs. This approach presents significant challenges related to limited resources but opens new opportunities for innovative and secure applications.
Iceotope, a British company specializing in precision liquid cooling, has closed a $26 million Series B funding round. The investment, led by Barclays Climate Ventures and Two Seas Capital, aims to expand the company's product line and patent portfolio, addressing the growing need to manage heat generated by high-density AI hardware, which now exceeds the capabilities of traditional air cooling systems.
Nvidia has reportedly resolved issues concerning its upcoming Vera Rubin platform, with the supply chain aiming for a production ramp-up in the third quarter of 2026. This timeline is crucial for enterprises planning on-premise AI infrastructures, impacting the availability and deployment strategy for demanding workloads and TCO management.
A growing interest surrounds modded GPUs from China, such as RTX 4090 variants with 48GB of VRAM, for on-premise AI. While offering increased memory crucial for Large Language Models, a significant lack of reliable information in English raises critical questions about software compatibility, stability, long-term reliability, and actual performance. The tech community seeks answers to assess the practical viability of these unconventional hardware solutions.
Foxconn is making a significant strategic move, transitioning from validation to commercialization for AI servers, robotics, electric vehicles, and LEO satellites. This step underscores the company's commitment to expanding its influence beyond traditional manufacturing, focusing on high-growth, technology-intensive sectors, with direct implications for on-premise deployment strategies and the availability of specialized hardware.
`llama.cpp` has released version `b9158`, introducing a significant optimization for Flash Attention specifically targeting AMD's RDNA3 GPU architecture. This update promises to substantially improve performance and efficiency when running Large Language Models (LLM) on AMD hardware, bolstering on-premise deployment capabilities for developers and enterprises focusing on self-hosted solutions.
The acceleration of AI servers is driving the industry towards increasingly advanced PCB technologies. This development is crucial for those managing Large Language Models (LLM) workloads on-premise, directly impacting processing capacity, thermal management, and operational costs. The article explores the implications of this transition for self-hosted infrastructures, highlighting how the choice of PCB technologies becomes an integral part of the deployment strategy.
Nan Ya PCB is increasing its production of high-end integrated circuit (IC) substrates, responding to the growing demand from the artificial intelligence market. This strategic move underscores the importance of advanced hardware components in supporting intensive LLM workloads, with direct implications for on-premise deployment architectures that require high performance and reliability.
Indium Phosphide (InP) compound semiconductors are emerging as a promising technology to overcome current power and bandwidth limitations in AI hardware. This innovation could redefine architectures for Large Language Model (LLM) inference and training, offering crucial advantages for on-premise deployments in terms of energy efficiency and performance, reducing Total Cost of Ownership (TCO), and supporting data sovereignty.
Users of AMD Radeon RX 7800 XT GPUs are reporting a fan management issue following a recent driver update. The Zero RPM feature, designed to silence the card under low load, appears to be causing unexpected temperature increases. This raises questions about software reliability and thermal stability, crucial aspects for on-premise deployments of intensive workloads like LLMs.