In July 2026, an innovative inference engine technology was introduced that enables massive AI models to run in typical PC environments. By dynamically reading only the necessary data from storage, the memory walls are virtually being removed.

The emergence of new technologies that achieve remarkable memory reductions

On July 10, 2026, the inference engine “Colibrì,” unveiled by developer JustVugg, caused a major shock in the AI industry. Typically, running ultra-large models like the “GLM-5.2” with 744 billion parameters or the “Hy3” with 295 billion parameters requires expensive servers equipped with hundreds of GB of VRAM or main memory. However, this new technology has made these monster-class models possible on typical consumer PCs with only 10GB to about 25GB of memory.

The biggest feature of this technology is that instead of loading the entire model into memory, only the necessary data is read from the SSD at any time. The fact that frontier-level open models, which previously had no hands or feet for individuals, now “run” on familiar devices like laptops holds the potential to fundamentally change the way AI inference works. As shown in the comparison diagram below, the traditional barrier of physical memory limitations is being overcome through a new approach of utilizing storage capacity.

The true identity of the ultra-large model “Hy3” that targeted

One of the main models featured in this technology is the “Hy3,” developed by the Tencent Hy team in China. The Hy3 boasts an enormous scale of 295 billion parameters (295B), and the high-precision BF16 format data size reaches approximately 597GB. This model adopts an architecture called “Mixture-of-Experts (MoE),” limiting the number of active parameters actually activated per inference (one token creation) to about 21 billion (21B).

Hy3 will be released under the Apache 2.0 license in July 2026 and is attracting attention as an open model capable of commercial use. In benchmarks, it claims top-class performance among models under 500B, and reportedly significantly reduced the incidence of hallucinations from the previous 12.5% to 5.4%. The significance of being able to run such ultra-large, high-precision models in environments with around 10GB of memory is extremely significant.

Innovative mechanisms that utilize storage

How “Expert Streaming” Works

The core technology used by the Colibrì series engines is called “expert streaming.” This maximizes the MoE model’s characteristic that “you don’t need to use all parameters during calculations.” Specifically, the model is managed by dividing it into the following two categories.

• Resident part: Approximately 17 billion parameters (about 9.9GB in int4 quantization) that always pass through all inferences, such as attention layers and shared experts, are resident in RAM.

• Streaming section: Tens of thousands of individual experts (about 370GB) selected based on input are placed on high-speed storage such as NVMe SSDs and read only when needed.

This method made it possible to reduce effective memory consumption to about 10GB of resident portions. This system, which allows you to perform OS demand paging in-house, successfully replaces the wall of VRAM with a wall of inexpensive disk space.

Optimization through cash techniques and speculative decoding

To alleviate the bottleneck of read speed from storage, multiple advanced optimization technologies have been introduced. First, in addition to the “LRU (Least Recently Used) cache,” which stores the experts loaded in RAM, it also includes a “learning cache” that learns frequently called experts from usage history and prioritizes them in memory at the next startup. This allows for the property that the more specific interactions are repeated, the faster the reasoning speed becomes.

Additionally, speculative decoding using “MTP (Multi-Token Prediction)” is implemented. This system reduces the number of calculations by predicting and verifying future tokens in advance. However, even with these optimizations applied, SSD bandwidth still dominates the overall speed. As shown in the table below, performance varies greatly depending on device specifications.

Practical Challenges and the Future of the AI Inference Market

Limitations of processing speed and hardware requirements

This innovative approach comes with an unavoidable high cost. That is “extreme slowing.” Since a large amount of expert data must be read from the SSD for each token generated, PCs equipped with standard NVMe SSDs typically generate at about 0.05 to 0.1 tokens per second. This means it can take several minutes to tens of minutes to get a short reply, making it unsuitable for real-time chat.

To ensure practical speeds, extremely high disk bandwidth is required. For example, in the Apple M5 Max environment, which supports fast access at 14.2GB/s, there are reports of improvements to about one token per second. Additionally, many unresolved issues remain for production operation, such as storage wear and heat generated by readout, and verification of accuracy degradation caused by int4 quantization.

AI Democratization and Future Prospects After 2026

This trend of “running massive models in low-spec environments” is creating a new trend in the global AI inference market. The AI inference market is projected to grow from $117.8 billion in 2026 to $312.6 billion by 2034, with one of the driving factors being the expansion of “edge inference.” Lightweight C language-based engines like this offer a powerful choice for users who prioritize privacy protection and cost savings while reducing cloud dependence.

In the future, as ultra-fast PCIe 5.0-compatible SSDs become widespread and more advanced cache algorithms develop, it may become common for individuals to run frontier-class AI at practical speeds on their home PCs. This possibility, demonstrated by a simple code of about 1,300 lines, can be seen as the first step toward true “democratization,” reclaiming AI intelligence from some giant corporations to individual hands.

[#AI #機械学習 #推論エンジン #生成AI #テクノロジー #MoE #大規模言語モデル]