A new technology called “Strata” has been unveiled that runs a large-scale AI model boasting 125 billion parameters at high speed on a home gaming PC. Behind the achievement of high-performance local inference without the need for the cloud, which breaks through the barrier of VRAM capacity, lies a thorough optimization of the memory hierarchy.
- Overview of Strata Release and Achievement of “GPT-4o-Class” Local Execution
- Why Does It Run on 12GB of VRAM? The Mechanism of MoE Architecture and Distributed Placement
- Measured performance across various hardware environments, from the RTX 3060 to multi-GPU setups
- Integration with Development Environments and Coding Agents and Application to Practical Work
- Strata Shatters the Conventional Wisdom That “Model Size = Required VRAM”
- Future Technological Evolution and Key Highlights of the Local LLM Ecosystem
Overview of Strata Release and Achievement of “GPT-4o-Class” Local Execution
The open-source inference engine “Strata”, released to the public on September 24, 2026, is attracting worldwide attention. It targets “Qwen3.8-Flash-Next”, an ultra-large language model with a total of 125 billion (125B) parameters developed by Alibaba. Traditionally, running a 125B-class model required data center-grade servers equipped with hundreds of gigabytes of VRAM or expensive specialized machines. However, with the advent of Strata, high-speed inference of 60 to 95 tokens per second has become possible on a PC equipped with standard consumer GPUs such as the GeForce RTX 3060 (12GB VRAM), RTX 5070 (12GB), and RTX 4090 (24GB), along with 64GB of main memory. Qwen3.8-Flash-Next is said to possess advanced reasoning, programming, and image analysis capabilities comparable to the flagship cloud AI “GPT-4o”. The impact of this running in a local home environment in real-time and without limitations is extremely significant. In actual operation footage, responses are generated smoothly without any cloud connection.

Why Does It Run on 12GB of VRAM? The Mechanism of MoE Architecture and Distributed Placement
The primary reason Strata was able to achieve overwhelmingly low VRAM usage lies in its architectural design, which maximizes the “MoE (Mixture of Experts)” structure adopted by Qwen3.8-Flash-Next. Although Qwen3.8-Flash-Next has a staggering 24,576 experts in total, the active parameters used during the generation of a single token are only 6B (6 billion), amounting to about 10 experts. While conventional inference engines attempt to load the entire model into VRAM, Strata rethought PC hardware resources as a multi-layered cache. It retains the constantly used attention layers and frequently accessed expert caches in VRAM, while keeping all remaining experts in the main memory (RAM) to process them in parallel with the CPU. Furthermore, it places a massive n-gram lookup table of approximately 28.8GB on the SSD, creating a mechanism to directly read only the few rows required for each token. Through this division of roles among the GPU, RAM, and SSD, combined with prefetching technology, bottlenecks are eliminated.

Field Testing Reveals the Reality of Performance and Practicality
Measured performance across various hardware environments, from the RTX 3060 to multi-GPU setups
According to actual device testing reports, Strata’s inference performance records high standards across various environments. In a configuration with a GeForce RTX 5070 (12GB), Ryzen 5 7600, and 64GB of memory, the 2-bit quantization model (Q2_0) recorded approximately 93 tokens per second in short conversations, and approximately 74 tokens per second even in long texts with a 128K context. In the higher-end RTX 4090 (24GB) environment, it maintained a high-speed output of approximately 80 tokens per second while deploying a model file exceeding 80GB. Furthermore, even with the entry-level RTX 3060 (12GB), it achieved a practical speed of approximately 44 tokens per second. In addition, Strata supports multi-GPU environments. For example, in a configuration combining two heterogeneous GPUs such as an RTX 5070 (12GB) and an RTX 5060 Ti (16GB), it was demonstrated that model layers can be automatically distributed and arranged, and by combining this with a method where experts reside permanently in main memory, delayed re-reading from the SSD can be avoided.
Integration with Development Environments and Coding Agents and Application to Practical Work
Strata is not merely a standalone chat app. Once launched, it can be immediately connected as a referenced model from local AI coding agents such as local “[Suspicious link removed] Code”. In a practical verification test where it was instructed to create a household survey data aggregation app, it automatically generated and completed debugging for a script that passed 11 custom-built test cases on the first try in just 196 seconds. For corporate client work handling confidential data, security constraints preventing data from being sent to external clouds have been a major barrier. However, by introducing Strata—which enables a high-performance 125B-class AI to run in a completely local, fully offline environment—a practical environment has been established to receive advanced summarization, rewriting, and programming assistance while strictly observing confidentiality obligations.
New Perspectives and Challenges in the Era of Local AI
Strata Shatters the Conventional Wisdom That “Model Size = Required VRAM”
In the conventional local AI community, the proportional relationship between the number of parameters and the required VRAM capacity was an absolute premise: “a 7B model requires 8GB VRAM,” “a 27B model requires 16GB VRAM,” and “70B or more requires a data center or Mac Studio.” Strata fundamentally upended this premise. Through a simple separation—”unused experts do not need to reside permanently in the fastest VRAM”—it forged a breakthrough purely via software architectural design. Currently, in the semiconductor market, accompanying the explosive surge in AI demand, prices for expensive GPU memories such as HBM and GDDR7 continue to soar. However, by combining the sparsity of MoE with the distributed placement of inference engines, a path has been illuminated enabling users to run massive AI models by fully utilizing the CPU, RAM, and SSD of their existing PCs without having to purchase additional expensive hardware. This breakthrough brought forth by computer science is decisively transforming the currents of local LLM development.

Future Technological Evolution and Key Highlights of the Local LLM Ecosystem
Since its release in late September 2026, Strata has been receiving rapid, almost daily updates. Although it initially focused primarily on NVIDIA CUDA environments and Windows/Linux, October 2026 saw a succession of announcements regarding support for Apple Silicon (Mac), support for cutting-edge quantization formats like NVFP4, and expanded compatibility with other models such as Qwen3.6-35B-A3B. Moving forward, the focus will be on mitigating intelligence degradation in harsher 2-bit quantization environments and optimizing for AMD Radeon and Intel Arc environments. Furthermore, it leaves room for additional speed enhancements, such as the evolution of Multi-Token Prediction (MTP) and an automatic learning feature for expert caches tailored to individual user conversation patterns. Freed from the pay-per-use billing and usage restrictions of cloud AI, the era of having a GPT-4o-class companion permanently resident in one’s personal production environment has finally become a reality.
Reference Page
- [Strata GitHub Repository] https://github.com/Niko1221/Strata
- 【Qwen3.8-Flash-Next Hugging Face】https://huggingface.co/Qwen/Qwen3.8-Flash-Next
【#Strata #LocalAI #Qwen38 #MoE #InferenceEngine #RTX3060 #OpenSource】


コメント