The Hao AI Lab at the University of California, San Diego (UCSD) has announced an innovative method called “JetSpec,” which accelerates the inference speed of large language models (LLMs) by up to 9.64 times. This technology is expected to play a crucial role in the AI agent society of 2026, where real-time capabilities are required, by resolving the trade-off between speed and accuracy faced by traditional inference methods through proprietary algorithms.

Breaking the limits of inference speed with Hao AI Lab’s discovery.

On June 24, 2026, Hao AI Lab, an AI research center at the University of California, San Diego, released a new framework called “JetSpec,” which dramatically improves the inference efficiency of LLMs. This method is a further evolution of the technology known as “speculative decoding,” achieving an astonishing 9.64 times faster performance on the mathematical inference benchmark MATH-500 compared to traditional sequential generation methods. As shown in the graph below, JetSpec delivers performance that far surpasses existing powerful acceleration methods like EAGLE-3 and DFlash.

The research team validated using the latest models such as the Qwen3-8B and achieved a 4.58x speed in MT-Bench, which measures complex interactional capabilities. Notably, the implementation using the NVIDIA B200 GPU records an overwhelming output speed of over 1,000 tokens per second. Hao AI Lab has made this technology open source, allowing access to code and models via GitHub and Hugging Face.

A dramatic shift toward reasoning in 2026 and its significance

The year 2026 is remembered as a landmark year in the history of AI computing, marked by an “inference flip” (inversion to inference). Until now, most AI computational resources have been spent training models, but by early 2026, inference workloads will account for about two-thirds of global computational load. Behind this is the evolution of AI from mere chatbots to “agent-type AI” that autonomously thinks and acts.

Agent-based AI requires extremely low latency in user interactions because it performs tool calls and iterative decision-making. Technologies like JetSpec are highly valued because even with the same hardware configuration, inference throughput can be increased several times over, directly reducing corporate infrastructure costs by tens of billions of yen. Thus, improving inference efficiency is no longer just a technical challenge—it has become the lifeline supporting the sustainability of AI businesses.

Why is it fast? How the innovative “parallel tree draft” works

The causal relationship and efficiency dilemma faced by conventional methods

To understand JetSpec’s core innovation, you first need to know the challenges of traditional “speculative decoding.” This technology operates in a small, fast “draft model” that creates drafts, which are then reviewed collectively by a large “target model,” creating a relationship similar to that of an assistant and master. However, previous methods faced the wall of the “causality and efficiency dilemma.”

Traditional autoregressive draft models have the drawback of high accuracy to account for causal relationships between words, but they have the drawback of slow generation because they need to create drafts one word at a time. On the other hand, the “block diffusion” method, which predicts multiple words at once, is fast but sometimes generates each word without regard for context, resulting in a higher probability of being rejected by the target model. JetSpec combined these two strengths and succeeded in generating multiple candidates in a tree-like structure while maintaining causality within a single computational path (forward pass). The diagram below shows the system configuration.

Solutions with the Tree Causal Attention Mask

The key to how JetSpec achieves “fast and highly accurate drafts” lies in its unique architecture called the “Tree Causal Attention Mask.” This technology restricts multiple word candidates in parallel while restricting each word to only refer to the information of its own “ancestor.” This made it possible to generate multiple branched contexts simultaneously and logically consistent in a single computational cycle.

Furthermore, by combining “Feature Fusion” technology, which injects the hidden states of the target model into the draft layer, the prediction accuracy of draft models is maximized. This design allowed us to increase the number of words (budget) to be proposed at one time without lowering the acceptance rate (acceptance rate). As a result, even complex tasks like mathematics or programming, where a single mistake could ruin the entire work, have achieved dramatic acceleration while maintaining high reliability.

Social Implementation and Ripple Effects on Next-Generation Hardware

vLLM integration and remarkable scaling for large-scale budgets

One of the major factors proving JetSpec’s practicality is its integration into the industry-standard inference engine “vLLM.” The research team developed custom GPU kernels (such as SM90 Paged FlashAttention) to build a mechanism to validate tree structure while minimizing memory usage and computational costs. This has shown that stable performance improvements can be achieved even under server load (batch size) close to actual commercial environments.

Experimental data shows that even when JetSpec increases the budget for proposing 256 words at once, it continues to improve without saturation. This contrasts with previous models like the EAGLE-3, where increasing the budget actually reduced speed due to overhead. The graph below shows changes in throughput when the draft budget was expanded, confirming the flexibility to adapt from small to large systems.

The future of an inference-focused society accelerated by NVIDIA Rubin and AMD MI400

The acceleration brought by JetSpec will be key to maximizing the performance of next-generation AI chips to be rolled out sequentially from the latter half of 2026. NVIDIA’s “Rubin” platform is equipped with HBM4 memory and aims to reduce inference costs to one-tenth of that of the conventional Blackwell generation, but with software-side optimizations like JetSpec, further cost reductions are possible. Similarly, the shift toward inference-focused hardware such as AMD’s MI400 series and Google’s Ironwood (TPU v7) is accelerating.

A key point of interest going forward is that such “ultra-fast inference” will also be applied to personal devices (Edge AI). In 2026, demand for cloud-free on-device inference is surging from privacy and latency perspectives, and JetSpec’s low-cost, high-efficiency algorithms have the potential to dramatically change how AI agents behave on smartphones and PCs. As AI becomes so fast that it hardly feels like “time to think,” human-AI interactions will evolve to a new level.

[#JetSpec #投機的デコーディング #AI推論高速化 #Qwen3 #vLLM #次世代ハードウェア #AIエージェント]