A research team from Georgia Tech and Nvidia has developed “BOOST,” an integrated system that significantly accelerates large language model (LLM) inference by simultaneously accessing host memory and HBM. The background to this development involves a structural computational bottleneck where, amid a shortage of high-bandwidth memory (HBM) capacity on GPUs, the host memory bandwidth on the connected CPU side is left unutilized.

Key Points of the BOOST Announcement and Development Background

In September 2026, a research group from Georgia Tech and Nvidia announced “BOOST,” an innovative system that simultaneously accesses HBM inside the GPU and host memory on the CPU side during LLM inference processing (the Decode phase) to fully utilize the overall memory bandwidth of the system. In recent years, alongside the massive increase in LLM parameters and the extension of context lengths, AI data centers have been facing severe shortages of memory semiconductors and soaring costs. Particularly in systems such as NVIDIA Grace Hopper (GH200), while high-bandwidth HBM is equipped on the GPU, the host memory on the CPU side is also connected via a C2G (CPU-to-GPU) interconnect. The C2G read bandwidth in the GH200 is approximately 450 GB/s, which potentially offers additional bandwidth equivalent to about 9% to 11% (effectively about 10%) of the HBM bandwidth. However, conventional inference infrastructures either read data solely from HBM or perform hierarchical control to migrate (prefetch) data from host memory to HBM in advance, completely failing to utilize the direct read bandwidth of the host memory. As shown in the system overview diagram below, BOOST concurrently utilizes this dual memory hierarchy as peer resources.

• Developed by: Research team from Georgia Tech and Nvidia
• Integration Target: Open-source LLM inference infrastructure “vLLM” (v0.17.0)
• Target system: NVIDIA GH200 Grace Hopper Superchip, etc.
• Fundamental Issue: Bandwidth Saturation Due to HBM-Only Dependency and Neglect of Host Memory Bandwidth

The Bandwidth Attenuation Problem Caused by Conventional Data Migration Methods

The reason why conventional data migration methods (such as prefetching or on-demand paging) fail to achieve sufficient acceleration lies in the consumption of HBM write bandwidth. When model weights or the KV cache exceed the HBM capacity, traditional offloading techniques pre-transfer data from the host memory to a staging buffer on the HBM. However, the very act of writing data read from the host into the HBM consumes the HBM’s half-duplex interface bandwidth, thereby directly constricting the data read bandwidth demanded by the GPU’s arithmetic units (SMs). Mathematically, even with ideal prefetching, the available effective requested bandwidth decreases from $1+\alpha$ to $1-\alpha$ of the theoretical limit. For example, when the host bandwidth ratio $\alpha$ is 10%, the prefetching method incurs a massive bandwidth handicap of 22% compared to the single-access method. Furthermore, it has been confirmed that page-fault-driven dynamic migration, such as CUDA Unified Memory, causes thrashing in the HBM as a page cache, causing throughput to plummet by up to a factor of 5 (an 80% reduction). BOOST completely eliminates this redundant data movement, adopting an approach that reads data directly while keeping it resident.

• Drawbacks of prefetching: Causes writes to HBM, hindering SM read bandwidth.
• Constraint at 10% Bandwidth Ratio: Prefetching shows a 22% performance penalty for simultaneous accesses.
• Page Migration Breakdown: Unified Memory Throughput Plummets to One-Fifth Due to Thrashing

A novel concept enabling concurrent and proportional-bandwidth access (CAP)

The core concept proposed by BOOST is “CAP (Concurrent and Proportional Access).” This is a technology that simultaneously distributes access to HBM and host memory at precise ratios corresponding to their respective bandwidth ratios ($\alpha$) for each wave of thread blocks executed concurrently on the GPU. This makes it possible to extract the theoretical maximum bandwidth (1 + $\alpha$), which is the sum of the HBM bandwidth and host memory bandwidth, without incurring the copy overhead associated with data movement. However, modern GPUs adopt large 2MB pages to reduce the load on the TLB (Translation Lookaside Buffer), and the number of pages accessed by a single wave is limited to only a few dozen. With the random allocation method (BAPP-R) developed in the 4KB page era 10 years ago, large variations occurred in the access ratio per wave across a small number of 2MB pages, causing a concentration of accesses to the slower host memory and idle HBM, which led to drastic performance degradation. BOOST overcomes this challenge and achieves deterministic access distribution without modifying the kernel source code.

• CAP Access Principle: Simultaneous requests to HBM and host memory according to the bandwidth ratio (1/α)
• The Structural Wall of Modern GPUs: Adoption of 2MB Pages Causes Random Placement to Fail with a Small Number of Page Accesses
• The BOOST Solution: Establishing Deterministic Page-Data Mapping Without Kernel Modification

Technical Mechanism, Demonstrated Performance, and Future Prospects of BOOST

Wave-Cognitive Modulo Page Placement Optimizing Static Weights

BOOST applies “Modulo-Based Page Placement (MPP)” to static tensors of fixed size, such as model weight data. This is executed by the Page Allocation Manager (PAM). For example, when the bandwidth ratio between HBM and host memory is 10 to 1 (K=10 or 12), the first K pages of a contiguous virtual address space are placed in HBM, and the (K+1)-th page is alternately allocated to the host memory. As a result, even for short access waves spanning only a dozen or so 2MB pages, the access ratio to the host memory is always kept constant, completely eliminating the access bias specific to random placement. Furthermore, if a matrix multiplication kernel’s thread block (CTA) has a footprint that crosses page boundaries, it constructs “superpages” of a least common multiple size and allocates them in batches, thereby preventing a single thread block from stalling across multiple memory hierarchies.

• Role of PAM: Assigning modulo placement (MPP) to static weight tensors
• Modulo Placement (MPP): Place one page out of every K pages in host memory and fix the access ratio
• Superpage Technology: Preventing Boundary-Crossing Delays via Bulk Allocation Tailored to Thread Block Footprints

Data Shape Remapping for Controlling Dynamic KV Cache

In the decode phase of LLM inference, the Key-Value (KV) cache of the attention mechanism consumes the most memory and fluctuates dynamically. Because KV pages are repeatedly freed and reallocated as requests are generated and completed, static address allocation alone disrupts the access ratio per wave. To address this issue, BOOST introduces the Data Shape Remapper (DSR). Leveraging the block structure of PagedAttention, DSR rebuilds the axis order of the KV tensor to make the memory area continuous in units of attention heads, and defines two types of physical blocks: “all-local blocks,” which place all heads in HBM, and “single-head host blocks,” which place only one head on the host. When a new request occurs, DSR greedily allocates these two block types from the dynamic pool based on a proportional distribution rule (Greedy Policy), precisely controlling the host memory utilization across all running waves to within ±1% of the ideal value.

• Role of DSR: Real-time control of dynamically fluctuating KV cache page allocation
• Block Bisection: Combining All-Local Blocks and Single-Head Host Blocks
• Accuracy Guarantee: Maintain the ratio of active KV head chunks during execution within an error margin of ±1%.

Performance Verification Results and Impact on the Next-Generation AI Semiconductor Market

In performance evaluations on the NVIDIA GH200 Grace Hopper system, BOOST—built by adding approximately 800 lines of Python/PyTorch code to vLLM—demonstrated overwhelming results. In low-latency inference verification using the same batch size (iso-batch), whereas conventional prefetching worsened the time per output token (TPOT) by 6% due to HBM write contention, BOOST shortened TPOT by 4.3% compared to using HBM alone, achieving processing 11% faster than prefetching. Furthermore, in high-throughput inference that integrates and utilizes host memory capacity as a KV cache area, by automatically expanding all batch sizes, it recorded an average throughput improvement of 31%, outperforming prefetching by 15%. Stable effects have been confirmed across various models, including Llama 3.3-70B (a dense model) and Qwen 3-Next-80B (an MoE model). Looking ahead, in high-bandwidth systems such as the next-generation Vera architecture where the C2G bandwidth ratio ($\alpha$) reaches up to 44%, the throughput improvement by BOOST is estimated to reach up to 3x, which is expected to accelerate structural transformation in the semiconductor memory market.

• Low-latency inference (Iso-batch): Reduced TPOT by 4.3% (11% faster than prefetching)
• High-Throughput Inference: System-wide token generation throughput improved by an average of 31%
• Future Outlook: Predicting up to a 3x performance improvement in next-generation Vera and other systems with increased C2G bandwidth ratios

Reference Page

• 【BOOST: Concurrent Access to Host Memory and HBM to Accelerate LLM Inference】https://arxiv.org/abs/2609.13592v1

[#LLM #AISemiconductors #HBM #vLLM #GPU #MemoryBandwidth #Algorithms]