Research teams from Fujitsu and RIKEN have announced a new architecture called “PHOTON,” which dramatically improves the inference efficiency of large language models. It solves the memory shortage problem faced by traditional Transformers during long-context processing with a unique design that mimics the hierarchical structure of languages.

Japanese research team breaks through structural bottlenecks in LLMs

On December 25, 2025, a joint research team from Fujitsu Limited, RIKEN AIP Center, Tokyo University of Science, and Tokai University announced a new architecture called “PHOTON (Parallel Hierarchical Operation for TOp-down Networks),” which dramatically improves inference in large language models (LLMs). was published on the paper submission site arXiv. This research has attracted worldwide attention as a powerful Japanese solution to the biggest weakness of the currently mainstream Transformer-based models—namely, the structural challenge of excessive computational resource consumption as text lengthens.

Until now, AI development has aimed to improve performance by increasing the number of model parameters or enhancing hardware capabilities, but PHOTON adopts an approach by redesigning the entire processing architecture. The research team includes Yuma Ichikawa, Naoya Takagi, Takumi Nakagawa, Yuji Kanazawa, and Akira Sakai, representing a collaborative effort by Japan’s academic and industrial communities to deeply explore the fundamental theories of LLMs. The diagram below shows the overall picture of PHOTON.

The ‘memory wall’ faced by conventional Transformers

Currently, the Transformer architecture, widely used in conversational AI such as ChatGPT and Claude, faces a critical problem where inference costs increase sharply as text (context) lengthens. The paper describes this behavior as a “horizontal per-token scanner.” Traditional models need to read all past word information (KV cache) from memory for each new word generated and use it for calculations.

As text lengths double, then quadruple, this “memory for looking back” (KV) strains GPU memory, making data read/write speeds—that is, memory bandwidth—a bottleneck. As a result, the GPU cannot fully utilize its inherent computational power, resulting in a significant drop in inference speed. This memory issue posed a serious wall in both cost and performance, especially for summarizing lengthy documents exceeding 100,000 tokens and handling requests from many users simultaneously.

The concept of ‘hierarchical scanning’ inspired by the structure of natural language

The question the PHOTON development team posed was simple. “Is generation really necessary to horizontally scan a continuous history token by token?” When we read text, instead of memorizing and understanding each character one by one, we unconsciously layer and summarize information by turning letters into words, words into sentences, sentences into paragraphs, and so on. PHOTON proposed directly incorporating the hierarchical structure of this natural language into the model, replacing traditional horizontal scans with “vertical, multi-resolution contextual scans.”

This method does not keep past context as a flat list of data, but instead manages information hierarchically, from coarse-grained “summary” level to fine-grained “specific word” level information. This eliminates the need to constantly review every detail of the past, allowing you to refer only to the necessary hierarchical data when needed. This shift in thinking became the key to structurally resolving the linear increase in computational load and memory consumption that had been a constraint of traditional transformers.

A unique hierarchical mechanism that breaks through memory walls

Encoder/decoder hierarchy that separates summarization and detail generation

PHOTON’s architecture consists of two components with different roles. One is a bottom-up encoder that gradually compresses the input token sequence and consolidates it into low-rate, abstract “latent streams.” Here, a module called Context Chunker collects a certain number of tokens into a single “chunk,” and the Context Encoder handles the dependencies between those chunks.

The other is a top-down decoder that reconstructs concrete token representations from higher-level abstract states. The Context Converter unfolds instructions from higher-level layers as conditional prefixes, and the Context Decoder receives them to generate localized text. Notably, the scope of attention (attention) for the Context Decoder is strictly limited (Bounded) only to instructions within the same chunk and from higher levels. This ensures that no matter how long the entire text gets, the computational complexity of each step remains constant (O(1)), allowing different chunks to be processed in parallel, dramatically improving throughput. For details on how it works, please refer to the diagram below.

The innovation of “RecGen” that accelerates reasoning to the extreme

In implementing hierarchical models, it is inefficient to re-run encoders across all layers every time a new token is generated. To solve this problem, PHOTON introduced a generation method called Recursive Generation (RecGen). Once the prompt processing (prefill) is complete, RecGen leaves only the highest-level KV cache on the GPU and discards encoder information at lower levels.

When generating new tokens, the top-level stream is directly updated using intermediate representations restored on the decoder side. In other words, you can completely skip the bottom-up re-encoding process and proceed with inference. The paper has mathematically proven that under certain conditions, this method produces outputs equivalent to conventional generation methods. By adopting RecGen, the memory footprint of GPU-resident models is halved, and the load on memory bandwidth during long text generation is minimized. This “bottom-up skipping” is precisely the core technology that makes PHOTON incredibly fast and memory-efficient.

Phenomenal Performance and a New Wave in LLMs

Processing power per memory unit overwhelmingly surpasses existing models

Experiments using a model on a scale of 1.2 billion parameters by the research team recorded astonishing numbers. Compared to conventional Transformers, the metric “TPM (Throughput Per Memory),” which indicates the number of processed tokens per 1GB of memory, has improved by up to about 1,000 times (10 to the power of 3). In particular, in the “Decode-heavy” scenario with heavy generation processing, a vastly superior TPM of up to 1,856 times over the vanilla Transformer has been reported.

Looking at specific resource consumption, KV cache memory consumption can be reduced by up to 10 times, while throughput itself has also increased by about 40 to 45 times. This means that for cloud vendors and companies who previously felt limited by the number of concurrent users and the length of text handled by a single GPU, operational costs can be dramatically reduced. The true significance of this experimental result lies in solving memory problems that previously relied solely on the power of numbers to solve memory problems with GPU technology, as well as architectural intelligence. The results of the comparative experiment are shown in the graph below.

Future Developments and Expectations for Scale

The “vertical and hierarchical approach” demonstrated by PHOTON holds the potential to dramatically transform the future of LLMs. Currently, verification is conducted at a maximum parameter scale of 1.2B, but if this scales to current flagship models like 100B (100 billion) or 1T (1 trillion), the economics of AI will fundamentally change. There may be an era where there is no longer a need to line up GPU servers costing tens of millions of yen to run high-performance AI, and we may be able to handle the context of several books on edge devices like PCs or smartphones.

On the other hand, challenges remain. The corpus used for training is single, and compared to existing Transformers, slight compromises in accuracy are required (such as a slight increase in WikiText perplexity). However, the research team suggests that optimizing design parameters may help bridge this quality gap. If we can structurally solve the explosion of computational complexity that was a weakness of Transformers and continue to maintain scaling laws, PHOTON should become a textbook research that marks a historic paradigm shift in computer science.

[#生成AI #LLM #富士通 #理化学研究所 #アーキテクチャ #テクノロジー #推論高速化 #PHOTON]