In April 2026, Chinese AI company DeepSeek adopted “OPD (On-Policy Distillation)” in its latest model, the “V4” series, achieving remarkable cost performance. Around the same time, a new method called “ExOPD,” which enables students to surpass teachers, was announced, marking a major shift in the AI development paradigm from “model giggerization” to “refining distillation algorithms.”

DeepSeek V4 Demonstrates the Implementation and Remarkable Achievements of ‘OPD’

On the third Thursday of April 2026, a historic turning point in the AI industry was likened to the ‘Yalta Conference.’ On this day, China’s DeepSeek released its latest “V4” series, revealing that it has introduced its proprietary “OPD Distillation Policy” as the core of post-learning. Despite its massive scale of 1.6 trillion parameters, DeepSeek V4 Pro succeeds in “losslessly” fusing knowledge from expert models in mathematics and programming through inverse KL divergence without relying on the traditional costly RLHF (Reinforcement Learning with Human Feedback).

The results of this approach are extremely concrete. In the global ranking of the competitive programming platform Codeforces, DeepSeek V4 Pro ranked 23rd worldwide. This is the first time an open-source model has stood shoulder to shoulder with top-tier closed models like GPT-5.4 in this ranking. Moreover, this algorithmic innovation has led to dramatic price reductions. The output price of the V4 Pro is set at 24 RMB (about 560 yen) per 1 million tokens, while the simultaneously announced V4 Flash offers a disruptive cost performance of 2 RMB (about 47 yen). The chart below shows the correlation between the performance and price of the latest model.

The impact of the new method “ExOPD,” which surpasses the performance of teacher models

An even greater shock to the world of knowledge distillation was the publication in March 2026 by a research team from Renmin University of China and Tencent, “ExOPD (Generalized on Policy Distillation with Compensation Extrapolation).” Traditional knowledge distillation has been considered to “transfer knowledge from large teacher models to smaller student models,” and it has been considered fundamentally difficult for students to surpass teachers. However, this study overturned conventional wisdom with a very simple idea of adjusting the scaling coefficient (λ) of the reward function.

The research team introduced ‘reward extrapolation,’ which sets the reward scaling coefficient λ greater than 1. This enables the student model to not simply trace the teacher model’s responses but to further amplify the “directions for improvement” the teacher has gained through learning. Experiments revealed that the setting λ=1.25 was the sweet spot, achieving a “teacher surpass” by consistently outperforming the teacher model in benchmarks for mathematical reasoning and code generation. This can be said to be a discovery that fundamentally changes the conventional wisdom of post-training (post-learning).

Mechanism and theoretical background of on-policy distillation (OPD)

Innovation in On-Policy Distillation (OPD) Breaking Traditional Methods

To understand why on-policy distillation (OPD) is receiving so much attention, it is necessary to know how it differs from traditional “off-policy distillation.” Until now, distillation typically involved student models learning data (templates) generated in advance by teacher models. However, this method had the drawbacks of the student model’s own writing habits during tests and its inability to learn from its own behavior. In other words, it’s like watching a teacher ride a bicycle and learning from it, but you can’t correct your feelings when you actually ride and fall.

In contrast, on-policy distillation takes an approach where the teacher model provides immediate instruction to responses currently being output by the student model. Every time a student advances one or several tokens, the teacher checks the deviation in the probability distribution (Logits) and immediately provides feedback to advance learning. This real-time educational process allows students to receive accurate corrections on their own path, dramatically improving learning efficiency. The following diagram explains the mechanism of student-teacher interaction in OPD.

The Power of ‘Reward Extrapolation’ Born from Redefining Reinforcement Learning

The effectiveness of on-policy distillation (OPD) goes far beyond mere “teaching methods.” Recent research has theoretically proven that OPD is a special case of “dense KL-constrained reinforcement learning (RL).” Specifically, it was clarified that the log-probability ratios of the teacher model and the reference model function as an “implicit reward function.” Based on this discovery, the aforementioned “G-OPD” framework was proposed.

The greatest benefit of this theory is that it allows for the freedom to change the weight of rewards. If λ is less than 1, students behave as an intermediate between the teacher and the base model, known as ‘reward interpolation,’ but if λ is greater than 1, it evolves into ‘reward extrapolation (ExOPD),’ where the teacher’s knowledge is emphasized. In this process, the student model tends to have longer responses and increased entropy, suggesting it may be able to explore richer reasoning paths. This was the moment when the definition of distillation was rewritten from mere imitation to “evolution” that understood the teacher’s intent.

Paradigm Shift in AI Development and Its Significance for Practical Application

The discovery of ‘reverse distillation’ that overturns parameter size supremacy

Until now, in the AI industry, the “scaling law” has been regarded as absolute, meaning model performance depends on the number of parameters and computational resources. However, the latest papers on on-policy distillation pose a strong counter to this paradigm. In the “reverse distillation” experiment conducted by the authors, a paradoxical comparison was deliberately made under the premise of making the student model smarter (or larger) than the teacher model.

The experiment showed that while even smart students struggled to learn the teacher’s habits with conventional distillation methods, on-policy distillation allowed them to smoothly learn teacher behavior in a short period. This fact suggests that the failure of distillation was not due to a lack of student ability, but rather a purely ‘teaching method (algorithm)’ issue. In other words, with the right teaching methods, it opens up the possibility of achieving high intelligence even with limited resources. This has become a beacon of hope for latecomers and resource-poor companies in today’s market, where monopolies on computing resources set the rules.

Practical value in integrating knowledge from multiple domains into one

From a practical standpoint, the most noteworthy aspect is that ExOPD has achieved outstanding results in ‘multi-teacher distillation.’ In current LLM development, it is common to create specialized models for each domain, such as mathematics, code, and dialogue, but a major challenge was that the performance of each domain degraded during the integration stage. In traditional SFTs and standard OPDs, it was not uncommon for integrated performance to fall below each teacher model.

However, by using ExOPD, it was demonstrated that a unified model of “surpassing all teachers” in both mathematics and coding domains could consistently surpass their respective specialized teacher models. For example, in average accuracy for mathematical reasoning, the domain teacher achieved 46.0%, while models integrated with ExOPD recorded 47.7%. This enables the provision of highly high-performance single models in production environments (actual services) that are specialized for specific tasks without losing versatility.

Future developments and key points of interest in intellectual property protection

Changes in industrial structure brought about by cost reduction and algorithmic efficiency

The spread of on-policy distillation is forcing structural changes in the business models of the AI industry. As DeepSeek V4 demonstrated, by refining algorithm efficiency, it has shown a path to delivering top-level intelligence while reducing dependence on expensive hardware resources (such as NVIDIA’s GPUs). While “computational resource monopolists” like OpenAI, which seek profits by aggressively raising API prices, DeepSeek is launching a “structural sniping” by insisting that the value of foundational models should be determined by algorithm efficiency.

Going forward, there will likely be a polarization between those who pay high costs to use “defined top productivity” and those who use the popular-priced intelligence pioneered by DeepSeek and others. As a trend for 2026, AI is shifting from a standalone performance race to a phase focused on integration into business processes and cost efficiency—an “operability” phase. Whether you can still stay at the intelligence table when your computational resources run out of cards will determine the survival of future players.

Accelerating technological development and new conflicts over intellectual property protection

On the other hand, technological advances also generate intense friction. This is because ‘distillation attacks,’ which acquire large amounts of output from other companies’ high-performance models and use them to train their own models, have become a serious problem. In April 2026, Anthropic accused Qwen, operated by China’s Alibaba, of conducting the “largest distillation attack ever” to illegally acquire Claude’s capabilities. It is said that about 25,000 fraudulent accounts were used, with approximately 28.8 million exchanges.

In response, OpenAI, Anthropic, and Google have joined forces through the industry group Frontier Model Forum (FMF) to detect and prevent such “hostile distillation.” Current laws (such as the Copyright Act and the Unfair Competition Prevention Act) have limitations in regulating training using AI outputs, so for the time being, the main approach is to treat it as a breach of contract under Terms of Use (ToS). AI development has now entered a new era of defense: not only about “how to improve performance,” but also about “how to protect your own technology (intelligence).”

[#知識蒸留 #OPD #DeepSeek #ExOPD #AI研究 #強化学習 #生成AI #知的財産]