“Magnitude,” an open-source LLM inference engine optimized for AI agents, has been released, demonstrating up to a 2x speedup compared to conventional technologies. This comes amid a growing need to automatically optimize long-running and increasingly complex agent processes to match the hardware characteristics of local PCs.

Challenges in LLM Inference and the Background of Magnitude’s Emergence

In September 2026, Magnitude, an open-source large language model inference engine for AI agents, was released. Traditionally, general-purpose engines such as llama.cpp and Ollama have been widely used to run LLMs in local environments. However, in AI agent processing—which involves repeated file operations and command execution—conversations become extremely long, and the simultaneous, parallel execution of multiple agents occurs frequently. As a result, conventional inference engines have faced bottlenecks in terms of computational efficiency and memory usage. Furthermore, data center inference engines are designed for high-volume batch processing, which limits their ability to be optimized for individual endpoint environments. To solve these challenges, the development team adopted a new approach that directly optimizes computational processing for device-specific hardware structures. Please refer to the figure below for an overview of the architecture of this technology, which supports the utilization of open-weight models.

This engine is provided completely free of charge under the Apache License 2.0 and is rapidly gaining attention as a new standard in local AI operations.

On-Device Automatic Compilation and Optimization Mechanisms

The defining feature of Magnitude lies in its mechanism of automatically compiling kernels (the minimum unit of computation) directly on the device running the model and dynamically tuning the parameters. Conventional inference engines have used pre-compiled, general-purpose kernels in order to operate across various environments. However, even with the same Apple Silicon or NVIDIA GPU, the memory bandwidth and compute unit configuration differ for each model variant. Magnitude detects the hardware characteristics of the PC actually running the model at startup, and generates and adjusts kernel code best suited to that chip. Furthermore, when selecting a model, it is equipped with a feature that suggests an execution configuration combining the optimal quantization precision and context length for the user’s PC, based on proprietary metrics such as “Speed” and “Intelligence.” This optimization process makes it possible to push the device’s potential performance to its absolute limits without requiring complex manual parameter configurations.

Performance Comparison and Memory Management Innovation

Comparison of generation speed and processing capacity with conventional engines

In benchmark measurements, the 4-bit quantized Qwen 3.6 35B A3B model was executed with a context length of 64K tokens, and a comparative verification against llama.cpp was conducted. In the Metal environment on a Mac equipped with an M4 Pro and 48GB of memory, the token generation speed reached 57 tokens per second for Magnitude compared to 30 tokens per second for llama.cpp, achieving a significant speedup of approximately 92%. The prefill speed for processing input text also improved by about 9%, from 466 tokens per second to 507 tokens per second. Furthermore, in the CUDA environment powered by NVIDIA DGX Spark, the generation speed increased from 49 tokens per second to 58 tokens per second (an increase of about 19%), and the prefill speed improved from 2,033 tokens per second to 2,507 tokens per second (an increase of about 23%). Please refer to the figure below for detailed speed improvement rates and processing performance comparison data in each environment.

These measured values indicate that the responsiveness of AI agents on a local PC is dramatically improved.

Dynamic Memory Release and Cache Sharing Supporting Multi-Agent Operations

When operating AI agents for extended periods or running multiple agents in parallel, curbing memory consumption is essential. Magnitude has successfully reduced memory consumption per agent by approximately 28% in Metal environments and about 27% in CUDA environments. To achieve this, it features a management mechanism that allocates only the minimum memory area required to hold weights during model initialization and dynamically expands the area as dialogue sessions grow. Because the allocated area is released immediately when an agent stops processing, it is easy to perform other tasks concurrently on the same PC. Furthermore, it adopts a technology that globally shares a “prefix cache,” which retains the calculation results of system prompts and tool definitions shared across multiple sessions. This eliminates the waste of repeatedly processing agent-specific long instructions, significantly enhancing the operational efficiency of the entire system.

Supported Environments and Deployment into Practical Ecosystems

Broad hardware support and an easy deployment process

Magnitude offers flexible compatibility with a wide range of operating environments, supporting macOS, Windows, and Linux. In terms of hardware, it runs on Apple Silicon, NVIDIA GPUs, and AMD GPUs, as well as CPU-only configurations without a dedicated GPU. Installation is completed via a desktop application or the included CLI tool, eliminating the need for complex environment setup. Once a model is downloaded, it operates completely offline without an internet connection, and all sent files and prompt information are kept entirely on the local device, ensuring a high level of privacy protection. Furthermore, it can connect directly with major AI agent tools such as Pi, OpenCode, Hermes, Codex, Claude Code, and Cline, and is also accessible from various external applications via an OpenAI-compatible API. For the connection structure of the entire ecosystem, please refer to the conceptual diagram below.

Future Feature Expansion and Impact on the AI Agent Society

Going forward, the development team plans to develop new features that further push the execution limits of large-scale models. At the core of this is “Expert streaming” technology, which places expert regions of MoE (Mixture of Experts) models that do not fit into GPU memory into RAM or storage, dynamically loading them as needed. Additionally, plans are underway to develop a dedicated compiler that automatically determines the kernel structure itself, and to promote composite device optimization that utilizes CPU, GPU, RAM, and storage as an integrated unit. With LLM inference costs projected to be drastically reduced toward 2030, establishing an environment where high-performance autonomous AI agents can operate constantly, quickly, and securely on personal devices will be the key to dramatically enhancing personal productivity. Magnitude’s efforts to reduce cloud dependency and maximize computational efficiency in local environments can be called a major turning point that will significantly accelerate the democratization and practical application of AI technology.

Reference page

• [Magnitude Official Repository (GitHub)] https://github.com/magnitudedev/magnitude
• [Magnitude Official Website] https://magnitude.dev/
• [Hacker News Announcement Thread] https://news.ycombinator.com/item?id=49911995

【#Magnitude #LLMInference #AIAgents #LocalAI #OpenSource】