【Explanation】Introducing “Ollaya,” an app for running decision-making models locally

IT

In September 2026, the open-source application “Ollaya,” which allows decision-making models to be run directly in local environments, was released. Against the backdrop of information leakage risks associated with providing data to cloud-based AI and ballooning API costs, demand is growing to complete fast and clean decision-making processes entirely in-house.

Key points of events and the structure of decision-making models

Released on September 15, 2026, the decision-making model “Jev” has attracted significant attention for its ability to generate judgments on options and confidence scores with high speed and accuracy. Following this trend, its developer, Convai Innovations, has released “Ollaya,” an open-source framework specialized for the local execution of decision-making models. Ollaya is compatible with Windows, macOS, Linux, and Docker environments. Unlike traditional LLMs that sequentially generate text tokens one by one, its standout feature is the ability to make instantaneous judgments using only a single forward pass. For instance, given a customer inquiry text, it can batch-output structured data such as intent (refund request: 91%), urgency (yes: 92%), and sentiment analysis (anger level: 86%). Please refer to the conceptual diagram of its operation below.

Figure 1

Additionally, in a GeForce RTX 4090 environment, it achieves ultra-fast inference performance, completing the processing of 5 queries in just 8.9 milliseconds. The code has been released as open source under the Apache License Version 2.0, allowing anyone to deploy and use it in their own environment.

Differences in Processing Methods Between Traditional LLMs and Decision-Making Models

While general large language models (LLMs) excel at generating entire texts, they require anywhere from hundreds of milliseconds to several seconds for responses, resulting in heavy API usage fees and a high load on computational resources. In contrast, the decision-making models handled by Ollaya calculate outputs as type-defined probabilities or scores according to predefined question formats. Specific judgment formats available include “choice” for selecting a single optimal solution from multiple candidates, “score” for quantifying evaluation levels, and “noul” for making binary judgments. By skipping the wasteful generation of text, results can be obtained in a consistent format while keeping the consumption of computational resources to a minimum. At the API layer, it complies with TypeSafe specifications and natively provides endpoints such as /v1/systemone and /v1/models. It can be seamlessly invoked from existing client tools such as the official Python SDK (version 0.7.1) without any code modifications.

The technical advantages brought by “Ollaya” and a comparison with “Ollama”

Functional differences with the competing framework “Ollama”

“Ollama,” which has gained popularity as a local execution environment for LLMs, also began supporting decision-making models in version 0.35. However, while the models available in Ollama are limited to just three types—”nimble,” “tev1,” and “tev1:0.8b”—Ollaya has the advantage of being able to run a wider variety of high-quality open-weight models. Furthermore, there is a significant difference in the number of choices that can be configured in a single request. Ollama limits the maximum number of choices to 15, whereas Ollaya supports up to 255 choices, offering the flexibility to handle complex business classification tasks. Please refer to the feature comparison below.

Figure 2

As foundation models for Ollaya, the “Laya” and “Winnow” series are available, allowing users to choose from multilingual models, models equipped with automatic routing features, and more depending on their use case.

Data Privacy and Inference Performance in On-Premises Environments

In corporate AI deployment, security risks associated with sending customer personal information, unreleased data, and internal ticket details to external clouds are viewed as a serious concern. Ollaya is designed with data privacy as the highest priority, and sensitive data input is processed entirely within the internal network or local device. Since no data is transmitted to external servers whatsoever, the risk of information leakage can be structurally eliminated. Furthermore, it has been demonstrated that even in local PC environments equipped with mid-range GPUs such as the GeForce RTX 4070, latency is significantly reduced while maintaining accuracy equal to or greater than cloud-based APIs. This makes it possible to safely and cost-effectively build internal operational workflows—such as real-time-demanding customer support primary responses, automated incident triage, and severity (SEV) assessment.

Challenges and Future Outlook of Implementing Local LLMs and Decision-Making Models

Hardware Requirements and Limits of Accuracy Verification Faced During Implementation

To ensure the stable operation of AI models in a local environment, securing adequate computing resources is indispensable. The capacity of the GPU’s video memory (VRAM) is particularly important; it is recommended to select a GPU equipped with at least 12GB, and for full-scale practical operation, 16GB or more of VRAM (such as the NVIDIA RTX 4060 Ti 16GB or RTX 4090), or an Apple Silicon Mac with 32GB or more of unified memory. On the other hand, over-relying solely on a single accuracy rate when evaluating decision-making models harbors risks. In model validation experiments, cases have been reported where, even if the correct answer is selected, outputs that differ from expected values are obtained in metadata areas such as technical domain classification or requirement prioritization, and phenomena where logical inconsistencies arise between questions. In actual operation, rather than taking a single output at face value, a dual governance design is required, involving setting thresholds for confidence scores, checking consistency between questions, and routing low-confidence answers to human review.

Future Developments and Key Considerations in Enterprise Operations

Local execution environments for decision-making models, including Ollaya, are expected to grow into core components in the future development of enterprise AI systems. In particular, they are anticipated to be applied in areas such as infrastructure operations automation, initial alert triage, and security policy compliance checks. Within the development community, the standardization of container deployment manifests for various cloud infrastructures and Kubernetes environments is progressing, with full-scale adoption in enterprise environments expected to accelerate from 2026 onward. Going forward, the design of a hybrid AI architecture—coordinating text-generation AI for chat and RAG with decision-making AI that performs ultra-fast judgment and classification based on their respective roles—will become a critical focus that determines corporate competitiveness.

Reference Page

【#Ollaya #DecisionMakingModel #LocalLLM #AITechnology #OpenSource #Jev】

コメント

Copied title and URL