On September 24, 2026, Full Stack Data Lab and Modal open-sourced “Quail,” an AI-SQL engine designed to accelerate large language model operations. The launch addresses growing challenges in database processing powered by LLMs, specifically long GPU idle times and escalating computational costs.

Key Highlights of “Quail,” Open-Sourced in September 2026

On September 24, 2026, Shreya Shankar and her team at Full Stack Data Lab, an R&D organization in data analytics, released “Quail”—an open-source AI-SQL engine developed in collaboration with Modal. The source code is publicly available on GitHub under the MIT License. AI-SQL is a technology that integrates natural language conditions into SQL, the traditional database querying language, enabling complex filtering and joining powered by the deep understanding capabilities of Large Language Models (LLMs). Previously, filtering rows based on specific conditions from text datasets required sending data to external LLM APIs via client libraries in iterative loops. With Quail, users can register data tables in Apache Arrow format within a Python environment and write custom AI functions directly inside SQL queries. This allows evaluation tasks to run in batch mode on local or cloud-based GPUs. For instance, in a movie review database, queries can evaluate multiple natural language conditions simultaneously—such as checking whether a review contains spoilers and whether it recommends the film—and efficiently retrieve only the matching record IDs. As shown in the processing architecture diagram below, Quail seamlessly unifies data pipelines and LLM inference.

Computational Challenges in Traditional LLM Inference and Natural Language SQL Processing

The demand for applying LLM-based evaluation to unstructured text stored in databases is surging. However, as dataset row counts and evaluation criteria increase, the required number of LLM inference calls explodes, leading to prohibitive execution times and ballooning operational costs. When using standard LLM inference engines like vLLM for AI-SQL workloads, two primary technical bottlenecks emerge. First, CPU-bound overhead in task management and execution scheduling introduces significant idle time on GPUs, leaving high-performance tensor cores underutilized. Second, engines often fail to effectively retain and reuse Key-Value (KV) caches generated during prompt processing. When KV caches are prematurely evicted from GPU memory during multi-step evaluations, subsequent filtering steps are forced to recompute identical text sequences, severely degrading inference throughput. Quail was specifically built to eliminate these computational inefficiencies and maximize GPU utilization for heavy-duty evaluation workloads.

Innovative Architecture and Processing Optimization Behind Quail

Unifying Query Planning and LLM Execution Control

Quail’s most defining architectural feature is its tight integration of the SQL logical query planner with the LLM inference engine, optimizing both components as a single cohesive unit. Traditional architectures decoupled query planning from inference, with SQL engines determining data retrieval orders before sequentially dispatching evaluation requests to an external LLM server. In contrast, Quail analyzes the query holistically to anticipate all required LLM calls upfront. This allows Quail to chunk data into batches and pipeline execution: while the GPU performs parallel tensor operations on the current batch, the CPU concurrently prepares the next batch in the background. This overlapping design minimizes CPU-GPU synchronization delays, keeping GPU utilization consistently high. Furthermore, for AI join operations across two datasets, Quail applies state-of-the-art structured attention techniques—such as SpecInfer and Hydragen—to share forward pass computation across overlapping comparison texts.

Enhancing Memory Efficiency via KV Cache Reuse and Continuous Batching

When evaluating multiple natural language conditions across thousands of text rows, managing High Bandwidth Memory (HBM) on GPUs becomes the paramount factor determining throughput. Quail incorporates a specialized cache management mechanism that strategically retains the KV caches generated when reading text rows, passing them directly to subsequent evaluation steps. For example, after evaluating the first condition (“Does the review mention the ending?”), Quail retains the intermediate KV states in memory and transitions directly to evaluating the second condition (“Does it recommend the movie?”), completely eliminating redundant text re-reading and re-computation. Unneeded caches from previous steps are instantly freed to allocate memory for upcoming batches. Similarly, for AI joins comparing one row against multiple candidates, Quail aggregates operations to prevent repetitive KV cache reloading. As illustrated in the cache management diagram below, memory waste is drastically minimized.

Performance Benchmarks and Future Outlook

Speedup Results and Validation on Real-World Benchmarks

To rigorously assess Quail’s performance, the development team evaluated the engine using “QUAIL-B,” a benchmark suite comprising 29 distinct queries across diverse domain datasets, including movie reviews, medical reports, legal documents, and AI agent execution logs. The test environment utilized a single NVIDIA H100 SXM GPU paired with Qwen3 4B FP8, a lightweight 4-billion-parameter model. In comparative experiments using 10% of the baseline dataset scale, Quail outperformed a tuned vLLM baseline on 27 out of 29 queries, achieving a geometric mean speedup of 1.84x. Furthermore, in a full-scale benchmark (“BIO-4”) analyzing 5,000 medical documents against 4,144 adverse event terms using two-stage joins and filters, vLLM required 6.84 hours, whereas Quail completed the execution in just 29.26 minutes—a staggering 14.04x speedup. During this run, Quail achieved an input processing throughput of 19.03 million tokens per second, translating to over 1 billion tokens per minute.

Current Limitations, Development Roadmap, and Key Takeaways

Despite its impressive achievements, benchmark testing also revealed clear limitations in Quail’s current implementation. On the “AGENT-1” query, which analyzes AI agent execution logs, vLLM outperformed Quail by 2.32x. This performance gap stems from vLLM’s automatic prefix caching feature, which detects and shares common text prefixes across different rows. While Quail excels at reusing KV caches across multiple conditions within the same row, it currently lacks support for cross-row prefix caching. The development team has prioritized implementing inter-row prefix caching as their top near-term objective. Future development plans also include expanding the library of supported AI-SQL operators, adding integration for additional open-source models, and broadening hardware support beyond NVIDIA GPUs. By blurring the boundary between database query processing and LLM inference, Quail’s unified approach stands out as a foundational technology for next-generation analytics on large-scale unstructured data.

Reference Pages

• [Full Stack Data Lab Official Blog] https://fsdatalab.github.io/blog/introducing-quail/

• [Quail GitHub Repository] https://github.com/fsdatalab/quail

[#Quail #AISQL #LLMAcceleration #Database #OpenSource #GenerativeAI #GPUOptimization]