Greatest Native LLM for RAG: 15 Open Fashions Examined

“Our findings present that selecting an LLM for RAG will not be merely a query of mannequin measurement or reply high quality. A dependable enterprise assistant should retrieve the correct info, observe source-priority guidelines, and use instruments as meant. These behaviors have to be examined explicitly and strengthened by the system structure, slightly than left to the mannequin to resolve.”

— Vadzim Tashlikovich, Head of ERP Options Division, SCAND

The smallest mannequin that may summarize a doc will not be essentially the smallest mannequin you’ll be able to belief together with your firm data base. To seek out the most effective native LLM for RAG on actual firm paperwork, SCAND’s R&D group examined 15 open-weight mannequin configurations on fundamental operations: summarization, knowledge-base retrieval, instruction following, and spreadsheet calculations.

Whereas summarization is a comparatively easy job, answering questions over inside paperwork requires a mannequin to retrieve the correct info, observe source-selection directions, and supply dependable solutions grounded within the obtainable context.

In SCAND’s R&D exams, most open-weight LLMs dealt with fundamental doc duties, however solely two efficiently discovered info in inside paperwork whereas following the required source-selection directions.

The outcomes spotlight why selecting a mannequin for personal RAG requires greater than evaluating parameter counts or benchmark scores.

The Problem: Working AI on Non-public Firm Information

Enterprises need AI assistants that may reply questions on inside insurance policies, analyze enterprise paperwork, summarize reviews, and retrieve info from company data bases. But many organizations can’t ship confidential information to public AI APIs due to safety, privateness, or regulatory necessities.

Self-hosted, open-weight giant language fashions (LLMs) provide an alternate. They will run on infrastructure managed by the group, permitting corporations to retain better management over their information and AI workflows.

A extra complete deployment might require organizations to construct a personal LLM resolution round their very own information, infrastructure, and safety necessities. Relying on the use case, smaller on-device LLMs may be appropriate when native processing, diminished information switch, or restricted connectivity is a precedence.

Nonetheless, non-public deployment comes with a value. GPU reminiscence necessities, {hardware} availability, inference efficiency, and common upkeep all affect the overall price of possession. A bigger mannequin might provide stronger capabilities, however it could additionally make deployment far more costly.

This results in a sensible query for enterprise know-how leaders: what’s the smallest mannequin that may carry out on a regular basis duties over an organization data base with out compromising safety or accuracy?

At first look, the reply may appear easy. Most trendy LLMs can summarize textual content, extract info, and reply questions on paperwork they obtain immediately.

However a knowledge-base assistant should do greater than course of equipped textual content. It should resolve the place to search for info, choose the correct instruments, observe directions, and distinguish verified inside data from common data.

These are examples of low-agency AI duties: operations that contain restricted decision-making and a small variety of instrument calls slightly than lengthy, autonomous plans. Typical duties embody:

• Looking out inside paperwork and retrieving related info.
• Summarizing reviews and analyzing insurance policies.
• Evaluating tables and checking enterprise guidelines.
• Studying spreadsheets and calculating balances.
• Drafting directions and answering worker assist questions.
• Monitoring modifications and retrieving historic info.

These duties don’t require a totally autonomous AI agent. They do, nonetheless, require a mannequin that follows the appliance’s guidelines. The encompassing workflow is simply as vital because the mannequin itself, which is why AI agent growth entails designing instrument entry, directions, and safeguards alongside mannequin choice.

Our R&D group evaluated open-weight LLMs on consultant knowledge-base operations to know the place smaller fashions are ample, the place bigger fashions make a distinction, and which failure modes matter most in a personal enterprise setting.

One discovering stood out: the principle threat was not merely an incorrect reply. It was a mannequin that silently skipped the interior data base, relied on its common data, or tried to make use of an exterior search instrument when the duty required inside sources.

How We Examined the Fashions

We in contrast open-weight fashions utilizing the identical set of inside company paperwork, seven on a regular basis questions, and a unified infrastructure for brokers geared up with auxiliary instruments.

Every mannequin was instructed to prioritize inside paperwork. Questions have been posed a number of occasions to evaluate how constantly the fashions adhered to the required workflow. This issues as a result of two AI brokers with the identical LLM can provide completely different outcomes when their prompts, instruments, runtime settings, or surrounding workflows differ.

Our LLM tool-calling benchmark targeted on whether or not fashions might use obtainable instruments appropriately, retrieve proof from the right supply, and observe the appliance’s directions slightly than merely generate believable solutions.

The agent harness offered 5 capabilities:

• Inner knowledge-base search.
• File-system instructions.
• Web and Wikipedia search.
• Calculations.
• Excel file studying.

We grouped the seven questions into 4 classes.

Every query (a number of runs) was scored as a cross or fail, producing a most rating of seven. The analysis intentionally coated greater than the standard of the ultimate response.

We additionally examined the mannequin’s conduct: Did it use the suitable instrument? Did it retrieve proof from the right supply? Did it observe the standing instruction, or did it take a shortcut?

This distinction is vital. A solution will be factually right and nonetheless be unsuitable for an enterprise utility if it was obtained by way of an unauthorized workflow.

The analysis is a task-specific take a look at slightly than a common rating of each mannequin’s tool-use capabilities. A broader instrument calling benchmark LLM comparability would want to cowl further instruments, job varieties, and runtime configurations.

Fashions and {Hardware} Necessities

We evaluated fashions of various sizes and architectures, together with dense fashions and mixture-of-experts (MoE) fashions. We additionally in contrast completely different quantization settings for chosen fashions.

Quantization reduces the precision used to characterize mannequin weights, typically decreasing reminiscence necessities and making native deployment extra accessible. Nonetheless, it was decided that 4-bit quantization has the most effective stability between high quality and measurement.

When a mannequin struggles with domain-specific terminology or directions, adaptation methods resembling parameter-efficient fine-tuning (PEFT) may be value evaluating. Fantastic-tuning is a separate consideration from the exams reported right here, nonetheless, and ought to be assessed towards the identical task-specific standards as the unique mannequin.

The next desk summarizes the mannequin configurations and reminiscence necessities recorded throughout our exams.

Precise deployment necessities depend upon the inference runtime, context size, KV cache, concurrent requests, and {hardware} configuration.

The reminiscence comparability highlights an vital consideration: parameter depend alone doesn’t decide the sensible price of working a mannequin. Structure, quantization, energetic parameters, and implementation all matter.

For enterprise deployments, the related query will not be merely how giant a mannequin is. It’s how a lot {hardware} it must ship dependable efficiency on the meant workload.

Key Outcomes: Summarization Is Simple; Dependable Retrieval Is Not

The outcomes revealed a major distinction between processing info that’s already obtainable and deciding the place to retrieve info from.

Fourteen of the 15 fashions within the scored comparability handed each summarization questions. In distinction, solely two fashions handed each knowledge-base retrieval questions.

Discovering 1: Summarization Is Largely Solved for This Workload

Fourteen fashions handed each summarization questions. Even comparatively small fashions might condense equipped paperwork into helpful summaries.

This makes summarization a poor standalone benchmark for choosing a personal knowledge-base assistant. A mannequin can carry out nicely when the related textual content is positioned immediately in its context and nonetheless fail when it should retrieve that textual content independently.

For organizations that want solely fixed-prompt doc processing, a small mannequin could also be ample. For an assistant that should search throughout inside repositories, extra intensive analysis is critical.

Discovering 2: Information-Base Retrieval Is the Actual Dividing Line

Solely Qwen 3.8 27B at 4-bit quantization (small mannequin) and DeepSeek V4.1 Flash (medium-sized mannequin) handed each retrieval questions.

The opposite fashions steadily did not retrieve the anticipated info from inside paperwork, regardless of gaining access to a knowledge-base search instrument and being instructed to prioritize it.

That is the distinction between figuring out how one can reply a query and figuring out how one can receive the proof required to reply it.

In a company setting, an assistant should retrieve info from the right supply, not merely produce a believable response. A solution drawn from common coaching information could also be outdated, incomplete, or inconsistent with the corporate’s present insurance policies. A purpose-built non-public RAG chatbot for an inside data base illustrates the sort of workflow wherein reliable retrieval is central to the outcome.

Discovering 3: Multilingual Retrieval Can Work Throughout Language Boundaries

Qwen 3.8 27B efficiently retrieved details about Poland from a doc written in one other language.

For worldwide organizations, this functionality will be significantly worthwhile. Company data bases typically comprise paperwork created by completely different regional workplaces, native authorized groups, or enterprise items working in numerous languages.

Language protection, doc high quality, retrieval configuration, and the complexity of the query ought to all be included in a manufacturing analysis.

Discovering 4: Following Directions Was More durable Than Answering Questions

Eleven of the 15 fashions handed neither instruction-following query. No mannequin handed each instruction questions cleanly.

The fashions struggled once they have been anticipated to make use of inside sources even for questions they appeared able to answering from common data.

This exposes a limitation of counting on system prompts alone. A standing instruction resembling “All the time search inside paperwork first” might affect conduct, nevertheless it doesn’t assure that the mannequin will observe the required workflow each time.

For enterprise techniques, the appliance ought to implement essential source-selection guidelines wherever potential slightly than treating them as elective mannequin conduct.

Discovering 5: Spreadsheet Calculations Profit From Devoted Instruments

Ten of the 15 fashions reportedly calculated the spreadsheet stability accurately, typically by utilizing a calculator.

This can be a reminder that language fashions don’t must carry out each operation themselves. For arithmetic, spreadsheet processing, and different deterministic duties, specialised instruments will be extra dependable than asking a mannequin to purpose by way of uncooked values in textual content.

The mannequin’s function is to establish the related information and invoke the suitable operation. The instrument ought to carry out the calculation, and the appliance ought to validate the outcome the place accuracy is vital.

Discovering 6: Extra Parameters Do Not Assure Higher Outcomes

Qwen 3.8 27B at 4-bit quantization handed six of seven duties, whereas gpt-oss 120B (the golden trade normal) handed 4.

Equally, Gemma 4 26B MoE and Qwen 3.8 27B at 4-bit quantization had broadly comparable reported reminiscence necessities however considerably completely different scores.

The sensible lesson is easy: parameter depend will not be an alternative choice to task-specific analysis.

A mannequin with fewer parameters could be the better option if it follows directions extra reliably, makes use of instruments appropriately, and produces higher outcomes on the corporate’s precise workload.

Commerce-Offs: Accuracy, Privateness, and Infrastructure Value

The outcomes recommend 4 broad teams of fashions for the examined workload. These tiers assist slender down the most effective open supply LLM for RAG for a given workload, however they need to be handled as evaluation-specific steering slightly than a common rating.

These tiers describe efficiency in our analysis, not a common rating of mannequin high quality. When evaluating the most effective LLM fashions for RAG, the correct selection is dependent upon whether or not the appliance prioritizes retrieval, summarization, instrument use, or a mixture of duties.

Privateness Is an Utility-Stage Accountability

One of the crucial consequential findings issues exterior search. If a mannequin can name an web search instrument and chooses to make use of it for a query about inside firm information, the question might go away the group’s managed setting.

Relying on the search supplier, the question might reveal confidential enterprise context, buyer info, or particulars about inside operations. After all for manufacturing techniques, web search instruments have to be turned off.

For a corporation with strict data-handling necessities, that may be a compliance or safety incident even when the ultimate reply is right.

This threat will not be inherent to each open-weight mannequin or each RAG structure. It is dependent upon how the appliance, instruments, permissions, and community entry are configured.

The necessities will be particularly strict in regulated settings: for instance, non-public AI for clinics should account for the confidentiality of affected person info, whereas non-public AI for regulation companies should shield privileged communications and delicate case supplies.

A safer design makes the appliance chargeable for supply choice:

• Search the approved inside data base first.
• Prohibit or disable exterior seek for delicate workflows.
• Apply entry controls earlier than retrieving paperwork.
• Hold instrument permissions slender and auditable.
• Require specific authorization earlier than any motion that might disclose delicate info.
• Log retrieval selections and gear requires analysis and incident investigation.

If the appliance can’t discover ample proof internally, it ought to observe an outlined fallback coverage: ask for clarification, report that the data is unavailable, or request a licensed exterior lookup. It mustn’t silently substitute public info for inside proof.

Quantization and Configuration Matter

The identical Qwen 3.8 27B mannequin achieved six passes at 4-bit quantization and three at 8-bit quantization within the take a look at outcomes.

This can be a noteworthy distinction, nevertheless it shouldn’t be interpreted as proof that 4-bit quantization is inherently superior. Quantization alone doesn’t clarify the end result. Runtime settings, prompts, sampling parameters, instrument integration, and take a look at variability may affect outcomes.

The right method is to benchmark the precise configuration meant for manufacturing, together with its quantization technique, inference engine, context size, and agent harness.

{Hardware} Prices Can Outweigh the Advantages

Qwen 3.8 27B at 4-bit quantization was the smaller of the 2 top-scoring fashions on this analysis. DeepSeek V4.1 Flash achieved the identical whole rating however had a a lot larger reported reminiscence requirement.

That distinction issues when designing a personal AI deployment. A mannequin that requires a whole bunch of gigabytes of reminiscence could also be tough to justify for a workload involving doc retrieval and easy calculations.

Infrastructure selections must also account for inference latency, concurrent customers, context size, availability necessities, and prices. The smallest mannequin that passes the exams might not be the one viable choice, nevertheless it is a vital start line. And it’s worthy to say that for the price of DeepSeek infrastructure, you’ll be able to afford a number of Qwen fashions.

Suggestions: The best way to Select a Non-public LLM

Understanding how to decide on LLM for RAG begins with defining the duties the mannequin should carry out and the implications of failure. Quite than choosing a mannequin by its parameter depend, consider retrieval high quality, instruction following, instrument use, {hardware} necessities, and privateness controls collectively.

1. Begin With Qwen 3.8 27B at 4-Bit Quantization for Analysis

Of the examined configurations, Qwen 3.8 27B at 4-bit quantization supplied the strongest stability between the reported job rating and reminiscence necessities.

DeepSeek V4.1 Flash achieved the identical rating, however its reported reminiscence requirement was considerably larger. It could be value evaluating when its specific capabilities justify the infrastructure price.

gpt-oss 120B may be thought-about for workloads the place its strengths are related, however its retrieval and instruction-following conduct ought to be examined and the agent harness configured to match its capabilities.

These are beginning factors for analysis, not common manufacturing suggestions. For those who’re selecting LLM for RAG, benchmark the mannequin with your personal paperwork and deployment configuration earlier than making a closing choice.

2. Use Smaller Fashions for Easier Doc Operations

For summarization and chosen duties involving a clearly recognized doc, Granite 4.2 8B and Qwen 3.5 9B are affordable candidates to guage.

Smaller fashions could also be ample when the appliance provides the related textual content, limits the obtainable instruments, and doesn’t depend upon the mannequin independently navigating a big data base.

This will scale back infrastructure prices with out paying for capabilities the workload doesn’t want.

3. Consider the Greatest Native LLM for Instrument Calling

The very best native LLM for instrument calling will not be essentially the biggest or essentially the most correct on abnormal question-answering benchmarks. It should reliably choose the correct instrument, present legitimate inputs, interpret the instrument’s output, and observe the appliance’s guidelines.

In our exams, the strongest retrieval outcomes didn’t translate into excellent instruction following. This means that instrument choice and source-selection conduct ought to be evaluated individually from reply high quality.

For a manufacturing agent, contemplate limiting the obtainable instruments, validating arguments earlier than execution, checking instrument outputs, and recording every instrument name for auditing.

4. Implement Supply Choice within the Utility

Don’t rely completely on a mannequin’s promise to make use of inside paperwork. For delicate workflows, the appliance ought to management retrieval order, entry permissions, and exterior connectivity. It may be achieved by pressured graph AI-agent structure.

If exterior search is prohibited, the mannequin mustn’t have entry to that instrument within the first place. This architectural safeguard is extra reliable than a immediate alone.

5. Use Devoted Instruments for Calculations

For balances, monetary totals, spreadsheet comparisons, and different numerical operations, let a deterministic instrument carry out the calculation.

The mannequin ought to establish the related rows and columns, invoke the suitable operation, and clarify the outcome. The appliance ought to validate inputs and outputs when errors might have materials penalties.

6. Deal with Very Small Fashions Cautiously in Agent Workflows

On this take a look at, the smallest fashions struggled with instrument use, retrieval, or calculations. Fashions round 8B parameters and above have been typically extra helpful for doc operations, however measurement alone didn’t assure dependable agent conduct.

Quite than adopting a common parameter threshold, consider fashions towards the precise operations required by the appliance. A smaller mannequin with restricted obligations could also be preferable to a bigger mannequin with poorly managed entry to instruments.

7. Take a look at With Actual Firm Paperwork Earlier than Deployment

Public benchmarks and general-purpose evaluations can’t absolutely characterize an organization’s terminology, doc construction, permissions, languages, or insurance policies. A helpful analysis set ought to embody:

• Questions whose solutions are current in inside paperwork.
• Questions that require looking out throughout a number of information.
• Paperwork written in numerous languages.
• Questions the mannequin is prone to imagine it could reply from common data.
• Conflicting or outdated paperwork.
• Spreadsheet calculations and information extraction.
• Questions for which the right response is that the proof is inadequate.
• Makes an attempt to entry info the person will not be approved to see.

Consider not solely the ultimate reply but in addition the retrieval path, instrument calls, supply citations, access-control conduct, and consistency throughout repeated runs. This sort of validation is a vital step when transferring an AI pilot to manufacturing, as a result of a promising demonstration doesn’t by itself set up manufacturing readiness.

How SCAND Approaches Non-public LLM Choice

At SCAND, choosing an LLM for an enterprise resolution will not be merely a matter of selecting the biggest mannequin that matches the obtainable {hardware}.

By our non-public LLM growth companies, we consider candidate fashions towards the shopper’s paperwork, goal duties, and infrastructure constraints. For retrieval-driven purposes, this consists of testing the structure and retrieval pipeline as a part of RAG growth, not treating the language mannequin as the one issue that determines high quality.

The purpose is to establish a configuration that provides the correct stability of reply high quality, retrieval reliability, privateness, efficiency, and value. This analysis helps reply sensible questions earlier than manufacturing deployment:

• Can the mannequin retrieve the correct info from the shopper’s data base?
• Does it observe the appliance’s source-selection and safety guidelines?
• Can it deal with the shopper’s paperwork and terminology?
• Which duties require specialised instruments slightly than language-model reasoning?
• What {hardware} is required to satisfy efficiency and concurrency expectations?

These concerns additionally apply to specialised use instances, resembling a non-public LLM for supply code documentation, the place the assistant should retrieve correct technical info from inside repositories slightly than depend on generic solutions.

The outcomes then inform the structure, mannequin configuration, and safeguards wanted for the particular use case. Organizations contemplating an analogous method can run this analysis in your information to know which mannequin and deployment configuration greatest match their paperwork, safety necessities, and infrastructure.

Conclusion: The Smallest Mannequin That Works Is the One That Follows the Guidelines

Our analysis exhibits that summarization and dependable knowledge-base help are completely different capabilities.

Most examined fashions might summarize equipped textual content, however solely two achieved the strongest total rating of six out of seven duties. Neither handed each instruction-following questions cleanly. Even the strongest candidates subsequently require application-level safeguards and additional validation earlier than manufacturing use.

For enterprises, the purpose shouldn’t be to search out the smallest LLM that may produce a convincing reply. It ought to be to search out the smallest sensible configuration that retrieves the correct proof, follows the required workflow, makes use of instruments accurately, and respects information boundaries.

That’s the reason non-public LLM choice ought to start with the workload and the dangers, not the parameter depend. A fastidiously evaluated smaller mannequin, supported by a well-designed RAG structure, generally is a extra sensible enterprise resolution than a a lot bigger mannequin deployed with out sufficient controls.

The very best non-public AI assistant will not be the mannequin that is aware of essentially the most. It’s the one which is aware of the place to look, follows the foundations, and stays inside the boundaries of the system.

Ceaselessly Requested Questions (FAQs)

Which LLM is greatest for RAG?

In our exams, Qwen 3.8 27B at 4-bit quantization and DeepSeek V4.1 Flash achieved the very best whole rating, with six out of seven duties handed. Neither handed each instruction-following questions cleanly, so your best option is dependent upon the workload, {hardware}, and application-level safeguards.

What’s the greatest LLM for a RAG chatbot?

The very best LLM for RAG chatbot purposes is one which retrieves the correct proof constantly, follows source-selection guidelines, and works inside the obtainable {hardware} funds. Qwen 3.8 27B at 4-bit quantization is a helpful start line from our analysis, nevertheless it ought to be examined towards the chatbot’s precise paperwork and questions.

What’s the smallest LLM for RAG?

The smallest LLM for RAG is dependent upon the complexity of the paperwork, the retrieval pipeline, and whether or not the mannequin should use instruments independently. In our analysis, smaller fashions dealt with summarization higher than knowledge-base retrieval, so a compact mannequin may fit for fixed-prompt duties however require extra validation for interactive RAG. In our exams 27B+ fashions is the correct parameter measurement begin.

What’s the greatest native LLM for brokers or agentic duties?

The very best native LLM for brokers is dependent upon the instruments obtainable, the complexity of the workflow, and the way essential instruction following is. Our outcomes present that sturdy retrieval alone will not be sufficient: brokers additionally want managed instrument entry, clear source-selection guidelines, and analysis throughout repeated runs.

Can a smaller native mannequin outperform a bigger mannequin in RAG?

Sure. In our analysis, Qwen 3.8 27B at 4-bit quantization achieved a better rating than gpt-oss 120B whereas requiring considerably much less reported reminiscence. This doesn’t set up a common rating, nevertheless it demonstrates why actual job efficiency issues greater than parameter depend alone.

How do you cease a RAG assistant from sending queries to the web?

Take away the net search instrument from delicate workflows as a substitute of counting on a immediate. In our exams, fashions typically selected exterior search even when advised to make use of inside paperwork first. The appliance ought to management retrieval order, community entry, and gear permissions, and log each instrument name for assessment.