RAG vs Fine-Tuning for a Business Assistant

TL;DR

RAG vs fine-tuning is mostly a question of knowledge versus behavior. RAG gives a model your documents at question time. Fine-tuning changes how the model writes or classifies.
For an assistant that answers questions about contracts, products or procedures, start with RAG: it handles change, citations and access rights.
Fine-tuning earns its place for a strict format, a stable tone or a narrow task with many labelled examples.
A hybrid is common in mature projects, but only after an evaluation shows what prompts and retrieval cannot fix.
Settle the debate with your own question set, not with general rules.

In this article

What each approach does
RAG vs fine-tuning: how to choose
Preparing documents, chunking and retrieval
Access rights, citations and fresh knowledge
Cost, maintenance and the hybrid approach
A worked example and how to evaluate
Risks and common mistakes
Where to start
FAQ

A team that wants an assistant for its own knowledge hits the same fork quickly. Should it use retrieval, or train the model? The RAG vs fine-tuning question looks technical, but the answer depends on business facts: how often your content changes, who may read what, and who maintains the system after launch.

This guide is written for practitioners and decision makers. It explains both approaches, gives a decision grid, and covers the unglamorous work that decides quality: document preparation, chunking, retrieval, citations and evaluation.

What each approach does

RAG in plain words

Retrieval-augmented generation keeps the language model unchanged. When a user asks a question, the system searches your documents, selects the most relevant passages and hands them to the model with the question. The model answers from those passages, and can point to them.

Think of an open-book exam. The model does not need to remember your policy. It needs to read the right page quickly.

Fine-tuning in plain words

Fine-tuning continues the training of a model on a set of examples, so that it adopts a pattern. The pattern can be a tone, an output format, a labelling scheme or a specialised phrasing of answers.

Think of coaching. You are shaping habits, not filing new reference material. Facts absorbed this way are hard to update, hard to cite and hard to remove.

The difference that matters

RAG changes what the model sees. Fine-tuning changes how the model behaves. Most disappointing projects come from using one to do the job of the other.

RAG vs fine-tuning: how to choose

Use this grid as a first filter. It is not a verdict, but it quickly shows which side a project leans toward.

Decision grid

Does your knowledge change often? If yes (prices, policies, product documentation), lean toward RAG. If it is stable and small, either can work.
Must answers cite a source? If yes, RAG. A fine-tuned model cannot reliably tell you where a statement came from.
Do users have different access rights? If yes, RAG with permission filtering. Training data cannot be filtered per user afterwards.
Is the problem a format, a tone or a classification? If yes, fine-tuning becomes a serious candidate.
Do you have hundreds of clean, labelled examples? Without them, fine-tuning is premature.
Who will maintain it? If the content owners are business teams, RAG lets them fix a document instead of requesting a retraining.

Side by side

Knowledge updates: RAG, edit or add a document and re-index. Fine-tuning, prepare data and retrain, then test again.

Source citations: RAG, natural, because the passages are known. Fine-tuning, not reliable.

Access control: RAG, enforced at retrieval time. Fine-tuning, not practical per user.

Style and format control: RAG, through instructions, with limits. Fine-tuning, strong when examples are consistent.

Error correction: RAG, fix the source or the retrieval step. Fine-tuning, add examples and retrain.

Effort at launch: RAG, document preparation and search quality. Fine-tuning, building a high-quality example set.

The pattern is clear. For knowledge, RAG. For behavior, fine-tuning. For a serious business assistant, you usually need the first and sometimes the second.

Preparing documents, chunking and retrieval

In a RAG project, the quality of the answers is mostly decided before the model is involved. A strong model fed with the wrong passages will answer confidently and wrongly.

Clean the sources first

Collect the documents the assistant is allowed to rely on, and remove what should not be there: outdated versions, drafts, duplicates, contradictory copies. Give each document an owner, a date and a status.

Convert files carefully. Tables, headers, footers and scanned pages often break during extraction. Check a sample by hand. Garbage in the index means garbage in the answers.

Chunking: cut where meaning cuts

Documents are split into chunks, and each chunk is indexed separately. Chunks that are too small lose context. Chunks that are too large dilute the match and waste the model's attention.

Split along the structure of the document: sections, clauses, procedure steps, questions in a FAQ.
Keep a little overlap so that a sentence is never cut away from the condition that qualifies it.
Attach metadata to every chunk: document title, section, version, language, audience, date.
Keep tables and numbered procedures whole when you can.

There is no universal chunk size. Test several strategies on your own questions and keep the one that retrieves the right passage most often.

Retrieval: search is the product

Semantic search finds passages with similar meaning, even with different words. Keyword search finds exact terms, such as a product code or a clause number. Business content needs both, so most good systems combine the two and then re-rank the candidates.

Add filters on metadata (language, product line, version) before the search. Then inspect what is retrieved for real questions. When an answer is wrong, the first question is always the same: was the right passage retrieved?

Planning a cloud migration?

See how our senior consultants assess, plan and run migrations to AWS or Azure, with a tested return path.

Explore our cloud consulting

Access rights, citations and fresh knowledge

Access rights

A company assistant that ignores permissions is a data leak waiting to happen. The rule is simple: the assistant may only retrieve what the asking user may open. Filtering must happen at retrieval time, using the same identity and groups as your source systems, not in the prompt.

Do not rely on an instruction like "do not reveal confidential documents". Instructions are not security. If a passage should not be seen, it must never reach the model for that user.

Citations

Every answer should show the passages it relied on, with the document name, the section and a link to open it. This lets a person verify in seconds, and it changes behavior: users trust the assistant where it is checkable and stop trusting it where it is not.

Design for the case where nothing relevant is found. The right output is "I could not find this in the documents", with a suggestion of who to ask. A refusal that is honest is worth more than a fluent guess.

Keeping knowledge fresh

With RAG, freshness is a pipeline question. Decide how a new or changed document reaches the index, how a removed document leaves it, and who is alerted when an ingestion fails. Version your sources so that an answer can be traced to the document state that produced it.

With fine-tuning, new knowledge means a new training cycle. That is acceptable for stable behavior. It is a poor fit for prices, policies and procedures.

Cost, maintenance and the hybrid approach

Cost and maintenance, qualitatively

Rather than quoting figures, compare where the effort goes.

RAG: effort is front-loaded in document cleaning and search quality, then continues as a light, steady routine of ingestion monitoring, evaluation runs and content fixes. Each question also costs a model call with extra context.
Fine-tuning: effort is front-loaded in building and reviewing examples, and recurs at every retraining. You also take on the job of re-testing after each change of the base model.

In both cases, the largest hidden cost is maintenance without an owner. Name one before launch.

The hybrid approach

Hybrid means RAG for knowledge, plus a light fine-tune for behavior. For example, retrieval supplies the passages, and the tuned model writes in a strict house format or follows a precise reply structure.

Go hybrid only when evaluation shows a behavior gap that clearer instructions and better examples in the prompt cannot close. Do not tune a model to compensate for weak retrieval. That hides the real defect and makes it more expensive to fix.

A worked example and how to evaluate

An illustration: a fictional software vendor

Take a fictional software vendor with a support team, a sales team and a large pile of documents: product guides, release notes, security answers for customers, and internal pricing rules. It wants one assistant that both teams can use.

The team first lists its questions. Support asks about configuration steps and known issues. Sales asks about plan limits and security answers. Pricing rules are visible to sales only.

Step one is RAG. Documents are cleaned, release notes are versioned, and each chunk carries product, version and audience metadata. Retrieval filters by audience, so support never sees internal pricing rules. Answers show their sources.

The first evaluation shows two things. Retrieval fails on questions that mix a product name with an older version. And answers are correct but written in many different formats, while support wants a fixed structure: symptom, cause, steps, link.

The fix for the first problem is on the search side: version metadata and better chunk boundaries. The second problem is behavior, so the team tests a small tuned model on a few hundred reviewed support replies. It compares it to a well-written prompt with examples. The decision is made on the evaluation, not on preference. If the prompt is good enough, they keep the simpler system.

This example is an illustration, not a client case. It shows the order of work: knowledge first, behavior second, evidence at each step.

Evaluating on your own question set

Build a set of real questions, with the expected answer and the document that supports it. Include easy questions, ambiguous ones, questions with no answer in the documents, and questions a user should not be allowed to get an answer to.

Then run each option on the same set and score the following.

Retrieval: was the right passage among those retrieved?
Faithfulness: does the answer stay within the passages?
Completeness: does it cover what a person needs?
Citations: are the sources correct and clickable?
Refusals: does it decline properly when the answer is not in the documents?
Permissions: does it stay silent on restricted content?

Have business experts grade a sample, then use an automated grader to scale, checking it against the human grades. Rerun the set after every change of documents, prompts, models or chunking. Evaluation is a regression test for your assistant.

Risks and common mistakes

The main risks

Hallucinations are fluent statements the sources do not support. Reduce them by retrieving well, instructing the model to answer only from the passages, requiring citations and allowing it to say it does not know. You will not reach zero, so keep a human in the loop where errors are costly.

Data leakage has several routes: a permission gap in retrieval, sensitive documents in the index that should never have been there, logs that store questions and answers without protection, and training data that contains secrets. Apply least privilege, review what is indexed, and treat logs as sensitive data.

Prompt injection is text inside a document that tries to give the model orders. Treat retrieved content as data, limit what the assistant can do on its own, and keep actions behind confirmation.

Common mistakes

Choosing fine-tuning to teach facts that will change next quarter.
Indexing everything without cleaning, then blaming the model.
Using one chunk size for every type of document.
Enforcing permissions in the prompt instead of at retrieval.
Skipping citations because "the answers look right".
Launching without an evaluation set, so that no one can tell if a change helped.
Having no owner for the content after launch.
Going hybrid to hide a retrieval problem.

Where to start

If you are weighing RAG vs fine-tuning today, do this in order.

Write 50 to 100 real questions, with expected answers and source documents.
List the documents, their owners and who may read them.
Build a simple RAG baseline with clean sources, sensible chunking, combined search and citations.
Run the evaluation and read the failures one by one.
Fix retrieval and documents first, then instructions.
Consider fine-tuning only for a behavior gap that remains.

A senior custom AI application consultant can compress these steps and help you avoid the usual dead ends. Our AI practice covers AI strategy to frame the use case, and custom AI applications to build and run the assistant, on a sound cloud foundation and with your data prepared for retrieval.

Request the free audit and talk to a consultant: book a discovery call. We will review your use case, your documents and your constraints, and tell you honestly which approach fits.

FAQ

Is RAG always cheaper than fine-tuning?

Not always, but it is usually the cheaper way to start, because you avoid building a large training set. RAG adds cost per question and an ingestion pipeline to maintain. Compare both on your own volumes and update frequency.

Can fine-tuning remove hallucinations?

No. It can make a model more consistent in format and tone, but it does not give the model a reliable memory of your documents. For grounded answers, retrieval and citations remain the main tools.

Can I use RAG with confidential documents?

Yes, if you design for it. Enforce permissions at retrieval, choose a deployment that meets your data requirements, protect logs, and review what is indexed. Confidentiality is an architecture decision, not a prompt.

How many documents do I need before RAG is worth it?

There is no threshold. If people regularly lose time searching for answers in documents, RAG can help. Quality and ownership of the content matter more than volume.

When should I move from RAG to a hybrid setup?

When your evaluation shows a stable behavior gap, such as an inconsistent format, that better prompts and examples cannot fix. Retrieval problems should be solved on the retrieval side first.

What does a custom AI application consultant do in such a project?

The consultant frames the use case, prepares the evaluation set, designs retrieval and permissions, builds the first version, and sets up the routine that keeps it reliable. The goal is an assistant your teams can trust and your own people can maintain.

RAG vs fine-tuning for a business assistant