Good technology decisions start with a clear problem and a realistic view of the work involved. This guide sets out the questions we would bring into a discovery conversation, the tradeoffs to examine and the evidence worth collecting before committing to a delivery plan. The aim is practical clarity for Australian product teams and business owners.
Give the model something useful to read
Retrieval-augmented generation, or RAG, combines a search step with a generation step. Relevant material is retrieved from an approved collection and supplied as context for an answer. The approach can make internal knowledge easier to use, but does not guarantee correctness. The quality of your documents, permissions and retrieval process remains central to the outcome.
Consider a staff member asking which leave policy applies to a particular situation. The system needs the current policy, its scope and any relevant exception—not simply a passage containing the word “leave”. A source-backed answer should let that person inspect the policy themselves. RAG is useful when it makes the evidence easier to find; a fluent response without the right evidence is still a search failure.
Prepare the knowledge before the chatbot
Identify authoritative documents, remove stale duplicates and assign content owners. Preserve dates, titles and meaningful section boundaries during ingestion. A policy without an effective date or an owner is difficult for either a person or a model to interpret. Build a refresh process so a revised document replaces its older representation rather than silently existing beside it.
Build a small content register before ingestion. For each source, record its owner, intended audience, effective date and replacement process. Decide what should happen when a source is removed. Test whether links in old answers remain understandable after a document update. These are knowledge-management decisions that affect the reliability of the assistant just as directly as the choice of embedding model.
Retrieval is its own product problem
Queries rarely use the exact vocabulary of the source material. Test keyword, vector and hybrid retrieval approaches on representative questions. Consider how tables, diagrams and multi-step procedures survive text extraction. Measure whether the correct evidence appears in the retrieved set before adjusting the wording of the final answer. A better prompt cannot reliably rescue missing evidence.
Use failed retrieval examples to guide improvement. If the right passage is present but buried, ranking may need attention. If it never appears, examine extraction, chunk boundaries and query interpretation. If two policies conflict, the collection needs editorial work. Keeping these failure types separate prevents the team from changing prompts repeatedly when the underlying issue is elsewhere in the information pipeline.
Permissions must apply before generation
Limit retrieval according to the current user’s access, and carry those restrictions into caches and logs. Do not retrieve everything and ask the model to hide restricted passages afterwards. Test users with different roles and include attempts to infer confidential information indirectly. Permission changes should have a documented propagation time and an observable invalidation process.
Create test identities that represent realistic access levels. Ask the same question as a general employee, a manager and someone whose access has just been revoked. Check quoted passages, document titles and cached answers as well as the final response. A system can reveal sensitive information indirectly even when the main paragraph appears harmless. Record the expected behaviour before running the test.
Make sources and gaps visible
Provide links to the passages used in an answer and let the user inspect them. If evidence is insufficient, the assistant should say so or suggest a narrower question. Test questions whose answers are deliberately absent from the knowledge base. Confidence in the prose is not evidence that the answer is supported. A useful system knows when to stop.
Agree on a useful “not enough evidence” response with the people using the system. It might link to an authoritative collection, identify the missing detail or suggest a more precise question. It should not invent a policy to avoid leaving the user without an answer. Include those cases in demonstrations so stakeholders see the boundary as a designed capability rather than an embarrassing exception.
Maintain the system like any other service
Monitor ingestion failures, retrieval quality, response latency and permission issues. Keep a regression set for document, embedding and model changes. Give staff an escalation route when a generated answer conflicts with an authoritative source. RAG can reduce the effort of finding information, but ownership of the underlying knowledge remains an organisational responsibility.
Give the assistant an operational owner and a content owner; they may be different people. The first watches ingestion, access and availability. The second resolves stale or contradictory source material. A simple incident record should distinguish a missing document from an incorrect answer grounded in an existing document. That distinction helps the right team fix the cause and verify that the same question now behaves as intended.
Bring the right questions to the table
Before choosing a supplier or solution, write a short decision brief. Describe the outcome you need, the constraints you cannot change and the assumptions you can test. Ask your team to review it from the perspective of customers, operations and ongoing ownership. Differences in those perspectives are useful: they often reveal an overlooked requirement before it becomes expensive to change.
You do not need a perfect specification to begin a conversation. You do need a shared understanding of the next decision. BonsaiCo can help frame a discovery engagement, work through options and prepare a practical delivery plan. Start small, make the work visible and let evidence guide the next stage of growth.