Good technology decisions start with a clear problem and a realistic view of the work involved. This guide sets out the questions we would bring into a discovery conversation, the tradeoffs to examine and the evidence worth collecting before committing to a delivery plan. The aim is practical clarity for Australian product teams and business owners.
Choose one useful job
Begin with a repeated task that has a clear input, a reviewable output and an accountable owner. Drafting a summary of approved material is easier to evaluate than asking an assistant to run an entire department. Record how the task works today, including time spent checking the result. Your baseline should include review effort, because an impressive first draft can still create extra work.
A document-summary pilot is easier to reason about when the expected output is specific: five agreed fields, links to the supporting passages and a flag when a field is absent. Compare it with the existing process using the same documents. Record corrections as well as time saved. A system that is fast on easy examples but unreliable on the ordinary exceptions may not yet reduce the team’s total workload.
Define the boundaries before the prompt
Decide what information the system can access, what it may produce and which actions require human approval. Do not give a prototype broad access to production systems simply to make the demonstration convenient. Use a restricted test environment and representative, appropriately authorised data. Treat generated text as an untrusted output until it has passed the checks appropriate to the task.
Draw an action boundary around the pilot. Reading an approved document, drafting a response and sending that response are separate permissions. A useful first release may prepare work for a person to review while keeping execution outside the model’s control. That separation makes failure recovery simpler. It also helps colleagues understand exactly what they are trusting the system to do.
Build an evaluation set
Collect examples of ordinary requests, ambiguous cases and situations where the assistant should decline or ask a question. Write down what a good result looks like before selecting a model. Evaluate factual support, completeness, tone and the consequences of an incorrect answer. Keep a separate set of cases for regression checks when prompts, retrieval or model versions change.
Keep evaluation examples independent of prompt-writing examples where practical. Include a request with missing information, a contradictory source and content that attempts to redirect the assistant. Score the result against the expected behaviour, not against how persuasive it sounds. If a reviewer changes a score, record why. Over time, those decisions become a more useful quality guide than an ever-longer prompt.
Design for the reviewer
Show source material next to an answer and make uncertainty visible. Provide a simple way to correct an output and record why it was rejected. A review interface should help a person make a decision, not encourage them to approve a confident paragraph without looking. For consequential actions, separate preparation from execution and require explicit approval at the appropriate boundary.
Try the reviewer’s workflow without an AI model first. Give a colleague a draft, its source material and the proposed approval controls. Watch whether they can identify an unsupported statement and correct it efficiently. If the interface makes evidence hard to inspect, better model quality will not solve the whole problem. Improving review is part of building the product, rather than a task to add after the demonstration.
Plan for cost and reliability
Estimate cost per completed workflow, including retrieval, model calls, retries, monitoring and human review. Define timeouts and a fallback when the model service is unavailable. Cache only where permissions and freshness allow it. Track failure categories so the team can distinguish a retrieval problem from a poorly framed task or a genuinely unsuitable use case.
Test the failure path deliberately. Make a source unavailable, simulate a timeout and supply a document that cannot be read. The user should receive a clear explanation and a practical way to continue. Operations staff need enough information to distinguish those cases without retaining unnecessary document contents in logs. Assign a person who can pause the pilot if quality or privacy concerns arise.
Roll out in small steps
Start with a limited group and compare results to your baseline. Expand only when quality and operational ownership are clear. Publish a short guide explaining what the assistant is for, what it cannot do and how to report a problem. The strongest AI rollout often feels modest: one useful task done reliably, with enough evidence to decide what should come next.
A sensible rollout review asks three separate questions: is the output good enough, is the whole workflow more useful and can the organisation operate it responsibly? A positive answer to one does not settle the others. Keep the initial audience small enough that you can investigate mistakes in detail. Expand after the evidence supports a clear operational benefit, rather than because the demonstration generated enthusiasm.
Bring the right questions to the table
Before choosing a supplier or solution, write a short decision brief. Describe the outcome you need, the constraints you cannot change and the assumptions you can test. Ask your team to review it from the perspective of customers, operations and ongoing ownership. Differences in those perspectives are useful: they often reveal an overlooked requirement before it becomes expensive to change.
You do not need a perfect specification to begin a conversation. You do need a shared understanding of the next decision. BonsaiCo can help frame a discovery engagement, work through options and prepare a practical delivery plan. Start small, make the work visible and let evidence guide the next stage of growth.