AI Development for Regulated and Sensitive Environments
I'm Matt Ratcliffe, a software developer and architect in Ballarat, Victoria. I build AI features into systems that cannot afford to behave unpredictably — health services, platforms holding personal and payment data, and businesses where a confidently wrong answer has a consequence. The interesting engineering is rarely the model; it is everything around it.
The part most AI projects underestimate
Getting a language model to produce something impressive takes an afternoon. Getting one to behave acceptably every time, on real data, for users with different entitlements, is the actual project — and it is mostly conventional software engineering wearing a new hat. Retrieval, permissions, evaluation, observability and failure handling are where the time goes, and where AI projects quietly fail after a demo that went well.
The demo-to-production gap is wider here than in ordinary software, because the failure mode is different. A conventional system that breaks stops working, visibly. A language model that goes wrong keeps producing fluent, plausible output that nobody flags. That changes what you have to build: not just tests, but ways of knowing you are still correct once you are live.
AI over sensitive data
This is the intersection where most of my work sits, and where the risk is most commonly misjudged. Retrieval-augmented generation that ignores the requesting user's permissions will summarise records that user was never entitled to see — and it will do it politely, without an error, in a way no access log makes obvious. The model did not leak anything; the retrieval layer did, because it was written as a search index rather than as part of the authorisation model.
The same reasoning applies to what leaves your boundary, what is retained by a provider, what ends up in a prompt log, and which jurisdiction the inference happens in. Under the Australian Privacy Principles and the other frameworks these systems fall under, those are compliance decisions, not infrastructure preferences.
What I build
LLM application development
Features built on language models inside real products — extraction, summarisation, classification, drafting — with the retrieval and guardrails that make them dependable.
Retrieval over private data
RAG built as part of the authorisation model rather than beside it, so what a user can retrieve is exactly what they were already entitled to see.
Agents and automation
Multi-step agents where the task genuinely warrants one, with explicit boundaries on what they may do unattended and a full trail of what they did.
Evaluation & observability
Evaluation sets drawn from real cases, continuous scoring, and the instrumentation to notice when quality drifts after a prompt or model change.
AI in regulated environments
Data residency, retention, provider boundaries and audit — the questions that decide whether an AI feature is deployable in health or financial services.
Feasibility assessment
A short, honest read on whether AI is the right answer for a problem, what it would take, and what it would cost to run at your volume.
Agents, and when they are the wrong answer
Agents earn their complexity when a task genuinely requires deciding what to do next rather than following a known sequence. A great many problems presented as agent problems are workflows with a language model in one step, and they are cheaper, faster and far easier to debug built that way. I would rather tell you that at the start than bill you for discovering it.
Where agents are the right answer, the engineering questions are about boundaries: what the agent can do without asking, what it must escalate, how its actions are logged, and how you stop it. Those questions matter more in a regulated environment than anywhere else, and they are design decisions rather than prompt engineering.
Evaluation is the deliverable
If you cannot measure whether the system is behaving, you do not have a product — you have a demo that has not failed yet. Building an evaluation set from real cases, deciding what "correct" means for your domain, and running it continuously is usually the highest-value thing an AI project does, and the first thing dropped under schedule pressure. It is also what makes it safe to change a prompt or a model version later without guessing.
Working with me on AI
Engagements range from a short assessment — is this problem actually suited to AI, and what would it take — through to building and running the thing. I am equally willing to tell you a project is not worth doing, which given how much AI spend currently goes on unmeasured pilots is sometimes the most valuable output. Engagement models are on the freelance engagements page.
Writing on AI
- Building AI Products for Sensitive Data: Security Cannot Be Added Later
Why security in an AI product is an architectural decision from day one, not a pre-launch task.
- AI Agents Are More Useful Than Chatbots for Real Business Work
The difference between answering questions and completing work, and when each is appropriate.
- The Agent That Knew What Wasn't Its Call
Designing escalation boundaries — what an agent should refuse to decide on its own.
- Trust Needs an Action Trail
Why autonomous behaviour is only acceptable when it is fully reconstructable afterwards.
- Emulation Is Not Evidence
A model producing plausible output is not the same as a system that works.
- The Best AI Search Doesn't Just Find Information — It Restores Momentum
What retrieval is actually for in a working environment, and how to judge it.
- The Cheapest Context Is the One You Never Load
Context economics, and why retrieval discipline drives both quality and running cost.
- The Prototype Was Never The Spec
The demo-to-production gap, which is wider for AI features than almost anything else.
- Delegate The Goal, Not The Steps
How to frame work for a capable agent, and why step-by-step instruction wastes it.
- A Two-Person Team That Shipped 500 Pull Requests In 14 Days
What AI-assisted delivery actually looked like in practice, including what it did not solve.
Common questions
Should we build on a hosted model or run our own?
For most organisations, a hosted model from a major provider is the right starting point: the quality is higher, the operational burden is far lower, and enterprise agreements generally cover training exclusion and retention. Self-hosting becomes worth considering when data residency or classification genuinely prohibits sending content outside your boundary, or at volumes where inference cost dominates. It is a compliance and economics decision far more often than a technical one.
Can we use AI over patient or personal data?
Usually yes, but the design has to account for it rather than the policy forbidding it. The questions that matter are which data leaves your boundary, what the provider retains and whether it trains on it, where inference happens, what lands in prompt logs, and — most commonly missed — whether retrieval respects the requesting user's permissions. Get those right and the rest is ordinary engineering. Get retrieval wrong and you have built a very articulate data leak.
How do we know whether it is actually working?
By building an evaluation set from real cases before you launch, deciding what correct means for your domain, and scoring continuously. Without that, quality assessment is somebody's impression from the last few interactions, and you cannot safely change a prompt or upgrade a model. This is the single most common thing missing from AI projects that stall after a good demo.
Do we need agents, or would something simpler do?
Most tasks presented as agent problems are workflows with a language model in one step, and they are cheaper, faster and much easier to debug built that way. Agents earn their complexity when the sequence of steps genuinely cannot be known in advance. I will tell you which one you have before we start building.
What does an AI feature cost to run?
It depends on volume, context size and model choice, and it is worth modelling before you commit rather than after. Context discipline matters more than model price for most applications — the cheapest token is the one you never send. I include a running-cost estimate in feasibility work, because a feature that is unaffordable at scale is not a feature.
Are you an AI agency?
No. I am a software developer and architect who has been building systems for over twenty years and now does a lot of that work with AI in it. The distinction matters: most of what makes an AI feature succeed is conventional engineering — data modelling, access control, testing, observability — and that is the part I bring.
Thinking about AI over data you have to be careful with? Describe the problem — or read about high-risk data systems and healthcare integration, which is where most of these questions come up in practice.