IanCSS
← All knowledgeAI in software

Where AI belongs in your software, and where it doesn't

Most useful AI features are one small model step inside an otherwise ordinary process. A practical guide for automation and reporting.

Ian Ketelings · October 5, 2026 · 4 min read

A lot of AI projects start with “let’s add a chatbot”. The ones that keep earning their place usually start somewhere duller, like “someone spends two days a month copying figures out of PDFs”.

There’s a simple way to decide where a language model fits.

Use code for whatever must be exact. Use a model for whatever is fuzzy.

Code is perfect at adding up an invoice and hopeless at reading a badly worded email. A language model is the reverse. Good AI features put each one where it’s strong.

The shape that works

Most reliable features look like an ordinary pipeline with one narrow model step in it:

  1. Collect the input with normal code.
  2. Ask the model one small, specific question, and require a structured answer.
  3. Validate that answer with normal code: is the format right, do the totals match, does this customer exist?
  4. Act if everything checks out, or send it to a person if it doesn’t.

The model never touches your database directly. It proposes, and your code decides. If you replace the model next year, the pipeline stays.

Automation: earn the right to remove the human

Take incoming supplier invoices. The model reads each document and returns the supplier, date, line items and total as structured data. Then code takes over: does the supplier exist, do the line items add up to the total, does it match a purchase order?

Roll it out in stages:

  1. Shadow mode. It runs alongside the existing process, and nobody acts on the output. You simply compare.
  2. Suggestion mode. People see a filled-in form and approve or correct it. Every correction is free test data.
  3. Automatic, with exceptions. Cases that pass every check go straight through. Everything else still goes to a person.

Move to the next stage because the measurements say so, and not because the demo looked good.

Reporting: numbers from SQL, words from the model

Language models are tempting for reports and dangerous for arithmetic. The fix is a strict division of labour. Your database calculates every figure, including changes and percentages. The model only writes the sentences around them.

And because the numbers came from code, you can verify the text afterwards:

// Every figure, including the changes, comes from the database.
const figures = await db.weeklyRevenue(week);

const draft = await model.write({
  instructions: 'Summarise these figures in three sentences. Use only numbers from the data.',
  data: figures,
});

// Each number in the text must exist in the data.
const invented = numbersIn(draft).filter((n) => !figures.includes(n));
if (invented.length > 0) return plainTemplateReport(figures);

return draft;

That one check removes the worst failure, which is a confident sentence with an invented number in it. If it fails, the reader gets a duller report built from a template, which is still a correct one.

You need a test set, even for prompts

A prompt is code, and it changes behaviour just as much. So is the model version you’re calling.

Collect 50 to 200 real examples with known correct answers, including the awkward ones. Run them whenever the prompt or model changes, and track the score. Without this, every change is a guess, and “it seems better” is the only result you’ll ever get.

Things to decide before you build

  • What happens when the model is wrong? If the answer is “a person notices and fixes it in 30 seconds”, you have lots of room. If it’s “a customer gets the wrong invoice”, you need strong checks.
  • What happens when the model is down? The process should degrade, not stop. Queue the work or fall back to the manual route.
  • Which data leaves your systems? Know what you send, to whom, and under what agreement. Send the minimum, and under GDPR check your processor agreement and data location before the first test on real data.
  • What does one run cost? You pay per token, so measure it early. Very often a smaller model does the job once the task is narrow enough.

A quick test for good candidates

A task is probably a good fit when:

  • the input is messy text, documents or images,
  • a person could do it without specialist judgement,
  • a wrong answer can be caught or cheaply corrected,
  • you have real examples to test on,
  • and it happens often enough to matter.

And it’s probably a bad fit if a clear set of rules would solve it. Some of the best advice I can give about an AI project is that it doesn’t need to be one.