Agent demos are everywhere. I do them!  And yet I find that people say, “that’s interesting, can you do X instead?”.  The common ones: an agent that reads invoices, answers customer questions, or drafts purchase orders. The demos are genuinely impressive. Then I ask how many are running in production, unattended, against real transactions. Anyone?

ERP people have seen this pattern before. Every implementation includes a conference room pilot, where the software handles a scripted scenario with clean data and everyone nods. Nobody confuses the conference room pilot with go-live. Go-live is where the exceptions, the bad master data, and the actual users show up. Most organizations’ agents are at the conference room pilot stage, and the industry is talking about them like they’ve gone live.

> They have tried to automate too much, too quickly.

As an AI ERP MVP, I do demos of so many out of the box agents, and find that our team is still building custom.  Because most business processes are not the same; they don’t fit into a single box.  And this is fine, because it helps us humans understand what the software is capable of.

What the demo misses

A demo runs a happy path with curated data. Production is everything the rehearsal left out.

Identity and permissions. What credentials does the agent act under? An agent running with a human’s full access is a security finding waiting to happen. Least-privilege for agents is important, yet demos skip it.

Being wrong. What happens when the agent is wrong? Not if. When. Who catches the error, how fast, and what did it cost by then? The demo never answers this because the demo was rehearsed until it stopped being wrong.  Belive me, because I make and deliver demos 😬

Audit & observability. You need a record of what the agent decided and why, in a form a controller or an auditor will accept, understand, and eventually trust. “The computer chose to” is not an answer anyone signs off on.

Ownership. Prompts drift because models are changing, the rug is moved underneath you (unless you have evals set up!). Someone has to own the agent, its actions, and quality – like an employee. If nobody owns it, the agent will break, and you’ll find out from the business.

Treat it like a go-live, because it is one

The organizations getting agents into production are treating them the way they’d treat any system that touches transactions. Start with a process where mistakes are cheap and visible. Keep a human approving outputs until you have measured the error rate, not guessed at it. Test against production-scale data, including the messy records, before cutover. Define what rollback looks like. Assign an owner and ensure periodic reviews are happening.  Owners frequently do not have enough access to the telemetry of those agents, so IT assistance is needed.

Demos prove the technology works. Production deployment proves the organization uses and manages AI appropriately.

Stay frosty ;)!