The most dangerous AI agent is the one that almost works

    Max Västhav
    aigovernancequalityautomation

    The agent that gives the right answer nine times out of ten feels reliable. The tenth time, it sends an incorrect price to a customer. Or publishes a post with the wrong tone. Or updates the CRM with data from the wrong person.

    Half-working automation is more dangerous than no automation at all — because it builds trust before it does damage.

    Why "almost" is the problem

    A manual workflow has built-in checks. You read through the email before sending it. You double-check the price before giving the quote. You think.

    An agent that almost works removes those checks. The team stops reviewing output. "The AI is usually right." Until it isn't.

    Three common patterns

    1. The agent responds to customers without review

    A support agent generating responses is often correct. But sometimes it makes up policies, mixes up cases, or promises things the company doesn't offer. Since 90% of responses are good, the team stops checking — and customers pay the price.

    2. The agent sends external communication

    A LinkedIn post, a newsletter, a follow-up email. The agent has seen thousands of examples and can write convincingly. But without human review, tone, facts, or timing can be completely off.

    3. The agent changes data in production systems

    CRM fields, invoices, inventory status. The agent "helps" by filling in what it thinks is missing. But the assumptions can be wrong, and the consequences are discovered weeks later.

    The solution: checkpoints before external actions

    The principle is simple: the agent can do whatever it wants internally, but everything going out — to customers, systems, or the public — requires a checkpoint.

    Define three categories:

    Free (no approval):

    • Read data
    • Summarize information
    • Classify cases
    • Create internal drafts

    Approval required:

    • Send email to customer
    • Publish content
    • Update CRM fields
    • Create quotes

    Forbidden (the agent may not):

    • Delete customer data
    • Change prices in production systems
    • Send legal information

    How to build the checkpoints

    1. List all of the agent's outputs. What does the agent produce? Where does it end up?
    2. Mark external outputs. Everything a customer, partner, or the public can see.
    3. Set approval before each external output. A human confirms before it's sent.
    4. Log everything. Input, the agent's suggestion, who approved, what was sent.

    It takes an hour to define. It saves the day you discover the agent was wrong.

    The best agent makes mistakes visible

    A good agent isn't one that's never wrong. It's an agent that makes its mistakes easy to find, review, and correct.

    Build logs. Build checkpoints. Build trust step by step.

    The agent that almost works can become truly great — if you let it make mistakes in the right places.

    Cookies. Privacy