AI in Zero-Error Environments

As NASA administrator in the 1960s, James Webb introduced computer chips to the Apollo spaceflight control system against the advice of his colleagues.

He would trust this then-adolescent technology, for which Apollo would be 60% of total demand, to guide U.S. astronauts to the moon. Conviction is required when the stakes are high and errors are unacceptable.

Like computer chips, artificial intelligence promises to transform how we live and work. Our industry is no exception. In the most recent Council of Insurance Agents & Brokers Brokerage Operations Survey, more than one-fifth of responding firms expressed a desire to invest heavily in AI capabilities like document processing and client content generation. Incorporating these systems, the next decade of brokerage operations will inevitably look different from the last.

However, brokerage workflows are often high-precision, error-sensitive endeavors. AI errors can have vast and severe consequences. Incorrect reconciliation or policy lifecycle processing pollutes the systems of records. Inaccurate analyses threaten client retention and incur errors and omissions liabilities. Above a certain error rate, review and remediation costs may ultimately exceed initial savings. It is imperative to build a framework for assessing solutions in such zero-error tolerant environments.

At Eventual (fka Eventual Treasury), we have been fortunate to work with many leading brokerages on their AI initiatives for middle- and back-office workflows, including direct bill reconciliation, agency bill automation, and policy servicing workflow. Through our partnerships, we’ve had the opportunity to stress-test our systems across diverse brokerage operating models and technology stacks, and to refine our platform and processes for deploying into error-sensitive workflows.

We have distilled the following questions from our conversations. They can serve as a simple litmus test for the intellectual honesty and technical maturity of any insurance AI vendor.

What Is the Error Profile of Your Underlying Technologies?

AI is inherently non-deterministic, and enterprise-grade systems must often combine numerous technologies to perform. While it’s tempting to lean into marketing hype around large language models (LLMs), delivering practical solutions requires a deep understanding of the insurance context and underlying technologies. For instance:

• Despite the excitement around visual LLMs, they struggle with a variety of basic tasks. Specialized optical character recognition models still outperform LLMs for document extraction on accuracy and precision.
• Even though LLMs’ context sizes (the maximum number of tokens they can consider) continue to increase, when the input is too large, the LLM’s ability to reason degrades and error rates spike. LLM-driven features often require sophisticated context engineering.

How Does Your System Detect and Mitigate Errors?

Detection and mitigation systems are precursors for reliable AI deployments. Best-in-class vendors often combine numerous guardrails to prevent silent errors, including numerical validation, anomaly detection, or verifier-metaverifier architectures. For errors that must escalate to human review, continuous learning systems can then incorporate user feedback, reduce escalations over time, and increase operational efficiency.

When Does Your Product Incorporate Human Intervention?

Human command and control is a feature, not a bug, of error-sensitive systems. For insurance AI, enabling human review and governance is as important as pushing the system’s reliability. Document intelligence should come with side-by-side review; error-sensitive workflows should incorporate human approval; and when a system is not sufficiently confident, it must fall back to manual intervention.

Choosing technologies during periods of rapid change is no mean feat, especially when they are multiyear decisions. When James Webb signed off on the Apollo platform, he was committing to using that platform for a decade: go too conservative and the United States would get left behind in the space race; too aggressive and lives would be lost.

Likewise, the choice of brokerage technologies can be long marriages. Executives today must navigate both the necessities and the uncertainties of change. Operational and technological leaders should therefore forge a common path forward. Together, we can safely and confidently usher in the next chapter of our industry.