Do you know thesetwelve things?

An agent is an AI model given a goal, some tools and permission to act. It reads the case, decides the next step, uses a tool (a claims system, an inbox, a spreadsheet), checks what happened, and goes round again until the job is done or it hands over to a person.

A chatbot answers; an agent acts. So when it gets something wrong, it doesn’t just say it, it does it. 62% of organisations are already building, testing or deploying agents,⁵ and three in four enterprise AI use cases are now a vendor’s, not their own.⁴

A task is one piece of real work, written down so an agent can try it again and again and be marked automatically each time. It has seven parts: the instruction, the starting state, the tools, the grader, a reference answer, a label and the pass rates.

The pass rates are the point. They come from many runs, never one, and they belong to your task, not to a public leaderboard. Real work is hard: the top agent completes about one in five computer workflows of around 1.6 hours.¹²

Every task costs money before anything goes wrong: the minutes it takes, the cost of the person doing it, and the software that manages it. Multiply that by how often it comes round, hundreds or thousands of times a week in a large company, and one ordinary task adds up fast.

That is where an AI decision starts: what each task costs, how often it happens, and which ones an agent can be trusted with. Mistakes matter too, for your customers and your reputation, which is why an agent must clear the bar on held-back cases before it gets the job.

Benchmarks often report pass@k: the task counts as solved if the agent gets it right at least once in k tries. Your customers get one try each, every time, so what matters is pass^k: right on every one of k tries.

The gap is large. An agent right 75% of the time on one try gets three in a row right less than half the time.⁹ On τ-bench, a 2024 model’s 61.2% on one try fell under 25% when all eight tries had to succeed.¹⁰

A perfect streak proves less than it looks. Eight clean runs out of eight still leave room for the agent to fail up to 31% of the time. A hundred clean runs bring that down to about 3%, and it takes 59 to claim 95% reliability.⁶¹

So Butterfly Tasks will test each agent on real cases held back from the start: at least 20 a task, typically about a hundred.

Ask a hosted model the same thing twice, with the same settings, and you can get two answers: 1,000 identical requests at temperature 0 gave 80 different outputs.¹⁹ More thinking doesn’t always help either: in one large study, higher reasoning effort reduced accuracy in a majority of runs.¹⁸

So one run is a sample of one, which is why the Agent Reliability Index will test each agent across many held-back cases, not on a single answer.

Your vendor’s agent, your own team’s and your adviser’s pick will each come with a score, and the scores rarely agree. A private benchmark settles it: a test made from your own work, run in your own cloud and kept to you. It answers the question a public test can’t: which agent is ready for this job, here.

Public tests also wear out. A frontier lab stopped using SWE-bench Verified after judging 59.4% of the hard tasks it audited broken, as reported.¹⁶ Nearly a third of Terminal-Bench 2.0’s reviewed tasks needed fixes within six months.¹⁵ And the model that tops one test can trail on another.¹⁷

When an agent clears the bar, it gets a blueprint: the access it needs and no more, where a person stays in, and the test it must keep passing. The task owner approves it before it goes live.

Then it is tested again every week, and fully each quarter, because models change, and so do the rules. A person in the loop is not a formality: court cases involving AI-hallucinated content now number 2,217, 76 of them in the UK.²²

Butterfly Tasks is built to run inside your own cloud, and Butterfly staff will have zero access to your data. But where a prompt is processed depends on the model: on Amazon Bedrock, model calls run in accounts AWS owns, in the region the model is offered in.⁴⁵

So before the first comparison, a residency tier is agreed with you. From London, some models run in-Region, some through the EU profile, and some through global routing alone,⁴⁷ which join a comparison when you opt in, never by default.

Taking the names out is not enough. Data counts as anonymous when no one can single out a person, link records about them, or infer something about them, like a diagnosis or a salary.⁴⁸ Italy fined IQVIA €7m over health data it called anonymous: a persistent ID with birth year, sex, diagnoses and location let people be singled out.⁴⁴

Synthetic data needs testing too: attacks recovered 78–100% of outlier records even when similarity tests passed.⁴³ Butterfly Tasks will read your records inside your own account, and personal data about your customers never leaves it: what may leave, with approval, includes synthetic examples, each checked six ways.

The EU AI Act arrives in stages. Its transparency duties applied from 2 August 2026; high-risk duties for workplace AI apply from 2 December 2027; AI built into regulated products follows on 2 August 2028.⁵¹ High-risk uses include credit scoring, life and health insurance pricing, recruitment, and AI that allocates or evaluates work.⁵⁴

Waiting is not safe: data protection law already bites on automated decisions. The Netherlands fined Uber €824.99m over drivers deactivated without human review.⁵⁰ Butterfly supplies the test evidence; you stay responsible for meeting each rule.

AI labs train and test agents on tasks: real work with a known right answer. They typically pay a few hundred to a few thousand dollars a training task, and much more for exclusivity.³⁶

A task stops teaching a model once it can pass it: on the same Terminal-Bench tasks, the top score rose from 62.9% to 76.0% in four months.¹⁴ So fresh, harder work from real jobs keeps its value. Butterfly’s data program licenses the tasks you choose, never your records.