How to prove AI training worked: use a comparison group

If you want to prove AI training worked, you need to compare two groups: people who've had the training and similar people who haven't had it yet. Measure both on the same tasks, marked the same way, at the same two points in time. The difference between how much each group improved is the effect of the training. A before-and-after on one group can't give you that number, however carefully you run it.

What's wrong with before and after?

There are three things that push results up, whether or not the training worked.

• Practice. People tend to do better the second time they sit an assessment. A large review of 50 studies, covering more than 134,000 people, found scores on cognitive ability tests typically rose on a second attempt. The gain was modest, roughly enough to move someone from the middle of the group to the 60th percentile. It was larger with coaching, but it appeared even without it.
• Everything else going on in the firm. New hires, a quiet quarter, or a better version of the tool. Any of them can move the numbers.
• Self-report. People's own estimates of the time AI saves them often don't match what can be measured, as the trials below show.

The government trials are a good example. Civil servants in the UK's cross-government Copilot experiment reported saving 26 minutes a day. The Department for Business and Trade, which looked harder at its own pilot, said: "We did not find robust evidence to suggest that time savings are leading to improved productivity." And in Denmark, a study of 25,000 workers found people reported saving about 3% of their time with AI chatbots, but no significant effect on their earnings or recorded hours.

What do studies with a comparison group find?

When researchers use a proper comparison, the effect is real, but it's uneven.

• In a 2023 Harvard Business School and BCG field experiment, 758 consultants were randomly assigned to work with or without GPT-4. On tasks within its abilities, those using it completed 12.2% more tasks, 25.1% faster, with work rated more than 40% higher in quality. But on a task just outside its abilities, they were 19 percentage points less likely to get the right answer.
• A study of 5,179 customer support agents found AI raised the number of issues resolved per hour by 14% on average, and by 34% for newer and less skilled agents, with little effect for the most experienced. AI was rolled out in stages, so the agents who got it later acted as the comparison.
• A randomised trial across 66 firms and more than 7,000 workers, run by researchers at Microsoft and Harvard, found people given Copilot spent less time on email, but there was no significant change in time spent in meetings.
• In England, an Education Endowment Foundation trial found teachers using ChatGPT with a guide cut their lesson planning time by 31%, with no noticeable difference in quality.

I think there are two lessons here for firms. The effect varies by task and by person, so one average across the whole firm hides most of the story. And training can cut both ways. In the consultants' study, the group given a short prompting overview gained more on quality within AI's abilities, but did worse on the task outside them.

How do you set up a comparison group in a firm?

You don't need a research team, just four decisions made before anyone is assessed.

1. Decide who goes first. Split people into two similar groups, by role and seniority, before the first assessment, so nobody can pick the groups to flatter the result.
2. Stagger, don't withhold. The comparison group gets the same training later, say a quarter on. HM Treasury's guidance on evaluating AI describes staggering a rollout as a way to build in a comparison.
3. Use the same tasks and the same marking. Both groups do the same real work, marked against the same criteria by reviewers who don't know who was trained.
4. Measure twice, and compare the change. Assess both groups before the training and again after. The trained group's improvement minus the comparison group's improvement is the effect of the training.

That last step is what really matters. For example, if the trained group's typical score rose 18 points and the comparison group's rose 4 just from practice, the training was worth 14 points, not 18. As HM Treasury's Magenta Book puts it, the comparison group "acts as a proxy for what would have happened to the affected group in the absence of the intervention".

Why don't more firms do this?

I think it's mostly habit. Training tends to be judged on attendance and feedback forms rather than on what changes in the work. AI training is also still new for most people. The CIPD's Good Work Index 2026, a YouGov survey of 5,074 UK employees carried out between December 2025 and February 2026, found only 13% had received any training in using AI in their current job.

Holding people back can also feel awkward. Staggering solves that, as everyone gets the training, just not all at once.

What if you can't use a comparison group?

Then I'd report the change over time, and say plainly that there was no comparison. Telling a leadership team "scores rose 18 points, but we can't separate the training from practice" is more useful than a confident number that won't stand up.

It's how I've designed Osava to report it too. With a comparison group, the firm report separates the effect of training from everything else going on. Without one, it shows how results changed from one round to the next, and says clearly that there's no comparison. For the wider picture, have a look at how to measure the ROI of AI.