Why usage data won't tell you if AI is paying off
Logins and licence counts measure activity. Here's what to measure instead, and how to start without a data team.
By Justin Davidson, Founder
Published · 2 min read
Most firms can tell you how many AI licences they pay for. Far fewer can tell you what those licences return. In the Thomson Reuters Institute's 2026 AI in Professional Services Report, 18% of respondents said their organisation tracks the return on its AI tools.
The usual response is to reach for usage data. It's easy to get and it produces a neat chart, but it doesn't answer the question the partners are asking.
What does usage data actually tell you?
It tells you that people opened a tool. It doesn't tell you what they did with it, whether the output was any good, or whether it saved time on work that matters. A high login count can mean real value, but it can also mean someone rewording their emails.
What should you measure instead?
I'd look at three things, in this order:
- Adoption by kind of work. Not who logged in, but who uses AI on client work, and on which tasks.
- Capability on real tasks. Outputs from real client work, scored by senior people against a short set of criteria.
- Impact against a comparison group. The same tasks and the same reviewers, for people who were and weren't trained.
Why the comparison group matters
Why is a comparison group worth the effort?
Firms change all the time: new clients, new hires, a busy quarter. If all you measure is before and after, you can't tell which changes came from AI and which would have happened anyway. A comparison group, measured the same way, removes most of that noise. I go into more detail in How to prove AI training worked.
Where should you start?
Pick one team, one kind of task and a small comparison group. Record a baseline before any training. Ask two senior reviewers to score a sample of outputs against the same criteria. Then repeat the exercise once the training has had time to take effect.
That's enough to give the partners a real answer. It's also the approach I'm taking with Osava.
Source: Thomson Reuters Institute, 2026 AI in Professional Services Report.