The most dangerous moment in an AI assistant project is the week after launch.
The assistant is live. Usage is high because the team is curious. The demo numbers look good. The project is declared a success.
Then the attention moves on, usage slowly normalises, and six months later nobody is quite sure whether the assistant is still helping.
Here are the five metrics we track in the first 90 days after deployment.
- Active usage rate
What percentage of the team is using the assistant at least once a week? If usage is concentrated in one or two people while the rest of the team ignores it, the assistant has an adoption problem — regardless of how well it performs for those who do use it.
A healthy active usage rate at 90 days is above 70% of the intended user base. Below 50% is a signal that something went wrong in onboarding or product fit.
- Task completion rate
When a user asks the assistant to do something, what percentage of the time does it produce an output the user finds useful without significant correction?
This is different from accuracy. An accurate response that's formatted wrong or not actionable won't get used. Task completion measures whether the output is actually useful, not just technically correct.
- Time saved per task
Compare how long the target task takes with the assistant versus without it. Measure this on real tasks, not benchmarks. The delta is the value the assistant is delivering — and the number that justifies the investment.
- Error and correction rate
How often are users correcting the assistant's outputs before using them? A correction rate that's dropping over time means the assistant is improving. A correction rate that's rising is a signal that something in the knowledge base or prompts needs attention.
- Out-of-scope rate
How often are users asking the assistant something it can't help with? A high rate means either the scope was too narrow, or users have found use cases you didn't anticipate — both of which are useful signals for how to develop the assistant next.
Review these metrics monthly for the first 90 days. The first version of any assistant is never the best version. These numbers tell you where to invest the improvement.