The AI agency market is crowded and noisy. Everyone claims to build custom AI assistants. Very few have shipped something in production that a real team uses every day.
If you're evaluating vendors, here are the questions we'd want a client to ask us — and the answers that should make you think twice.
- Can you show me something you've built that's in production?
Not a demo built for the sales process. Something a real team uses as part of their actual workflow. If the answer is no, you're evaluating an agency that's promising something they haven't done yet.
- Who will actually be doing the work?
Ask specifically. Is it the senior people you're meeting, or will it be handed to a junior team after the contract is signed? For AI assistant projects, the quality of judgment in the build matters enormously.
- What happens after deployment?
A good vendor has a clear answer: what support is included, for how long, what the process is for refinement, and when the project is officially complete. If the answer is vague, the post-launch experience will be too.
- How do you handle edge cases and wrong answers?
Ask them to explain their approach to hallucination, uncertainty, and out-of-scope questions. If they say "the model handles it," they haven't thought about it carefully enough. A good answer describes specific design decisions.
- What's your discovery process?
Before building anything, how do they learn about your workflow? What does discovery produce? If the answer is "we ask a few questions and then start building," the scope will be defined by whoever is doing the work, not by what you actually need.
A note on red flags
The biggest red flag isn't a wrong answer to any of these questions. It's vagueness. An agency that's done this before has clear, specific answers. An agency that hasn't tends to speak in generalities and pivot to capabilities rather than evidence.
Ask specific questions. Listen for specific answers. The gap between the two tells you most of what you need to know.