In this blog post I'm going to give you the 38-question AI vendor evaluation checklist I use when picking AI tools for client engagements. The version that cuts through demo theatre. Not the version that exists to make you feel sophisticated while still buying the wrong tool.
Most "AI vendor evaluation" content in 2026 is written for enterprise procurement teams. Massive scorecards, weighted matrices, RFP templates. Useless if you're a non-technical marketer with two days to decide between three options.
This checklist is the practical version. 38 questions in 6 categories. Each has a clear pass/fail signal. None require technical depth to ask. All surface the difference between a vendor that will work for you and one that won't.
I've been a marketing consultant for twenty-one years. I went all in on AI in 2024. I evaluate AI vendors continuously, for my own business, for client engagements, and for my newsletter audience. The checklist below is built from real evaluations, real disappointments, and real "I should have asked that" moments.
By the end of this blog you'll have the checklist, the disqualifying answers per question, and the order to use it in.
TL;DR
Six categories, 38 questions:
- Fit and scope (6 questions), does this tool do what we need
- Data handling (6 questions), what happens to our data
- Integration (6 questions), will it play with our existing stack
- Support and reliability (6 questions), will it be around in 12 months
- Pricing and commercial terms (8 questions), total cost of ownership
- Compliance and risk (6 questions), what could go wrong
Decision rule: pass all 6 categories, or don't buy. One failed category = walk away.
Category 1: Fit and scope (6 questions)
1. Show me how this works on data shaped like ours.
Why it matters: Generic demos use clean data. Your data is messier. Ask them to run the demo on a sample (anonymised) version of your real data.
Disqualifying answer: "Just trust the demo." If they won't show you the tool working on your data shape, that's a signal.
2. What's the most common reason customers DON'T use this tool effectively?
Why it matters: Honest vendors know the failure modes. Dishonest vendors will say "everyone uses it effectively."
Disqualifying answer: Anything that implies their tool has no failure modes.
3. How do you handle [your specific edge case]?
Why it matters: Every business has at least one workflow nuance that breaks templates. If the vendor can't articulate how they handle it, they probably can't.
Disqualifying answer: Vague gestures at "customisation" without specifics.
4. What's NOT in scope for this tool?
Why it matters: Good vendors define what they don't do. Bad ones promise everything.
Disqualifying answer: "We can do anything you need." Every tool has limits.
5. How does this tool make decisions I'd otherwise make?
Why it matters: If the tool makes high-stakes decisions, you need to understand its logic. If it can't be explained, it can't be governed.
Disqualifying answer: "It's proprietary" or "the AI just figures it out."
6. Can I see the actual outputs for a customer like me?
Why it matters: Marketing pages are aspirational. Actual outputs are what you'll live with.
Disqualifying answer: "Those are confidential" without offering an alternative way to validate output quality.
Category 2: Data handling (6 questions)
7. Is our data used to train your model?
Why it matters: Yes = your data becomes someone else's competitive advantage. No = your data stays yours.
Disqualifying answer: Hedging or "by default yes but you can opt out." Should be no by default.
8. Where is our data physically stored?
Why it matters: GDPR, data residency rules, your customers' expectations.
Disqualifying answer: "Various cloud regions" without specifics.
9. Who can access our data on your side?
Why it matters: Support team? Engineering? Sales? Each access tier is a risk surface.
Disqualifying answer: "Everyone with a need to know" without role definitions.
10. How do we delete our data when we leave?
Why it matters: "Reasonable timeframe" isn't a timeframe. You need specifics.
Disqualifying answer: No defined deletion timeline. Should be 30-90 days max.
11. What's your data breach disclosure timeline?
Why it matters: GDPR requires 72-hour notification to authorities. Your vendor should match or beat that.
Disqualifying answer: Anything longer than 72 hours or "we'll notify if required."
12. Do you sub-process our data?
Why it matters: Many AI vendors use multiple sub-processors (OpenAI, Anthropic, etc.). You need the full list and DPAs.
Disqualifying answer: No documented sub-processor list.
Category 3: Integration (6 questions)
13. What's your CRM/marketing-stack integration list?
Why it matters: Tools that don't integrate are islands. Manual data movement breaks workflows.
Disqualifying answer: "We can integrate with anything via API." That means "you build the integration."
14. Is the integration native or via Zapier/Make?
Why it matters: Native integrations are reliable. Zapier-based integrations break when things change.
Disqualifying answer: Heavy reliance on third-party glue tools.
15. What happens if your integration breaks?
Why it matters: When integrations break, workflows stall.
Disqualifying answer: No documented SLA for integration repair.
16. Can I export everything in standard formats?
Why it matters: Vendor lock-in via proprietary formats is a real risk.
Disqualifying answer: Anything other than yes, with CSV/JSON exports for all data.
17. Do you have a public API?
Why it matters: Public API = you can build automation around the tool. No public API = you're stuck with their UI.
Disqualifying answer: "It's coming soon."
18. What's the maximum API rate?
Why it matters: Low rate limits constrain workflow design.
Disqualifying answer: Anything below 60 requests per minute for active workflows.
Category 4: Support and reliability (6 questions)
19. What's your uptime SLA?
Why it matters: AI tools used in production need 99.5%+ uptime.
Disqualifying answer: No documented SLA or anything below 99%.
20. How long has the company existed?
Why it matters: AI vendors under 2 years old may not be around in 12 months.
Disqualifying answer: Under 18 months without strong funding.
21. How many customers do you have in our size/segment?
Why it matters: A vendor with 1,000 enterprise customers and 5 SMB customers won't prioritise SMB support.
Disqualifying answer: Vague answers or "we're equally focused on all segments."
22. Where's your support team based?
Why it matters: Time zone, language, response time.
Disqualifying answer: "Distributed globally" without specifics on your time zone coverage.
Want AI doing the heavy lifting in your marketing?
I build the systems that handle the boring 80 percent, so you get your week back. Done properly, with the human kept in.
23. Show me a recent uptime incident and how you handled it.
Why it matters: All vendors have outages. Good ones disclose them transparently.
Disqualifying answer: "We haven't had any incidents." Either they're hiding or they've not been around long.
24. What does your roadmap look like for the next 12 months?
Why it matters: Stagnant roadmaps signal a tool on autopilot or about to be deprecated.
Disqualifying answer: Generic platitudes or no specific upcoming features.
Category 5: Pricing and commercial terms (8 questions)
25. What's the total annual cost at our usage level?
Why it matters: Headline pricing rarely matches what you'll pay.
Disqualifying answer: Anything requiring more than one follow-up to get clarity.
26. What triggers price increases?
Why it matters: Token-based pricing can scale unpredictably. Seat-based can balloon.
Disqualifying answer: No clear pricing model documentation.
27. What's your annual price increase policy?
Why it matters: "Up to market rate" is meaningless. You need a cap.
Disqualifying answer: No cap on annual increases.
28. Is multi-year discount worth the lock-in?
Why it matters: Multi-year discounts trap you with bad tools.
Default position: Annual contracts only for the first cycle.
29. What are the exit terms?
Why it matters: Without exit terms in the contract, you're at their mercy.
Disqualifying answer: Vague exit terms or non-standard penalties.
30. Are there setup fees, integration fees, or training fees on top of subscription?
Why it matters: Hidden fees are common.
Disqualifying answer: Discovering fees you weren't told about during sales.
31. What's the auto-renewal policy?
Why it matters: Auto-renewal can lock you in if you miss the cancellation window.
Disqualifying answer: Anything shorter than 60 days notice for cancellation.
32. Can I see your invoice format?
Why it matters: Some vendors invoice in weird currencies, weird structures, or with surprise line items.
Disqualifying answer: Refusal to share a sample invoice.
Category 6: Compliance and risk (6 questions)
33. What certifications do you have?
Why it matters: SOC 2, ISO 27001, GDPR readiness. Each matters depending on your needs.
Disqualifying answer: None of the relevant certifications, or only "in progress" status.
34. Are you compliant with the EU AI Act?
Why it matters: EU regulation phases through 2025-2026. Affects high-risk AI use cases.
Disqualifying answer: "We don't operate in the EU" if you have any EU users.
35. Do you have liability insurance?
Why it matters: When AI output causes harm, you need to know who's liable.
Disqualifying answer: No documented liability coverage.
36. How do you handle model deprecation?
Why it matters: AI vendors retire models. Your workflow built on the old model may break.
Disqualifying answer: No model deprecation policy.
37. What's your incident response plan?
Why it matters: When something breaks, who decides what?
Disqualifying answer: "We'd figure it out at the time."
38. Can I talk to a customer who churned?
Why it matters: Current customers are biased. Past customers tell you the truth.
Disqualifying answer: Refusal. Good vendors keep relationships with thoughtful past customers.
How to use this checklist
Step 1: Send 5-7 questions in writing before the demo. The questions you most care about (usually Categories 2 and 5).
Step 2: Run the demo. Ask 5 more questions during the demo (usually Category 1 and 3).
Step 3: Schedule a follow-up call. Ask the remaining questions over a structured 45-minute conversation.
Step 4: Talk to 2-3 references. Use their answers to validate vendor answers.
Step 5: Make the decision based on full evidence, not demo dazzle.
Total evaluation time: 6-12 hours over 2-3 weeks. Saves you from a 12-month bad-tool commitment that costs 50-200x more in time and money.
What this checklist deliberately doesn't include
- Star ratings, they're easily gamed.
- Feature comparison matrices, they reward feature count, not feature quality.
- AI capability benchmarks, different vendors test on different benchmarks. Useless.
- Brand prestige, known vendors aren't automatically better.
Frequently asked questions
Should I use this for free AI tools too? Categories 2 and 6 still apply. The cost questions are obviously different.
Can I send this checklist to the vendor in advance? Yes for written questions. The best vendors will answer thoroughly. The worst will reject the exercise.
What if the vendor refuses to answer something? That's data. The refusal itself is the disqualifying answer.
How long should the evaluation take? 2-3 weeks for a meaningful AI tool decision. Longer if multi-stakeholder.
Should non-technical buyers do this alone? Categories 1, 4, 5 are non-technical. Categories 2, 3, 6 benefit from a technical reviewer (your CTO, a fractional, or a consultant).
Want help running this evaluation?
If the conclusion is "none of these fit, look at X instead," that's the conclusion. You walk away with the diagnosis and no further pitch.
Book an AI vendor evaluation →
I'm Lilach Bullock. I've been a marketing consultant for twenty-one years. I went all in on AI in 2024. I work with founders and marketing leaders who want AI to move their numbers, not just their tool stack.
For the bigger picture, see my full guide to AI marketing.
