A vendor demo is typically a scripted 30-minute presentation followed by 15 minutes of Q&A. The useful part is the Q&A. Use this list of 20 questions to extract real information and expose vendors who are hiding cost or capability gaps. Ask all 20 if you can; prioritize the first 10 if you have less time.
Cost and Attribution Questions (6 questions)
Ask these first. Cost and attribution are non-negotiable.
1. Walk me through your cost attribution interface. Pull up a specific resolved work item and show me the cost breakdown.
This is the key question. A vendor that can do this in real time has attribution built in. A vendor that says "I'll get back to you on that" doesn't. Watch them navigate to the dashboard. If it takes more than 30 seconds, attribution is not a first-class feature.
2. For that work item you just showed, what's included in the cost? Is it tokens only, or tokens plus retries, API calls, and vector retrieval?
Push back if the answer is "tokens only." That understates cost by 60–70%. You need a full cost breakdown.
3. What's the 95th percentile cost per resolved work item for a customer similar to us?
They should have this number memorized. If they don't, or if they say "it depends," they haven't benchmarked their cost carefully. Ask them to email you a case study or customer reference after the demo.
4. How does your pricing change if we exceed your standard tier? Give me an example.
Example: "Tier 1 is 0–10K items/month at $0.50 per item. Tier 2 is 10K–50K items/month at $0.40 per item." If they won't give you a specific number, they're hiding pricing elasticity. Never accept "we'll discuss that in contract negotiations."
5. What happens if our actual volume is 20% lower than we forecast? Do we get a credit or refund?
Most vendors will say "no, you pay for the full contract." Push back. Ask for a true-down clause. If they refuse, their pricing is inflexible and you're taking risk they should be taking.
6. Can you expose cost per work item in your API responses, or is cost data available only through a dashboard?
API-level attribution is better than dashboard attribution. It lets you join vendor cost with your outcome data. A vendor that exposes cost in the API has more confidence in their transparency.
Lock-In and Vendor Risk Questions (4 questions)
7. What foundation models do you support? Can I swap Claude for GPT-4, or am I locked into Claude?
If you're locked into one model, you have model-level lock-in. A vendor that supports multiple models gives you optionality.
8. If Anthropic deprecates Claude 3.5 Sonnet, what happens to my agent? How much notice do I get, and what's my upgrade path?
This is the model deprecation question. Ask them to show you a specific example (if it's happened before). A good answer: "We give 90 days' notice and we offer a free migration to the new model, or you can stay on the old model for an additional year." A bad answer: "You'll have to migrate immediately."
9. If we want to port our prompts and data to a different vendor, can you export them? What format, and how long does it take?
This tests data portability. A vendor that says "yes, JSON export, 24 hours" is low-lock-in. A vendor that says "that's not something we support" is high-lock-in.
10. Walk me through your data retention policy. If we terminate the contract, how long do I have to export my data? What happens if I don't export in time?
Good answer: "You have 30 days to export all data in JSON/CSV/Parquet. After 30 days, we delete it." Bad answer: "We'll delete it after 90 days." Worse: "We need to discuss that with legal."
Operations and Performance Questions (5 questions)
11. What's your published uptime SLA, and what's the credit if you miss it?
They should have a number (e.g., 99.5% uptime, 10% credit per 0.1% miss). If they don't have a published SLA, they're not serious about reliability.
12. What's your typical API latency (p50 and p99)? How does it scale if we 10x our volume?
A vendor that has measured latency percentiles is mature. p99 should be <1 second for most use cases. If p99 is >2 seconds, you'll see cascading timeout failures at scale.
13. What's your incident response process? If your service goes down on a Friday night, what's your time-to-detect and time-to-resolution?
This reveals operational maturity. Good vendors: "We detect within 5 minutes via automated monitoring, and we target 30-minute resolution." Bad vendors: "We monitor during business hours."
14. Do you offer geographic redundancy or multi-region failover? If your primary data center fails, what happens?
This is important if you're in a regulated vertical. A vendor with multi-region redundancy can survive datacenter failure. A vendor with no redundancy is a risk.
15. What's your customer support model? Do I get a dedicated account manager, and are they available 24/7?
This matters less for enterprise vendors but matters a lot if you're a small/mid-market company and the vendor is your sole support. If you get a dedicated AE but no technical support, that's a problem.
Compliance and Data Handling Questions (3 questions)
16. What compliance certifications do you hold? Can you share your SOC2 audit report under NDA?
They should have SOC2 Type II (not Type I—Type I is a one-time audit, Type II is ongoing). If they claim HIPAA or FedRAMP, ask to see proof. Never trust "we're compliant" without evidence.
17. Where are your data centers, and what data residency options do you offer?
If they say "we only store data in the US" but you need EU residency for GDPR, you have a problem. Ask explicitly about EU, US, and any other jurisdictions you need.
18. Do you have a Data Processing Agreement (DPA) and a sub-processor list?
This is table-stakes for any vendor. A vendor without a DPA is not GDPR-compliant. Ask to see it.
Sustainability and Roadmap Questions (2 questions)
19. How long have you been in business, and what's your current funding status? Are you profitable?
This is a vendor risk question. A well-funded, profitable vendor is more likely to survive and maintain service. An underfunded vendor might pivot, shut down, or get acquired. Ask for transparency.
20. What happened to your customers when your previous vendor relationships changed? (E.g., did your access to GPT-4 API change, and how did you handle it?)
This tells you how the vendor responds to external shocks. A vendor that proactively communicated and offered a migration path is mature. A vendor that suddenly changed terms and left customers stranded is a risk.
Demo scoring rubric
Score each vendor on these dimensions:
| Question Category | Score 5 | Score 3 | Score 1 | | --- | --- | --- | --- | | Cost & Attribution | Can show cost per work item in real time; 95th percentile benchmark available | Cost data available in dashboard; benchmark estimate provided | Cost is aggregated only; no work-item visibility | | Lock-In Risk | Model-agnostic; data export supported; <4 week switching cost | Model-specific; data export available; 4–8 week switching cost | Single model; data export difficult; >8 week switching cost | | Operations | Published SLA 99.5%+; <1s p99 latency; 24/7 support | SLA available; <2s p99; business hours support | No SLA; latency unclear; limited support | | Compliance | SOC2 Type II, HIPAA, GDPR with proof; DPA available | SOC2 Type II with report; GDPR claimed | No compliance; no certifications | | Sustainability | Profitable, well-funded, >5 years in business | Series B/C funded; 2–5 years in business | Early-stage or underfunded |
Add up scores across the five categories (max 25 points). Vendors scoring 20+ are strong; 15–20 are acceptable; <15 need significant negotiation or are not ready.
Prep before the demo
Before the demo, send the vendor your requirements:
"We're evaluating vendors for AI customer support. We resolve 5,000 tickets per month, and we need to understand cost transparency, lock-in, and data attribution. Please have a technical lead in the demo who can walk through a real example with cost breakdown."
This prevents the vendor from sending a sales-only team. It also signals that you're serious about cost and attribution.
During the demo, take notes on each question. After the demo, grade the vendor using the rubric above.
For the full evaluation process, see "How to Buy AI: The Executive's Vendor Evaluation Guide." For contract negotiation, see "How to Negotiate AI Vendor Contracts in 2026."
Want to see this in your stack?
Book a 30-minute walkthrough with a Runrate founder.
Was this article helpful?