The 3 Questions You Must Ask Your AI Vendor Before Signing

The AI vendor sales cycle is accelerating. Procurement teams are being pressured into decisions that were once made over months, now compressed into weeks. Contracts are being signed with models that shift underfoot, data pipelines that remain opaque, and liability clauses buried on page forty-seven. The result is a growing number of organizations discovering, after the ink dries, that they purchased something fundamentally different from what was demonstrated in the pitch.

The problem is not that buyers are careless. The problem is that the conversation is structured around the wrong questions. Most procurement dialogues orbit price, latency, and benchmark scores. Those matter, but they are symptoms, not causes. The three questions below get at the structural realities that determine whether an AI engagement becomes a durable asset or an expensive experiment that quietly erodes in value over eighteen months.

Question 1: How Do You Handle What Happens When the Model Is Wrong?

Every AI system fails. The question is not whether your vendor's model will produce an incorrect, harmful, or nonsensical output. It will. The question is what happens in the architecture, the workflow, and the contractual relationship when that failure occurs.

The technical layer. Ask your vendor to walk through their error-handling pipeline in concrete terms. When the model hallucinates a citation, what happens? Is there a confidence threshold that routes low-confidence outputs to a human reviewer? Is there a retrieval verification step that cross-checks generated claims against a grounded corpus? Does the system flag uncertainty, or does it present every output with equal authority?

Vendors who answer with "we use state-of-the-art models" or "we benchmark against industry standards" are describing the happy path. You need the unhappy path. Ask specifically: what is the maximum acceptable error rate for the use case you are deploying, and what mechanism guarantees that threshold is never exceeded in production?

The operational layer. Errors will reach end users regardless of how good the internal guardrails are. What is the escalation path? How quickly can a bad output be identified and suppressed? Is there a kill switch that halts the system without requiring a vendor call? Can your team override, correct, or suppress individual outputs in real time?

The contractual layer. This is where most buyers lose leverage before the contract is even signed. If the AI produces output that causes financial loss, regulatory exposure, or reputational damage, who bears that cost? Standard vendor agreements allocate liability to the customer for "how the output is used," which effectively means the vendor can generate a defamatory statement about a customer's client and the buyer absorbs the legal fallout. Push for language that ties vendor liability to model behavior, not just model availability. If the model drifts, degrades, or produces systematically biased output, the vendor's SLA should reflect that this is a service failure, not a customer misuse event.

A vendor that is comfortable with your error-handling question will have documentation, architecture diagrams, and named engineers who can discuss failure modes in detail. A vendor that is uncomfortable will pivot to pricing tiers.

Question 2: What Do You Actually Know About My Data, and What Do You Do With It?

"Data" in an AI vendor conversation refers to three distinct things that are routinely conflated in sales materials. Separating them is essential.

Your training data. Does the vendor use your data, your prompts, your outputs, or your metadata to improve their model? "Improvement" can mean many things: fine-tuning a shared model on your proprietary patterns, using your usage data to retrain base weights, or simply logging your inputs for internal product analytics. Each of these has different risk profiles. A model that has been fine-tuned on your customer's data and then resold to a competitor is a data leak with a service-level agreement attached.

Ask for the specific data flow diagram. Not a marketing one-pager. The actual architecture: what leaves your environment, in what format, to what destination, for what purpose, and for how long. If the answer is "we don't retain your data" but the model's behavior reflects your data patterns across multiple customers, the data is being retained in the weights. That is not a deletion. That is a transformation.

Your inference data. When your users interact with the system, what is captured? Prompts, responses, metadata, user identifiers, timestamps, session context? Is this data used for product improvement, quality assurance, billing, or model retraining? The answers should be granular. "We may use data for service improvement" is a clause that gives the vendor the right to use everything for anything indefinitely.

Your competitive data. This is the one most buyers forget to ask about. If your AI vendor serves your three largest competitors, and the model is multi-tenant, what isolation guarantees exist? Not at the database level, which is standard. At the model level. Does your competitor's usage pattern influence the weights that serve your requests? In multi-tenant fine-tuning scenarios, the answer can be yes, and the information transfer is not something you can audit from your side.

The practical test: ask the vendor to run a data isolation audit on a specific model instance you would be deployed on. If they resist or the answer is "our infrastructure is certified" without specifying which certification, for which data path, under which regulatory framework, you are being told to trust a process you have not seen.

Question 3: What Is the Real Cost of Leaving?

This question is not about contractual exit fees, though those matter. It is about the structural lock-in that makes leaving expensive in ways that are not visible on a price sheet.

Model lock-in. If you build a workflow, a prompt engineering layer, a RAG pipeline, or a fine-tuned adapter around Vendor A's model, that investment has a half-life. When you move to Vendor B, the prompts that worked may produce different outputs. The retrieval context windows may differ. The tokenization scheme changes, which changes cost per query in non-obvious ways. The fine-tuned adapter is useless. The evaluation suite you built is calibrated to a specific model's failure distribution and will not transfer.

Ask the vendor: what is the expected migration cost for a customer of your size, expressed in engineering weeks, not dollars? If they cannot answer, they have not thought about it, which means they have not thought about retention. A vendor that has thought about your exit is a vendor that has thought about the product.

Data lock-in. Your data, once ingested into their pipeline, may exist in formats that are specific to their system. Vector embeddings from one embedding model are not directly transferable to another without re-embedding, which means reprocessing your entire corpus. If your corpus is 50 million documents, re-embedding is a six-month project at current compute prices. The vendor knows this. The price of your first year may be subsidizing the price of your third year, when switching costs make renewal a non-decision.

Talent lock-in. Your engineers have learned the vendor's API, its SDK, its error codes, its debugging tools, its operational patterns. That expertise is specific. When you evaluate Vendor B, your team's knowledge of Vendor A's system is an asset only insofar as it helps you articulate what you need from Vendor B. The actual implementation expertise transfers poorly.

The practical test: negotiate a data export clause that specifies format, completeness, and timing. Not "we will provide access to your data" but "we will provide a complete export in [specific format] within [specific number] business days of termination, including all embeddings, fine-tuned weights, and prompt configurations." If the vendor resists specifying the format, they are protecting the switching cost.

The Meta-Question

These three questions share a common thread: they are questions about what happens when things go wrong, when the relationship ends, or when the model does something nobody predicted. Vendors are excellent at answering questions about what happens when things go right. The demos are polished. The benchmarks are cherry-picked. The latency numbers are measured under ideal conditions.

The organizations that deploy AI successfully are not the ones with the most sophisticated models. They are the ones whose vendors could be replaced, whose data could be moved, and whose failure modes were understood before the first production request was made. The contract is not the starting point of the relationship. The contract is the document that describes what happens when the relationship fails.

Ask the three questions. Listen to the quality of the answers, not the speed. The best vendors are not the ones who answer fastest. They are the ones who answer completely, including the parts that make their product look smaller than the demo suggested.