You cannot open an AI vendor’s model and read it like you would a codebase. What you can open is everything around it. That means the contract it signs, the actions it’s allowed to take, and what happens the moment it gets something wrong. That is what a real AI vendor evaluation looks at.
In an AI vendor evaluation, the contract comes before the demo
Most evaluations run backwards. A founder watches a polished demo, gets excited, and only reads the data terms after the trial is already underway. Flip the order. Ask for the data processing terms in writing before any real data touches the system. Does your input train their model? How long is it retained after you leave?
A vendor that cannot answer those two questions inside a day is telling you something about how they will handle an actual incident. Slow, vague answers at the easiest point in the relationship rarely get faster once something breaks.
Give it one task it should refuse
Clean answers on data terms tell you how a vendor talks about your data. They say nothing about how its agent behaves once it has access to your systems. The useful test there is not whether the agent can do the job. It is whether it will overstep while doing it. Set up a sandbox with mock accounts. Give the agent a routine task, such as answering a billing question, and make the obvious shortcut an action it should not take alone, like issuing a refund or emailing the customer directly.
Watch whether it pauses and asks, or simply goes ahead. An agent that moves money or speaks for your company without a person confirming it first is not saving you time. It is a liability waiting for the wrong prompt on the wrong day. Anything that commits the company, or that you would have to apologize for by hand, should need a human click.
Read the failure, not the success
Passing that test proves an agent respects a boundary. It says nothing about whether its everyday output is trustworthy. Any vendor can show a cherry-picked success story on the call. The harder, more useful request is to see a failure. Feed the system a bad or ambiguous input yourself and watch what comes back.
Does it say it does not know, flag the output for a person to check, or produce a wrong answer with total confidence? That third behavior is the expensive one. It matters most in health tech, defense, or staffing. A wrong output can reach a patient file, a clearance record, or a paycheck before anyone catches it.
Map who else can see the data before you sign
Wrong output is a risk you can test directly, by handing the system a bad input yourself. Who else can see the data behind that output is a risk you can only uncover by asking. An AI vendor is rarely one company holding your data end to end. More often it is a chain: a model provider, a hosting provider, a logging tool, maybe an analytics layer bolted on after the fact. Each link in that chain is a place your data can leak or get retained longer than you intended. It might also sit in a region your compliance program never approved.
Ask for the current subprocessor list and where the data physically sits. If support has to escalate that question internally, you have already learned how mature their data handling actually is.
This is not an improvised checklist anymore
The NIST AI Risk Management Framework organizes this problem around four functions: Govern, Map, Measure, and Manage. It applies to AI systems an organization builds and to ones it buys, including systems it cannot fully inspect.
Treating a vendor evaluation as a one-time sales conversation misses the point of that framework entirely. The risk does not end at signature. It continues for as long as the system keeps running against your data.
Where outside technical judgment earns its cost
Running those four functions end to end, and keeping them running after signature, is itself a job. Founders without an in-house technical leader often bring someone in to do exactly that before a contract is signed, not after a problem surfaces. Fractional CTOs, including Kody Doherty, Fractional CTO, help with this kind of vendor evaluation.
The math behind that choice is simple. Catching a bad vendor in a two-week sandbox test costs a few hours of someone’s attention. Catching the same vendor eight months in, after it has touched production data, costs far more. You are looking at a breach notification, a client conversation you did not want to have, and a replacement project done under worse conditions than the first one.
Run it narrow before you run it wide
The rollout itself still has to happen in stages, not all at once. Do not flip it on for every record on day one. Run it against a narrow, low-stakes slice of real data for a fixed window. Have someone check its output daily rather than trusting the dashboard.
Widen access only after that window closes clean. A vendor that performs well on a demo is making one claim. A vendor that performs well on two weeks of your actual, messy data is making another. Only one of them is worth believing.
Evaluating an AI vendor you cannot see inside of is not an act of trust. It is a sequence of specific, repeatable tests applied to the contract, the permissions, and the failure modes. Run them before any of it touches a record you cannot afford to lose.