Proof, not demo

A demo is a film about a product. A proof of concept is the product, in your environment, doing your work.

Script it once, run it three times

Write the script before the first vendor arrives, and give every vendor the same one: your data, your alerts, your identity provider, your ticketing tool, your change process. Anything a vendor needs to change about the script should be recorded, because "we need a different data source to look good" is itself a finding.

Count the hours

Record who from the vendor touched the environment and how many hours your own engineer spent. That number is the first honest draft of deployment effort, and it is the number that never appears in a proposal.

A product that needs two vendor engineers for a week to look good in a proof of concept will need them again at renewal, at the next major version, and on the day the person who built it leaves.

Reference calls you choose

Two per vendor, with customers you pick — from the review pool, from your own network, from anywhere except the list the vendor hands you. That list is a marketing asset, and it is selected on precisely one criterion.

Ask the reference: what surprised you after signature? What does support look like at 3am? What did the renewal quote look like against the first one?

Score it live

Fill the scorecard during the proof of concept, not from memory afterwards. Memory rewards the vendor whose engineer was most likeable, which is a real effect and not a small one.