Write the problem, not the wish list
12 min
Most evaluations are lost before they start. The problem is never written down, the shortlist is inherited from a peer at a conference, and the calendar is filled with demos the vendors control.
One page, before anything else
Write a single page that answers three questions.
- What is broken today? Describe the failure, not the missing feature. "We find out about endpoint compromise from the user, four hours in" is a problem. "We do not have EDR" is a shortlist.
- What does it cost? Hours, incidents, audit findings, missed maintenance windows. A number you can defend beats an adjective.
- What does fixed look like in twelve months? If you cannot describe the changed world, you cannot tell whether any product got you there.
Five must-haves, five nice-to-haves
Rank them. A requirement you cannot rank is not a requirement — it is a preference someone has not admitted to yet. Must-haves become pass/fail gates; nice-to-haves become tie-breakers, and never the other way round.
Name the team before the vendors arrive
Four people, and each one has a job:
- an owner, who will be accountable for the decision;
- an engineer, who will actually run the product every day;
- someone from procurement or finance, early rather than at signature;
- a sceptic, whose explicit job is to argue for the option you are about to reject.
The sceptic is not a formality. An evaluation with nobody assigned to disagree will produce consensus regardless of the evidence.
Build the scorecard now
Before any demo, write down how you will score. The five axes DBSE uses in reviews work as a backbone: deployment effort, support quality, features versus promises, pricing transparency, and return on investment. Add your must-haves as gates.
Scoring criteria written after the demos will be scoring criteria written to match the demo you enjoyed most.