Straight Fact · desk research and BespokeData methodology · 18 Nov 2026
An AI pilot needs a baseline
3 min read · Free article
An impressive demonstration cannot establish whether AI improves a business process. The relevant comparison is with what would otherwise happen: the current manual approach, a simple rule or no intervention. Decide that baseline before tuning the candidate or choosing favourable examples. A pilot without a comparison can produce many outputs and very little decision evidence. It may show that a system runs, but not that it is useful enough to justify its risks and operating effort.
NIST's voluntary AI RMF includes measurement within a wider risk-management structure [N2]. Our practical application is to evaluate both usefulness and consequential errors in the intended setting. This is not NIST certification or an endorsement of a particular test. The owner must choose context-appropriate measures and decide what evidence is sufficient to proceed, revise or stop.
Suppose a hypothetical test contains 100 orders, including 20 that later become late. A simple rule flags 30 orders and catches 12 of the 20 delays. Its recall is 12 divided by 20, or 60%; precision is 12 divided by 30, or 40%. An AI candidate flags 40 and catches 16 delays. Recall rises to 80%, while precision stays at 40%. It catches four more delays but also produces six more false alerts: 24 rather than 18.
That trade-off may be worthwhile, but only if the team has capacity to review the extra alerts and the missed delays have sufficient consequences. Measure review time and the action taken, not just predictions. Keep the same frozen evaluation cases for both approaches and separate them from tuning examples. If the candidate saw the answers during development, its apparent advantage is not a clean estimate of future performance.
Before the test, write a short protocol: authorised data, baseline rule, evaluation period, error definitions, review owner and stop conditions. Include hard cases and explain how excluded records are handled. Use shadow operation where appropriate so outputs do not automatically change customer commitments. Review false alerts and misses with the people who understand the process; an overall percentage can hide a serious failure in a small group.
This example is invented to explain arithmetic, not to claim an AI success rate. A historical test cannot guarantee future results, and changes in order mix or systems may invalidate it. The next action is to document one reproducible baseline and a named human decision owner. The full readiness guide provides a worksheet and go/no-go questions for deciding whether a limited pilot should begin at all.
Dated method & source register
Read and reviewed 18 Nov 2026. Desk research and original BespokeData recommendations and hypothetical examples; no primary study, independent expert review or Atora generation.
- [N2] NIST · AI RMF 1.0 (January 2023)
Core Govern, Map, Measure and Manage functions; contextual risk guidance.