How to run a face recognition pilot

Why run a pilot rather than compare accuracy figures?

Because accuracy figures are not portable. The same algorithm produces different error rates on visa photographs, mugshots and border-crossing images, and different rates again at a gate where the subject stops than in a corridor where they walk past at an angle. A number quoted without its dataset, its threshold and its capture geometry is not a claim a buyer can act on.

A pilot substitutes the buyer's own conditions for the vendor's. That makes the result specific and usable, and it also makes it uncomfortable — which is the reason to insist on it. A vendor who resists a pilot on your cameras is telling you something.

• Why this site publishes no accuracy percentageThe longer argument, and what to ask a vendor who answers every question with one.

What should be agreed before the pilot starts?

Everything below should be written down and signed by both sides before any equipment is installed. This is the step that decides whether the pilot produces a decision or an argument.

1. The task. One-to-one verification, one-to-many identification, or watchlist alerting — they are different problems with different failure modes, and a pilot that does not name one will end up measuring whichever went best.
2. The population and its size. How many people are enrolled, and how many distinct individuals will pass the capture point during the pilot.
3. The capture points, with their geometry and lighting recorded, and agreement that they will not be changed mid-pilot.
4. The threshold, or the process for setting it. If the vendor tunes the threshold during the pilot, the results before and after are not comparable.
5. The acceptance criteria, as numbers with a direction: what false-match rate and what missed-match rate are acceptable, at what throughput.
6. The fallback. What happens when recognition fails, who handles it, and whether the time it takes is part of what is being measured.
7. Who owns the data collected, how long it is kept, and when it is destroyed.

The threshold item is the one most often left open and most often decisive. Raising a threshold cuts false matches and raises missed matches; lowering it does the reverse. A vendor allowed to choose it after seeing the data can present almost any pilot as a success.

How many subjects and how long?

The honest answer is that it depends on the error rate you are trying to detect, and that most pilots are too small to distinguish a good system from a mediocre one. The arithmetic is worth doing rather than guessing at.

• To observe a failure mode at all, you need enough attempts that it would be expected to occur several times. If missed matches are expected in roughly 1 in 100 attempts, 50 attempts will frequently show none — and prove nothing.
• False matches need far more data than missed matches, because they are rarer. Detecting a false-match problem usually needs a gallery and a volume of non-matching attempts that a two-week pilot does not naturally produce.
• Run across the full daily and weekly cycle. A pilot that runs only during office hours misses the lighting conditions that break entrances.
• Include the population that fails. Children, wheelchair users, people in lawful head coverings, people wearing masks or heavy glasses — a pilot drawn from volunteers who are all staff aged 25 to 50 will overstate performance.

Where the volume for a statistically meaningful false-match measurement is genuinely unavailable, say so in the report rather than reporting a number the sample cannot support. A pilot that concludes "we could not measure this" is more useful than one that reports a rate with no power behind it.

What should be measured?

How should the results be reported?

A pilot report that a buyer can act on has a shape. It states the conditions, then the numbers, then what the numbers do not cover — in that order, because a number read without its conditions is what produces the accuracy claims this industry is full of.

1. The conditions: cameras, placement, lighting, threshold, gallery size, dates and times.
2. The counts, raw: attempts, matches, missed matches, false matches, failures to acquire. Rates are derived from these; publishing only rates hides the sample size.
3. The breakdown by subgroup, wherever the sample allows it.
4. What could not be measured, and why.
5. What would have to change to meet the acceptance criteria, if they were not met.

A pilot result belongs to the site it was run at. It does not transfer to another building, another camera set or another population, and a vendor quoting your pilot result to their next customer is misrepresenting it — including if that vendor is Ayonix.

What does a pilot not tell you?

• How the system behaves at full scale. A gallery of 500 and a gallery of 500,000 are different problems, and identification error rates rise with gallery size.
• How it degrades. Two weeks does not reveal what happens as enrolment images age, as the population changes, or as a camera drifts out of focus.
• How it behaves under attack. Presentation-attack resistance needs deliberate adversarial testing, not incidental observation.
• Whether operators will use it as designed. Automation bias appears under time pressure, which a pilot with observers present tends not to produce.

These are reasons to plan a review after deployment rather than reasons to skip the pilot. The pilot answers whether the system can work here; the review answers whether it still does.