Ayonix in NIST Face Recognition Evaluations

What is the NIST face recognition evaluation?

The National Institute of Standards and Technology, a United States federal agency, runs an ongoing independent evaluation of face recognition algorithms. Developers submit algorithms as compiled software; NIST runs them against large sequestered datasets the developers never see, and publishes comparative reports. The programme has run under the names FRVT and, more recently, FRTE.

The evaluation matters because almost nothing else in this market is independent. Vendor-published accuracy figures are measured by the vendor, on data the vendor selected, at a threshold the vendor chose. NIST removes all three of those degrees of freedom at once.

• Algorithms are tested on data the submitter has never seen, so tuning to the test set is not possible.
• All submitted algorithms face the same data, making cross-vendor comparison meaningful.
• Results are published openly, including error rates broken down by demographic group.
• Submissions are dated, so a result is tied to a specific algorithm version at a specific time.

What Ayonix's participation means, precisely

Ayonix algorithms appear in NIST face recognition evaluation publications across multiple test cycles, in both the 1:1 verification and 1:N identification tracks. That is the complete claim, and this page will not extend it. Ayonix did not take part in NIST's standalone demographic effects study: demographic figures for ayonix_000 appear inside its own verification report card because they are part of the standard template, which is not the same thing.

NIST publishes comparative evaluations and does not endorse vendors. There is no such thing as a NIST certification, a NIST approval, or a NIST licence for face recognition. Any vendor presenting a badge implying otherwise is describing something the agency does not issue.

Ayonix does not publish accuracy figures on this site, in NIST terms or any other. An error rate without its dataset, operating threshold and demographic breakdown is not a fact a buyer can act on, and quoting a favourable slice of a NIST report is the most common way that gets done. Readers who need comparative numbers should read the reports directly.

How to read a NIST face recognition report

1. Identify the track. One-to-one verification and one-to-many identification are different problems with different reports; a strong result in one says little about the other.
2. Check the algorithm submission date. Reports accumulate over years and a vendor's current algorithm may not be the one in the row being quoted.
3. Read the false-match and false-non-match rates together. An algorithm can be tuned to look excellent at one by being poor at the other.
4. Find the demographic differentials section. Error rates vary across age, sex and skin tone, and a single headline figure conceals that entirely.
5. Note the dataset. Visa photographs, mugshots and border-crossing images produce very different numbers for the same algorithm.
6. Treat the result as evidence about the algorithm, not a prediction about your deployment. Camera placement, lighting and population differ.

A NIST result is a strong signal that a vendor submitted to outside scrutiny. It is a weak predictor of how a system will behave at one particular door, in one particular building, with one particular population walking through it. Both statements are true simultaneously, and vendors tend to quote only the first.

Why Ayonix does not publish an accuracy number

An accuracy percentage is the most requested and least useful figure in biometrics. The same algorithm can be described as 99.9 percent accurate or 95 percent accurate truthfully, depending on the dataset, the matching threshold and whether the number refers to verification or identification.

• Threshold: raising it cuts false matches and raises missed matches. There is no single correct setting; it depends on whether a miss or a false alarm costs more.
• Gallery size: identification against 100 enrolled people and against 10 million are different problems.
• Image quality: a cooperative passport photo and a face captured at an angle in poor light are not comparable inputs.
• Demographics: an aggregate figure hides variation between groups, which is usually the number that matters most for fairness.

Ayonix instead offers pilots on the customer's own site, with the customer's cameras and population. A number produced that way answers the question a buyer is actually asking, which is how the system will perform where it is going to be installed.

What is the difference between FRVT and FRTE?

They are the same programme under two names. NIST ran its face recognition evaluations as FRVT — the Face Recognition Vendor Test — for many years, and has since renamed the ongoing tracks FRTE, the Face Recognition Technology Evaluation, with FATE covering analysis tasks that are not recognition. Older reports and URLs still carry the FRVT name, which is why a search turns up both, and why a vendor citing FRVT is not necessarily citing something retired.

• FRTE 1:1 — verification. Are these two images the same person? Reported with FMR and FNMR.
• FRTE 1:N — identification. Which of the N enrolled people, if any, is this? Reported with FPIR and FNIR.
• FATE — quality, morph detection, age estimation and similar tasks, evaluated separately.

The four error rates are worth keeping straight, because a vendor quotes whichever flatters. FMR is the false match rate: how often two different people are called the same. FNMR is the false non-match rate: how often the same person is not recognised. FPIR and FNIR are the identification equivalents — false positive and false negative identification rate — and unlike FMR and FNMR they move with the size of the gallery being searched, which is why an identification result cannot be inferred from a verification one.

Where to check the Ayonix results yourself

NIST publishes its participant lists and per-algorithm report cards openly. Any vendor's participation claim, this one included, should be checked at NIST rather than taken from the vendor's own site, so the exact records are named here rather than described.

• NIST FRTE 1:1 verification — participant listAyonix is listed there as Ayonix (JP).
• NIST report card for ayonix_000, 1:1 verificationDeveloper name, algorithm type and date of submission are stated at the top of the card.
• NIST FRTE 1:N identification — results and report cardsLinks the per-algorithm report cards, including those for ayonix_0, ayonix_1 and ayonix_2.
• NIST Face Technology Evaluations programme (FRTE and FATE)The programme itself, for current tracks and the reports published under them.

No figure from these reports is quoted on this page — not the error rates, and not the position in the ranked tables. The reports are linked in full instead. A vendor summarising its own evaluation results is choosing which results to summarise, and a reader is better served by the source than by the summary.

NIST publishes comparative evaluations and does not endorse vendors.

The disclaimer Ayonix attaches to every mention of NIST participation on this site