FBI Open-Source Facial Recognition RFI (FBI-BSS-FRT): Two Accuracy Questions

Date: August 30, 2026 · Author: Dmitrii Zatona 

The FBI’s market research on open-source face recognition capabilities, posted in September 2024, asks respondents about accuracy twice. Item 6 asks for “any accuracy results for the FRT that would have been generated by National Institute of Standards and Technology’s (NIST’s) Face Recognition Technology Evaluation (FRTE) program.” Item 4 asks for “Specific information on accuracy and error rates of FR algorithm(s) or FRT used in the application in conjunction with the open-source data.” The first names an evaluation whose measured object is fixed by the way that evaluation is built: a named, dated algorithm build, run by NIST behind a NIST-specified interface on NIST-held datasets, at operating points NIST chooses and reports. The second names no metric, no test protocol, no dataset, and no operating point. Whether two answers arriving under those items would be statements about the same measured object is not something the published text lets a reader establish — and the information that would settle it is not specifically requested or enumerated anywhere in the twelve items (item 9’s open field for “any other assumptions, comments, or supporting information” permits a respondent to volunteer it). It did not have to be: this is a request for information, not a performance specification. What follows is about what comparability between accuracy statements takes, and which of those pieces the reviewed documents supply.

1. What was published

The notice is FBI-BSS-FRT , “FBI BSS Facial Recognition Technology,” posted to SAM.gov on September 16, 2024 at 11:48 Eastern. SAM classifies it as a Sources Sought notice; its single attachment titles itself “REQUEST FOR INFORMATION (RFI) — Open-Source Facial Recognition (FR) Capabilities,” six pages, letterhead “Federal Bureau of Investigation — Finance and Facilities Division | Procurement Section” on every page. Requests for clarification were due October 1, 2024; responses by noon Eastern on October 15, 2024, capped at fifteen pages inclusive of cover page and table of contents. The notice auto-archived on October 30, 2024.

The record is plain. The history endpoint returns exactly one entry — the initial publish — and the search record shows zero modifications: no revisions, no cancellation, no parent or related notices, an empty award field. It carries no NAICS code, no product service code and no set-aside in its record data (verified as a property of the record rather than a retrieval artifact; the method is documented in the research ledger). The attachment instead asks respondents to self-report “Applicable NAICS Codes.” Place of performance is Clarksburg, WV 26306. The point of contact is Brandon James, bmjames@fbi.gov, listed on the PDF’s last page as “Contracting Point of Contact” — a procurement-section contact, and nothing below treats him as a program contact. The SAM description is two sentences of mission context reproduced from the attachment’s background section; it contains no measurement vocabulary at all.

Scope guard. The document says what it is, twice, in its own capitals (p. 2):

“THIS NOTICE IS FOR INFORMATIONAL PURPOSES ONLY. THIS IS NOT A REQUEST FOR PROPOSAL OR QUOTE. IT DOES NOT CONSTITUTE A SOLICITATION AND SHALL NOT BE CONSTRUED AS A COMMITMENT BY THE GOVERNMENT. RESPONSES IN ANY FORM ARE NOT OFFERS AND THE GOVERNMENT IS UNDER NO OBLIGATION TO AWARD A CONTRACT AS A RESULT OF THIS ANNOUNCEMENT.”

Page 5 repeats it and cites FAR 15.201(e). Everything below reads an information request on those terms: the notice is under no obligation to specify a metric, name a protocol or fix an operating point, and no statement in this article is a finding that it failed to. The subject is narrower and, for anyone who would have to answer such items, more practical — given what was asked, what would make the answers comparable, to each other and to the one evaluation the document names. Throughout, the twelve numbered asks are called information items, what a future evaluation would have to settle comparability choices; the word “requirement” appears only in quotations and in this series’ section headings.

Status, as a bounded fact about this notice’s own record. As of August 30, 2026 the notice’s own record shows no continuation: its related and award fields are empty, and it archived automatically on October 30, 2024. That negation is weak in both directions, because the notice’s own respondent form asks about “GSA Schedule” and “NASA SEWP Contract” — channels through which an order may proceed without a SAM notice.

Method note: this is the eighth teardown in a series (1 : whether a named primitive exists on the shelf; 2 : which assurance tier a notice’s words select; 3 : where the platform boundary sits; 4 : which operational choices a traceability sentence leaves open; 5 : who chooses the policy under which evidence composes; 6 : what anchors a biconditional guarantee; 7 : which referent a compound quality phrase names). The test here is comparability, a step earlier than the second article’s ladder: there one property had several assurance tiers and the notice selected none; here two pieces of requested information are both about accuracy, one attached to an evaluation whose object is fixed by construction, the other tied by the text to no measured object, and whether they coincide cannot be established from what was published.

2. What the requirement says

The document’s scope is set in one sentence on page 2, and it is the only place the output of the searched capability is described:

“Responses to this RFI are intended to identify capable parties who (1) conduct ongoing collection of open-source and publicly available images and (2) offer a face recognition (FR) capability to customers enabling them to search their face images against the collected open-source and publicly available images and receive a list of highest probably match candidates.”

The output phrase is quoted exactly as printed, spelling included (sic). The stated downstream use of responses, same page: “The information gathered may be used to support future acquisition strategies for competitive solicitations.”

Eligibility and delivery model, page 3: the RFI is “open to any interested party with current capabilities to supply law enforcement customers with robust, open-source and publicly available face image repositories and functioning FR capabilities to search the repository,” and responses “should identify the available FR tools, search engines, capabilities, and open-source repositories that must work in tandem to provide the capability as a Software As A Service (SAAS).” “Robust” and “functioning” are the only quality adjectives the document applies to the repository and the search capability.

Then twelve numbered information items:

Two facts about the corpus, both bounded by a full case-insensitive reading of the six pages and the SAM description, substring false positives excluded. First, item 6 is the only reference to a standards body or an evaluation programme anywhere in the notice or the attachment. The strings FRVT, ISO/IEC, EBTS, ANSI/NIST-ITL and FISWG do not appear; neither do threshold, FMR, FNMR, FNIR, FPIR, 1:1, 1:N, rank, probability, gallery, probe, enrollment, benchmark, test protocol, human review, audit, certification, training data, demographic, retention, or the word “biometric.” “Accuracy” occurs twice — items 4 and 6; “error rates” once, in item 4; “candidate” once, in the sentence quoted above.

Second, a fact of form rather than a complaint: “shall” occurs nine times, and every occurrence is directed at the response document — succinctness, classification marking, proprietary marking, pricing detail, page limits — or sits inside the boilerplate disclaimers. The single “must” is “must work in tandem,” about tools and repositories composing into a SaaS.

That is the whole measurement vocabulary of the notice, and it is why the comparability question arrives this early. Item 6 attaches its request to an evaluation whose measured object is fixed by that programme’s design. Item 4 attaches its request to “the open-source data” — a phrase identifying whose data is meant, but not what would be measured on it, under which protocol, or at which operating point. The text neither equates the two nor separates them, and supplies nothing that would let a reader decide.

3. What an accuracy figure is a figure about

The following five-part frame is this article’s organizing abstraction, marked (I); no reviewed document lays it out this way. An accuracy statement about face recognition search is a statement about a tuple, and two such statements are comparable only when their tuples agree in the places that matter:

1. The algorithm build — a specific, dated software artifact, not a product name.
2. The gallery — its provenance, composition and size N.
3. The operating point — the rank limit R (or list length L) and the score threshold T at which errors are counted.
4. The evaluation type — which class of test produced the figure.
5. Who ran it, on whose data, and what was reported.

The third element is why an accuracy figure without an operating point cannot be compared with, or interpreted against, another: identification metrics are parameterized as FNIR(N, R, T) and FPIR(N, T), and moving T trades one against the other. The fourth element is not an invention of this article. ISO/IEC 19795-1:2021, the framework part of the biometric performance testing series, defines three evaluation types in its terms clause: technology evaluation (3.13), an offline evaluation of algorithms over a corpus; scenario evaluation (3.14), end-to-end performance in a prototype or simulated application with a test crew; and operational evaluation (3.15) — performance measured in a specific application environment with a specific target population, in the standard’s paraphrased terms. They answer different questions; treating figures from different types as inputs to one comparison, without matching their stated conditions, is the mismatch this frame exists to make visible — an inference (I) of this article, not a clause of the standard.

What an FRTE result attests. The evaluation item 6 names is a technology evaluation in exactly the 3.13 sense, and its object is fixed by construction. Participants submit compiled black-box libraries implementing a NIST-specified C++ test interface; NIST states in the identification report that it “therefore does not describe how algorithms operate.” The six datasets — frontal and profile mugshots, desktop webcam photos, visa-like application photos, immigration lane photos, kiosk photos — are, in NIST’s words, “sequestered at NIST, meaning that developers do not have access to them for training or testing.” NIST chooses the enrolled population sizes and the operating points, and reports FNMR at a stated FMR for the 1:1 verification track , FNIR at stated (N, R, T) for the 1:N identification track . An FRTE result therefore attests one named, dated build’s behaviour on NIST’s galleries at NIST’s operating points. By design it attests nothing about a gallery NIST did not run, and nothing about a hosted service: the object under test is a library behind an API, not a deployment. The report says as much about its own shelf life — “FRVT reports are but a snapshot of contemporary capability.”

Of the two tracks, the one that fits a search returning a candidate list is 1:N identification: only there do the reported parameters FNIR/FPIR and the two reporting regimes appear — Identification with T greater than zero, and Investigation with the threshold set to zero and a fixed number of candidates returned. Verification reports different metrics for a different transaction. That is why the 1:N track carries the discussion below.

What item 4’s figure would be a figure about. Elements 1 and 2 are gestured at — the algorithm or FRT “used in the application,” running “in conjunction with the open-source data” — while 3, 4 and 5 are unstated, as are the gallery’s composition and size. If a future acquisition wanted an end-to-end claim about performance in a specific deployment against a specific population, ISO/IEC 19795-1 names that class: operational evaluation, term 3.15. The RFI does not say that this is what item 4 means, and does not specifically ask respondents to state which type of evaluation their figure came from, by what method, against what ground truth, or indexed to which gallery state — item 9’s open field permits volunteering any of it.

The finding, in its narrowest form: item 6’s object is known because FRTE fixes it; item 4’s object is not identified by the text; and the information that would place the two on the same footing — evaluation type, method, ground truth, the gallery state behind a figure — is not specifically requested or enumerated (item 9’s open field permits volunteering it). Nothing here establishes that the two objects differ. What the reviewed documents establish is that the published text does not let anyone determine whether they coincide.

4. Four comparability choices

First the positive, because it changes the shape of everything after it. A method class for the second kind of statement exists on the shelf. ISO/IEC 19795-6:2012, Testing methodologies for operational evaluation, is the part of the series on point: per its published abstract it provides guidance on operational testing, specifies metrics for operational systems, details data that may be retained to enable performance monitoring, and sets out test methods, recording and reporting rules. It is guidance-level for the test itself, and was last reviewed and confirmed in 2024. At practitioner level, FISWG’s Face Recognition Systems Operational Assurance: Deployment Testing v1.0 supplies procedures that operate on the deployed gallery itself, sampling a fixed percentage of the deployed enrollments for testing. It was approved November 21, 2025, fourteen months after this notice, and appears here only as current methodological context — never as something the RFI could have named. The gap below is method-shaped, not method-absent.

The four choices that follow are questions a future evaluation would have to answer, not defects in a market-research document. Each rests on a premise the reviewed record does not establish; those premises are marked where they occur.

4.1 Identity ground truth. The identification error metrics cited in this article — FNIR and FPIR as ISO/IEC 19795-1:2021 defines them, and the deployment-testing procedures FISWG describes — count errors against known mated and non-mated status: FISWG’s sampling presupposes known groups known to be present in the deployed gallery, and 19795-1’s definitions are stated over enrolled subjects with known status. (ISO/IEC 19795-6’s own metrics are known here only through its abstract — a bound carried through Section 8.) Where that status comes from is the open question. Premise, not established: the RFI does not state how identities in a repository are established — item 3 asks for the “Nature and source of the data available in the repository,” and no answer appears in the reviewed public record — so whether such a repository carries enrollment-time identity assertions is unknown from the reviewed record. If it does not, then among the documents surveyed here none supplies a procedure for establishing identity ground truth inside it. A conditional statement about a method gap, not a claim about anyone’s data.

4.2 The gallery state behind a stated rate. Identification error rates are defined as functions of the enrolled population size N — that is what the (N, R, T) parameterization means — so a stated rate is a statement about a particular gallery state, and comparing two rates requires knowing the states each was measured on. That is a property of the metrics, not of anyone’s repository: nothing here infers anything about any repository’s contents. The RFI did not specifically ask for the gallery state behind a figure, and had no obligation to.

4.3 What the returned list is, as a measurement object. For a candidate list to be measurable the reviewed documents need three parameters: the enrolled population N, the list length or rank limit R, and the threshold T. ISO/IEC 19795-1:2021 defines rank as a candidate’s position in a list ordered by descending similarity score (paraphrase of term 3.24); FISWG’s Scoring Thresholds document gives the practitioner mechanics in the same terms — list length set by the user, candidates sorted highest score first. The RFI’s output description fixes none of the three, and no reviewed document defines the notice’s output phrase as a reportable object.

One consequence holds only under an assumption the notice does not make. Premise, not established: the RFI does not say whether an implementation returns a fixed-length list or applies a threshold. If a system always returns a fixed number of candidates without applying a threshold on scores, then per ISO/IEC 19795-1:2021’s own note, FPIR “is not a meaningful metric” for it — and the false-positive half of “accuracy and error rates” has no defined metric until the list parameters are fixed. If the system applies a threshold, the metric is available and the threshold is part of the answer. FRTE’s identification track reports both regimes because both are real: threshold-limited lists on one side, a fixed fifty candidates at T = 0 on the other, where NIST states that “it is assumed and necessary that a human will be used to review the candidates returned from each search.” The RFI states neither regime, so which branch applies to a given respondent is not determinable from the notice.

4.4 The template question is two separable questions. Item 11 asks whether searching is possible without submitting an actual face photo, giving templating or encoding inside the “client owned IT boundary” as its example. Those are two questions with different answerability.

(a) Transport topology — whether a face photo leaves the client’s boundary, and what is transmitted instead — is a property of the deployed integration, checkable on its own terms by observing the interface. Nothing below weakens that: a respondent can answer item 11 as asked, and the answer is verifiable independently of (b).

(b) Properties of the transmitted encoding — whether the original image can be reconstructed from it, and whether two encodings of the same person can be linked — the RFI does not ask about, and here the shelf’s shape is specific. In the corpus surveyed, no document standardizes the content, format or interchange of face recognition feature templates; NIST states the reason in the identification report, verbatim: “the templates that hold features extracted from face images are entirely proprietary opaque binary data that embed considerable intellectual property of the developer,” concluding that “there is no prospect of a standard template” and that interoperability “remains solidly based on images.” ISO/IEC 24745:2022 does define the property in question, in term 3.26: irreversibility, of a transform whose output “cannot be used to determine any information about” the generative data. ISO/IEC 30136:2018 defines metrics for testing template protection schemes. Neither, in the texts retrieved, establishes a certification programme or names a body that certifies a vendor’s template as irreversible; both concern deliberately protected references, and no reviewed document states that an ordinary proprietary feature template has the 24745 property.

5. Scale

The figures below come from agency and peer-reviewed publications, each with the conditions under which it was measured. They are not set against one another: each measures its own object, and no arithmetic is performed on any of them.

Enrolled population sizes in the FRTE 1:N evaluation. NIST’s identification report (NISTIR 8271 , draft supplement dated August 4, 2026) documents its own galleries: mugshot trials at enrollment populations of 640,000, 1.6 million, 3 million, 6 million and 12 million subjects, one image each; a webcam probe set of 100,000 images; a mugshot-ageing trial over 3,068,801 enrolled subjects with 10,951,064 mate searches, against a stated statistical floor of 154,549 mugshot searches. These are NIST-held sequestered sets of stated provenance and type; the numbers describe them and nothing else.

How error rates move with gallery size. In the same report, “FNIR(N, R) generally grows as a power law, aN^b,” with NIST noting that b is often much less than 1, “implies very benign growth in FNIR.” Threshold-based rates behave likewise, “except at the high threshold values corresponding to low FPIR values” — NIST’s stated example is FPIR = 0.001, where rates “increase more rapidly with N above 3 000 000.” The condition attached to that figure is the whole of its meaning: it is measured on NIST’s mugshot enrollment sets at one stated false-positive operating point.

How much the input class matters. In the same evaluation: “Searches of webcam images give FNIR(N, T) values around 2 to 3 times higher than mugshot searches.” Two image classes, same evaluation, same algorithms.

One benchmark, two conditions. The peer-reviewed MegaFace benchmark  (CVPR 2016) was built to vary one thing — the number of distractors in the gallery — and measure the same algorithms under both conditions. Its reported result: algorithms above 95% on LFW, which the paper describes as the “equivalent of 10 distractors in our plots,” return 35–75% identification rates with 1M distractors. The benchmark is a million photos of more than 690,000 subjects. One paper’s measurement of its own algorithms on its own gallery, ten years old, cited for the single reason that it isolates the gallery-size condition inside one controlled comparison.

Together these make a narrow point about the frame in Section 3: within a single evaluation, a change of gallery size or of input class moves the reported figure. That is why an accuracy statement travels with its tuple, and why two statements whose tuples are unknown cannot be placed on one axis.

6. The record

What follows is a dated public record, in two groups. The documents do not connect these groups to each other, and no reviewed document connects any of them to the RFI. No causal statement is made below, and none is available from these sources.

The privacy documents. Two DOJ/FBI Privacy Impact Assessments bracket the notice in time. The PIA for the FACE Operations Services , approved June 24, 2024, describes searches against government-held repositories and states that the service “does not search probe photographs against any private, public, or non-governmental photograph system.” The PIA for the NGI Interstate Photo System , approved November 7, 2024, describes a facial recognition search capability over photographs submitted with tenprint fingerprints. Both are cited as dated documents with their own stated scope. Neither mentions this notice, the notice mentions neither, and no inference is drawn here from their proximity in time.

The oversight record. GAO-21-518 , June 3, 2021; its recommendation-status record notes an FBI facial recognition procurement, tracking and evaluation policy directive issued in August 2022, and risk assessments for the Bureau’s use of multiple commercial systems provided to GAO by June 2025. GAO-23-105607 , September 5, 2023, records approval of an FBI Facial Recognition Technology Use Policy Directive in December 2023. Dates and directives only; the counts in those reports are not reproduced here, because they measure federal usage and establish nothing about the measurement questions this article is about.

7. Who would build it

What is requested. The addressee is stated in the eligibility sentence: any party that, at the time of the notice, both maintains open-source and publicly available face image repositories and offers a search capability over them, delivered as SaaS. The response rules narrow the field further — fifteen pages inclusive of front matter, no proposals or offers, no past performance information or general descriptions of corporate experience, no requests to be considered for award, no telephone responses. The Required Information block asks for the usual identifiers plus Facility Clearance if applicable, GSA Schedule and NASA SEWP Contract. The government “reserves the right to conduct demonstrations or follow-up with clarifying questions,” with no criteria stated for either.

What is documented. The oversight documents named in Section 6 record, as dates and directives only, that federal use of commercial face recognition services has been the subject of review. Whether any vendor or service matches the full profile the eligibility sentence describes is not established by any reviewed record, and nothing in this article characterizes any vendor or product.

What is unproven. Whether anyone responded, how many did, what they submitted, and whether any of it informed a later acquisition appear in no reviewed public record as of August 30, 2026. The notice states only that the information gathered “may be used to support future acquisition strategies for competitive solicitations.” Roles are not assigned here, because the public record assigns none.

8. What does not exist as a standard

Each row states the version accessed on August 30, 2026 and its status relative to the notice date of September 16, 2024. Documents published after that date appear as current methodological context, never as something the notice could have named.

What the shelf holds is worth stating as plainly as what it does not. The evaluation named in item 6 publishes its interface, its datasets’ provenance and sizes, its metrics and its operating points, and a result carries its conditions with it. The taxonomy separating that kind of measurement from a deployment measurement is standardized, and so, at guidance level, is the methodology for the second kind — with practitioner procedures added since. A future evaluation wanting comparable answers would be choosing among published methods, not inventing one.

Four bounded negations, with their corpus stated. Among the documents surveyed here — the FRTE 1:1 and 1:N track pages and NISTIR 8271 (2026-08-04 supplement); ISO/IEC 19795-1:2021, 19795-2:2007 and 19795-6:2012; ISO/IEC 39794-5:2019 and 19794-5:2011; the ANSI/NIST-ITL programme page; FBI EBTS v11.2 and v11.3; ISO/IEC 24745:2022 and 30136:2018; and the FISWG published-documents list with two documents read in full — (1) none, in the texts retrieved (19795-6 at abstract level, as its row states), supplies a procedure for establishing identity ground truth in a gallery assembled without enrollment-time identity assertions; (2) none defines criteria, metrics or conformance tests for a face image repository as a data asset — the notice’s word “robust” is not given criteria by any reviewed document; (3) none defines the notice’s output phrase as a reportable object, and the parameters a candidate list needs — N, R, T — are defined but left unfixed by the notice; (4) none establishes certification of irreversibility for a face template, or states that an ordinary proprietary feature template has that property. I work on verifiable evidence chains and provenance records in an adjacent area, and that work is separate from this analysis.

Two accuracy questions, two items apart on the same page of a six-page document. One points at an evaluation that publishes exactly what it measured, on what, and at which operating point; the other points at data whose measurement conditions the text does not state. What separates the two kinds of answer is made of specific, published, choosable things: an evaluation type, a test method, a ground-truth procedure, the gallery state behind a figure, three list parameters. A market-research notice is entitled to ask its questions without settling any of them, and its open ninth item permits respondents to volunteer what the others do not enumerate. The record as it archived on October 30, 2024 settles none of the five, and leaves whoever reads the answers to work out what they are answers about.

If you are building a response to this document and the provenance or attestation piece has to be designed and built, that is contract work I take on.