From One Best Match to a Human-in-the-Loop Recognition Queue
Since the beginning of this year, when a person database was added to supplement the knowledge base with context knowledge about known people, no new article has appeared about the Person Recognition App.
That does not mean development has stalled. A lot of work has actually been done.
One major improvement has been the integration of the person-DB into the main application. A new person-service now acts as the single entry point for operations such as adding, renaming, removing and updating people. It coordinates changes across both the person database and the biometric knowledge base, instead of allowing the GUI to modify each data source independently.
Figure 1: The Persons DB editor integrated as a tab in the main Person Recognition application.
Even more importantly, the application has undergone a broader change in purpose.
The earlier versions concentrated mainly on one question: Who is shown in this image?
The current version also addresses the more difficult question: What should happen when the software has useful evidence, but not enough evidence to make a safe automatic decision?
That distinction changes recognition from a simple machine classification task into a human-in-the-loop workflow.
The Problem with One Name or ‘Unknown’
Almost everyone building recognition software eventually encounters the same, rather frustrating, situation.
A batch recognition run produces a relatively low identification rate. Yet, when the rejected images are inspected manually in the batch report, the correct person often appears at or near the top of the face or body rankings.
The system was not necessarily wrong. It may simply have been unwilling to accept the result because one strict condition was not met. The most common causes being: the best face distance was slightly above the configured threshold, the body similarity was strong, but the face evidence was weak. Worse causes are: the face and body models preferred different people, the best candidate was plausible, but insufficiently separated from the second-best candidate or the evidence was incomplete because no usable face or body embedding could be extracted.
The original application reduced all these situations to one final result:
{
"final_name": "Unknown"
}Listing 1: Final Result
That is safe, but it throws away valuable information. Recognized versus unknown is just too crude. A result can be unsuitable for automatic acceptance while still being very useful to a human reviewer.
The application therefore now distinguishes between:
- an identity that can be accepted automatically;
- an uncertain result that should be reviewed;
- a result for which there is not enough meaningful evidence.
Recognition as a Decision Pipeline
Face and body recognition remain separate pipelines. Their results are combined only after each model has produced its own evidence.
Figure 2: Separate face and body rankings feeding the automatic decision and unified contender workflows
The important point is that the models are not forced into one artificial score too early.
The face model measures distance. Lower values are better. The body model uses cosine similarity. Higher values are better.
Those values have different meanings, distributions and threshold directions. A face distance of 0.39 cannot sensibly be averaged with a body similarity of 0.85. They must remain separate pieces of evidence.
Why Face Distance and Body Similarity Must Remain Separate
The face recognizer compares a detected face embedding with known face embeddings. Its result is expressed as a distance: smaller distance means a more similar face.
The body recognizer compares body embeddings using cosine similarity: larger similarity means a more similar appearance. Apart from using opposite directions, the two signals also have different strengths and weaknesses.
Face evidence is often highly discriminative, but it can fail when the face n the image is small, the head is turned away, the face is blurred or lighting is poor, or sunglasses, masks or hair obscure important features.
Body evidence can remain useful in those situations, but it is more sensitive to clothing changes, pose, camera angle, cropping, seasonal appearance and of course multiple people in one image.
A face match and a body match therefore should not be treated as interchangeable measurements.
The application keeps the original values visible and records separately whether each modality is available, plausible, accepted, weak or conflicting. This makes the final decision explainable.
Merging Rankings by Identity
The face recognizer and body recognizer initially return rankings over individual embeddings. A single person may have several reference images in the knowledge base and may therefore occur multiple times in the raw ranking. Before presenting candidates to the user, the application merges those results by identity.
For each person, it retains the strongest available evidence, such as the best face distance, the best body similarity and the face rank, the body rank and whether each signal passes its plausibility threshold.
The result is no longer a list of individual embedding matches. It becomes an identity-level contender slate. For example:
{
"name": "Person A",
"face_distance": 0.391,
"face_rank": 2,
"body_similarity": 0.851,
"body_rank": 1,
"face_plausible": True,
"body_plausible": True,
"evidence": "face_and_body"
}Listing 2: Contender Slate
This is much more useful than showing separate lists in which the reviewer has to manually determine that the second face result and the first body result belong to the same person.
Rank Support Is Not a Confidence Percentage
To order contenders, the application may use rank information and evidence coverage.
For example, a person supported by both face and body evidence should usually appear above a person supported by only one modality. A person ranked first by both models should normally rank above someone ranked first by one model but third by the other.
However, this combined ranking is not a calibrated probability. A contender score of 0.82 does not mean there is an 82% probability that the identity is correct. Rank support answers a narrower question: Which identity has the strongest combined relative support among the available candidates? It does not answer: How certain is the system in an absolute statistical sense?
This distinction matters because ranking values depend on how many candidates were retrieved, which modalities were available, the quality of the input image and also the composition of the knowledge base, the distance between the first and second candidates and the thresholds configured for each model.
The interface therefore shows the original face distance and body similarity rather than disguising them as one confidence percentage.
A Richer Recognition Result
Instead of returning only one name, the recognizer can now return a structured result:
{
"final_name": "Unknown",
"disposition": "review",
"review_required": True,
"contenders": [
{
"name": "Person A",
"face_distance": 0.391,
"body_similarity": 0.851,
"face_plausible": True,
"body_plausible": True,
"evidence": "face_and_body"
},
{
"name": "Person B",
"face_distance": 0.372,
"body_similarity": 0.832,
"face_plausible": True,
"body_plausible": False,
"evidence": "face_only"
},
{
"name": "Person C",
"face_distance": 0.388,
"body_similarity": 0.792,
"face_plausible": True,
"body_plausible": False,
"evidence": "face_only"
}
]
}
Listing 3: Recognizer produced Structured Results (JSON)
This result deliberately separates the automatic decision, the disposition, the review requirement, the ordered alternatives and the original modality evidence.
The Three Dispositions
The recognition workflow derives one of three dispositions from the automatic decision and the available contender evidence.
Accepted
An accepted result has sufficient evidence for automatic classification.
Typical examples are:
• a strong face match with adequate separation from impostors;
• a strong body match where face evidence is unavailable;
• face and body evidence agreeing on the same person.
Accepted images are routed directly into the output folder for that person.
persons_processed/
└── Person A/
└── image_001.jpg
Review
A reasoning result contains meaningful evidence, but not enough for automatic ac-ceptance.
Examples include:
• the best face distance is slightly outside the strict threshold;
• body evidence supports the same person but is not independently decisive;
• the models disagree;
• two identities are close;
• only one weak modality is available.
These images are routed into a dedicated review queue.
persons_processed/
└── Review/
└── image_002.jpg
The review queue preserves the machine-generated contenders so the user does not have to repeat recognition.
Unknown
An unknown result contains no sufficiently useful identity evidence.
Examples include: no face detected, no valid body embedding, all candidates are far out-side their plausibility thresholds and the image is too poor to support a meaningful com-parison.
These files are routed separately.
persons_processed/
└── Unknown/
└── image_003.jpg
The distinction between review and unknown is essential. A review result says: The system has useful evidence, but wants a human decision. An unknown result says: The system currently has no reliable candidate to offer.
Crucially, this does not weaken the existing automatic-acceptance rules. Images are still assigned automatically only when the original decision engine accepts an identity. The new review disposition merely preserves plausible evidence that previously disappeared into the broad ‘Unknown’ category. It therefore improves usability and recoverable recall without increasing false automatic assignments.
Inside the Decision Engine
The automatic decision engine remains deliberately conservative. Its purpose is not to rank all possible identities, but to determine whether the strongest available evidence is safe enough for automatic acceptance.
The decide_identity() function evaluates face and body candidates using modality-specific thresholds and their separation from the respective runner-up. A strong face can be sufficient, but a clearly separated contradictory body result can prevent immediate acceptance. Face and body agreement can rescue weaker individual scores when both modalities independently support the same identity with sufficient margins. Body-only acceptance uses a stricter threshold because clothing and pose make body evidence less stable than facial identity.
When none of these rules permits automatic acceptance, the decision remains Unknown. That does not yet determine whether the image belongs in the Review or Unknown folder. A second step examines the unified contender slate. If at least one candidate remains plausible, the image receives the review disposition; otherwise, it remains genuinely unknown.
| Threshold group | Purpose |
| face_threshold_strong | Face evidence that may be sufficient by itself |
| face_threshold_extended | Moderately strong face range |
| face_threshold_supportable | Broad plausibility range for review |
| body_threshold_support | Body evidence that can support another modality |
| body_threshold_alone | Stricter body-only acceptance |
| Margin thresholds | Required separation from the runner-up |
face_strong = float(settings.get("face_threshold_strong", 0.35))
face_extended = float(settings.get("face_threshold_extended", 0.385))
face_supportable = float(settings.get("face_threshold_supportable", 0.42))
body_support = float(settings.get("body_threshold_support", 0.85))
body_alone = float(settings.get("body_threshold_alone", 0.87))
Listing 4: Several Decision Engine thresholds
With the thresholds in place, the reasoning can begin.
if strong_face and not clear_body_conflict:
accept(face.name, reason="strong_face")
elif face_and_body_agree_with_margins:
accept(face.name, reason="face_body_agreement")
elif moderate_face and not clear_body_conflict:
accept(face.name, reason="moderate_face_clear_margin")
elif plausible_face_body_conflict:
reject(reason="face_body_conflict")
elif strong_body and face_not_plausible:
accept(body.name, reason="strong_body_only")
else:
reject(reason="insufficient_identity_evidence")Listing 5: Major decision branches in compact pseudocode
For those interested in the actual source code, the complete method is reproduced at the end of this article.
Designing Reusable Contender Cards
The review interface presents the strongest candidates as clickable cards summarizing the evidence available for each identity.
Each card can show the person’s name, a sample image from the knowledge base, face distance and face rank, body similarity and body rank, plausible or weak evidence indicators, the combined evidence type and the selected state.
A typical card illustrates why a unified contender slate is useful.
Person A
Face: Distance 0.391, Rank 2, Plausible
Body: Similarity 0.851, Rank 1, Plausible
Evidence: Face and body
Neither model necessarily ranked Person A first on its own, but both support the same identity.
The Review Dialog
As we have seen above, the Decision engine assigns one of three dispositions. When the reasoning result contains meaningful evidence, but not enough for automatic acceptance, images are routed into a dedicated review queue.
From the Recognition menu, select Review Recognition Results, directly below Run Batch Session. The dialog opens with the review candidates queued during the most recent batch session.
The cards may represent different evidence patterns. One identity can be supported by both face and body evidence. Another may have only a plausible face match because clothing or cropping makes the body result weak. In a conflicting result, the best face candidate and best body candidate appear on separate identity cards, making the disagreement visible rather than hiding it inside one combined score.
An example is shown below. Note: Faces and names shown in application screenshots have been obscured to protect the privacy of real people.
Figure 3: The Recognition Review dialog presents three identity-level contenders. A candidate may be preselected, but the assignment still requires explicit human confirmation.
Separating Machine Suggestions from Human Decisions
The human reviewer can assign the queued images to one of the proposed contenders, choose another known person, keep the image unknown, defer the decision or leave the item unreviewed.
The important architectural rule is that the machine suggestion and the human decision are recorded separately. A review record can therefore contain:
{
"machine_disposition": "review",
"machine_contenders": [...],
"human_action": "assign",
"human_person_name": "Person A"
}Listing 6: Review record detail
The original recognition result is not overwritten. This makes it possible to later answer questions like: What did the system suggest? What did the reviewer decide? Was the first contender accepted? Was another person selected? Which cases were deferred? Which images remained unknown? Where was the file moved?
This separation is useful for debugging, evaluation and future threshold calibration.
Review Sessions as Auditable Operations
A batch review is treated as a session rather than a series of informal file moves. Each session receives a unique identifier and can generate: a JSON report, a CSV report, an append-only audit manifest, counts for assigned, unknown, deferred and unreviewed items, source and destination paths, hashes used to verify file integrity, failures that occurred while applying decisions.
This matters because review actions alter the organization of the photo collection.
If a file cannot be moved, already exists at the destination or changes before the decision is applied, the failure should be visible rather than silently ignored.
Human Confirmation Does Not Automatically Modify the Knowledge Base
Correctly classifying an image does not automatically make it a good reference image.
A recognition image may be: blurred, badly lit, heavily cropped, part of a group photograph and maybe useful for classification but unsuitable as a trusted biometric reference.
The application therefore separates two decisions: Who is shown in the image? and: Should this image become part of the person’s knowledge-base gallery?
After classification, the reviewer may optionally mark the image as a knowledge-base candidate. That candidate can later be promoted, rejected or deferred. Only promotion changes the person’s reference gallery and triggers re-encoding.
This avoids uncontrolled self-learning, in which one incorrect recognition could contaminate the knowledge base and influence later results.
Reusing the Contender Widget for Interactive Recognition
The contender-card interface was initially developed for batch review. However, the same component is also useful when identifying a single image interactively.
The interactive result dialog can reuse the identical contender slate to show: the accepted identity, alternative candidates, face and body evidence, modality agreement or conflict and the currently selected person.
This avoids maintaining two separate implementations for the same concept.
The batch-review dialog and interactive-recognition dialog can share: card rendering, evidence formatting, selection behaviour, sample-image loading and selected-person highlighting.
Only the surrounding workflow differs. In batch review, the selection becomes part of a review-session decision. In interactive recognition, the selection can be used to inspect alternatives or optionally queue the image for later knowledge-base curation.
From Recognition to Managed Uncertainty
The current application still uses strict automatic decision rules. That has not changed.
What has changed is how the application handles the large and important area between a strong automatic match and a complete lack of evidence.
The complete workflow is now closer to:
Figure 4: Complete recognition workflow, including automatic routing, human review and optional registration as a future KB candidate.
This is a more realistic model of recognition. The application is no longer expected to pretend that every image has one certain answer. Instead, it distinguishes between what the models measured, what the decision rules accepted, which alternatives remain plausible, what the human reviewer decided, and what may later be used to improve the knowledge base.
‘Unknown’ remains an important and valid result. It is simply no longer the only answer available when automatic recognition stops.
A short version of this post has been published on LinkedIn.
Source code of the Decision engine
def decide_identity(face: IdentityCandidate, body: IdentityCandidate, settings: Mapping[str, Any],
) -> tuple[dict[str, float], RecognitionDecision]:
face_strong = float(settings.get("face_threshold_strong", 0.35))
face_extended = float(settings.get("face_threshold_extended", 0.385))
face_supportable = float(settings.get("face_threshold_supportable", 0.42))
body_support = float(settings.get("body_threshold_support", 0.85))
body_alone = float(settings.get("body_threshold_alone", 0.87))
face_margin_min = float(settings.get("face_margin_min", 0.025))
face_conflict_margin_min = float(settings.get("face_conflict_margin_min", 0.050))
body_margin_min = float(settings.get("body_margin_min", 0.025))
face_agreement = float(settings.get("face_threshold_agreement", 0.50,))
body_agreement = float(settings.get("body_threshold_agreement",0.84,))
face_agreement_margin_min = float(settings.get("face_agreement_margin_min", 0.015,))
body_agreement_margin_min = float(settings.get("body_agreement_margin_min", 0.010,))
body_conflict_margin_min = float(settings.get("body_conflict_margin_min", 0.025,))
thresholds = {
"face_strong": face_strong,
"face_extended": face_extended,
"face_supportable": face_supportable,
"body_support": body_support,
"body_alone": body_alone,
"face_margin_min": face_margin_min,
"face_conflict_margin_min": face_conflict_margin_min,
"body_margin_min": body_margin_min,
"face_agreement": face_agreement,
"body_agreement": body_agreement,
"body_conflict_margin_min": body_conflict_margin_min,
"face_agreement_margin_min": face_agreement_margin_min,
"body_agreement_margin_min": body_agreement_margin_min,
}
face_available = face.available # object has .name and .score
body_available = body.available # object has .name and .score
# ---------------------------------------------------------
# Shared evidence states
# ---------------------------------------------------------
same_identity = (face_available and body_available and face.name == body.name)
body_conflicts = (face_available and body_available and face.name != body.name
and body.score is not None and body.score >= body_support)
# A conflicting body result is "clear" only when it also has meaningful
# separation from its runner-up.
clear_body_conflict = (body_conflicts and body.score is not None and body.score >= body_alone
and body.margin is not None and body.margin >= body_conflict_margin_min)
face_agreement_supported = (face_available and face.score is not None and face.score <= face_agreement
and face.margin is not None and face.margin >= face_agreement_margin_min)
body_agreement_supported = (body_available and body.score is not None and body.score >=
body_agreement and body.margin is not None and body.margin >= body_agreement_margin_min)
# 1. A genuinely strong face is sufficient.
# A clearly separated contradictory body result prevents immediate acceptance.
if (face_available and face.score is not None and face.score <= face_strong and face.margin is not None):
if clear_body_conflict:
return thresholds, RecognitionDecision(final_name="Unknown", reason="strong_face_clear_body_conflict",
accepted=False, modality="none",)
required_margin = (face_conflict_margin_min if body_conflicts else face_margin_min)
if face.margin >= required_margin:
return thresholds, RecognitionDecision(final_name=face.name, reason=("strong_face_overrides_weak_body" if
body_conflicts else "strong_face"), accepted=True, modality="face",)
# 2. Face and body independently agree. Require actual runner-up separation for both modalities.
if (same_identity and face.score is not None and face.score <= face_supportable
and body.score is not None and body.score >= body_support and face.margin is not None
and face.margin >= face_agreement_margin_min and body.margin is not None
and body.margin >= body_agreement_margin_min):
return thresholds, RecognitionDecision(final_name=face.name, reason="face_body_agreement",
accepted=True, modality="fused",)
# Face & Body agree both with clear margin but might fail a strict threshold
decisive_body_agreement = float(settings.get("body_threshold_decisive_agreement", 0.75,))
decisive_face_margin = float(settings.get("face_decisive_agreement_margin_min", 0.050, ))
decisive_body_margin = float(settings.get("body_decisive_agreement_margin_min", 0.050,))
if (same_identity and face.score is not None and face.score <= face_supportable and body.score is not None
and body.score >= decisive_body_agreement and face.margin is not None and face.margin >= decisive_face_margin
and body.margin is not None and body.margin >= decisive_body_margin):
return thresholds, RecognitionDecision(final_name=face.name, reason="decisive_subthreshold_agreement",
accepted=True, modality="fused",)
# 3. Weaker face/body agreement rescue
if (same_identity and face_agreement_supported and body_agreement_supported):
return thresholds, RecognitionDecision(final_name=face.name, reason="weak_face_body_agreement",
accepted=True, modality="fused",)
# 4. Extended face range
if (face_available and face.score is not None and face.score <= face_extended and face.margin is not None):
# Do not let a moderate face override a body result that is both strong and clearly separated.
if clear_body_conflict:
return thresholds, RecognitionDecision(final_name="Unknown", reason="moderate_face_clear_body_conflict",
accepted=False, modality="none",)
required_margin = (face_conflict_margin_min if body_conflicts else face_margin_min)
if face.margin >= required_margin:
return thresholds, RecognitionDecision(final_name=face.name, reason=("moderate_face_overrides_weak_body"
if body_conflicts else "moderate_face_clear_margin"), accepted=True, modality="face",)
# 5. Remaining plausible face/body disagreement
if (face_available and body_available and face.name != body.name and face.score is not None
and face.score <= face_supportable
and body.score is not None and body.score >= body_support):
return thresholds, RecognitionDecision(final_name="Unknown", reason=("clear_face_body_conflict"
if clear_body_conflict else "face_body_conflict"), accepted=False, modality="none",)
# 6. Strict body-only acceptance
face_is_not_plausible = (not face_available or face.score is None or face.score > face_supportable)
if (face_is_not_plausible and body_available and body.score is not None and body.score >= body_alone
and body.margin is not None and body.margin >= body_margin_min):
return thresholds, RecognitionDecision(final_name=body.name, reason="strong_body_only",
accepted=True, modality="body",)
# 7. Unknown
return thresholds, RecognitionDecision(final_name="Unknown", reason="insufficient_identity_evidence",
accepted=False, modality="none",)