Why Face Recognition Alone Is Not Enough — and How Context Turns Identification into Knowledge

Modern face-recognition systems excel at identifying people, but identification alone is rarely enough.
This article demonstrates how enriching a biometric recognition system with a structured, JSON-schema-driven people database adds context, meaning, and long-term usefulness.
By combining objective embeddings with subjective metadata and inspection tooling, recognition systems can evolve into knowledge platforms that remain consistent, explainable, and manageable over time.

Adding meaning beyond recognition through structured context

This article discusses the architecture of an existing Python-based person-recognition application built around a biometric knowledge base. It then introduces an efficient and scalable method for adding a second layer of contextual, human-defined data. By complementing objective biometric recognition with a structured, schema-driven database and suitable tooling, the system moves beyond identification and begins to produce insight, meaning, and long-term usefulness.

The real limitation of recognition systems

Modern person recognition systems have achieved remarkable accuracy in biometric identification, capably answering the fundamental question: who is this? Most recognition systems, however, stop at identity matching based on face embeddings—even when enhanced with full-body embeddings. Using similarity thresholds, they separate unknown persons from known ones. For a recognition system this is necessary, but by itself it is not yet very useful.

Once a person is identified, the system may still need to answer questions. For instance: Where is their media stored? How complete is my dataset? Is this person perhaps under-represented? Are there inconsistencies between what the model knows and what the filesystem contains? At this point, purely objective embeddings reach a ceiling. These are still primarily technical, system-level concerns.

A more fundamental distinction separates objective features—such as face vectors, body vectors, and similarity scores, which are measurable and reproducible—from subjective, contextual data, such as nicknames, annotations, ratings, and relationships, which are human-defined and evolve over time. A truly usable system needs both—and it also needs mechanisms to keep them consistent.

The strategic value of recognition systems grows dramatically as they evolve from identification tools into contextual analysis platforms. The next frontier lies in answering the more nuanced follow-up to the initial question: who is this, and what do I know about them?

For example, two photos may clearly depict the same individual, but without context the system cannot tell whether they are a family member, a colleague, or a recurring event participant.

Complementing embeddings with a human-readable people database

The core thesis of this article is that enriching an existing recognition system with a structured database of personal characteristics significantly increases its value and utility. This additional layer of descriptive data transforms the system from one that merely identifies individuals into one that can perform semantic queries about appearance, relationships, and context.

Alongside the objective features produced by the biometric recognition engine, a semantic layer is required: a declarative model of what the system believes to be true about a person. This data must include some system-level information—such as media-folder locations or the number of known samples—but should primarily focus on social and contextual characteristics. Examples include canonical names, nicknames, biographical data, relationships, teams, ratings, preferences, or any other attributes relevant to the specific domain of your collection.

A key insight here is that the recognition model doesn’t own this data — it consumes it in order to produce useful meaning for you.

Current System Architecture

Before discussing the proposed enhancements in detail, it is useful to understand the architecture of the existing system. The current application, built in Python with a PyQt6 graphical interface, provides an effective foundation for person recognition. Its modular design and dual-pronged recognition strategy demonstrate a clear separation of responsibilities and a focus on maintainability.

The system employs a dual-pronged recognition methodology that combines face recognition with full-body person recognition. This allows the system to identify individuals even when a clear facial view is unavailable.

The knowledge base itself is straightforward yet effective. A primary dataset folder contains person-specific subfolders of images. Biometric data—face and body embeddings—is processed and stored alongside person identifiers in a central encodings.pkl file. Recognition behavior is governed by parameters defined in settings.json.

In summary, the current framework is a capable system for objective identification. It answers the who question with a high degree of confidence. What it lacks is any mechanism to store, manage, or query descriptive, contextual data—leaving a clear opportunity for enrichment.

The Case for Enrichment: Integrating Personal Characteristics

Moving beyond purely biometric data is a necessity for unlocking the next level of analytical capability. While objective data answers who a person is, subjective data provides the context needed to answer the follow-up questions of what, when, and how. Integrating this descriptive layer transforms a photo management tool into a data analysis platform.

This ability to filter and search using a combination of identity and contextual attributes unlocks new use cases in areas such as event analysis, security, and personal photo organization. To capture this evolving and multi-faceted data efficiently and accurately, a robust and scalable data-entry mechanism is required.

A Scalable Approach to Data Entry: Leveraging JSON Schema

The primary challenge in capturing multi-faceted information lies in the data entry interface. While the existing settings UI is sufficient for managing a small, fixed set of configuration parameters, it is not well suited to a complex and evolving data model.

Ad-hoc dictionaries, implicit assumptions, and UI fields hard-coded to models tend to drift over time. To avoid this, the system adopts JSON Schema as the foundation for additional data-entry UI development.

JSON Schema is a formal, declarative vocabulary for describing and validating JSON documents. By defining structure, data types, constraints, and descriptions in a separate schema file, it becomes possible to establish a single source of truth that is independent of application code.

The real value of JSON Schema is that it becomes the contract between data, UI, and logic. This enables semi-automatic form generation, keeps the database, filesystem, and models aligned, and allows tools such as consistency checks to be built without special-case code.

What does it look like?

The following simplified example illustrates the principle. A JSON Schema defines a Person object with required properties such as firstName, lastName, and age, including type constraints and validation rules.

This schema governs the structure of persons.json, a flat-file database represented as an array of objects containing person records. The actual schema used in the application is richer, but this illustrates the principle clearly.

 {
  "$id": "https://example.com/person.schema.json",
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "Person",
  "type": "object",
  "properties": {
    "firstName": {
      "type": "string",
      "description": "The person's first name."
    },
    "lastName": {
      "type": "string",
      "description": "The person's last name."
    },
    "age": {
      "description": "Age in years which must be equal to or greater than zero.",
      "type": "integer",
      "minimum": 0
    }
  }
   "required": [ "firstName", "lastName", "age" ]
}

{"firstName": "John",  "lastName": "Doe", "age": 21},
{"firstName": "Jane",  "lastName": "Doe", "age": 25}

 {
  "$id": "https://example.com/person.schema.json",
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "Person",
  "type": "object",
  "properties": {
    "firstName": {
      "type": "string",
      "description": "The person's first name."
    },
    "lastName": {
      "type": "string",
      "description": "The person's last name."
    },
    "age": {
      "description": "Age in years which must be equal to or greater than zero.",
      "type": "integer",
      "minimum": 0
    }
  }
   "required": [ "firstName", "lastName", "age" ]
}

{"firstName": "John",  "lastName": "Doe", "age": 21},
{"firstName": "Jane",  "lastName": "Doe", "age": 25}

Validation

Because schemas do not enforce themselves, a validator is required. Python’s jsonschema library is used to validate the data file against its schema. This validation step ensures that data quality is enforced at the point of entry, not retroactively.

# validate_persons.py
import json
from jsonschema import Draft202012Validator

with open("person.schema.json", "r", encoding="utf-8") as f:
    schema = json.load(f)
validator = Draft202012Validator(schema)

def validate_doc(doc, idx=None):
    errors = sorted(validator.iter_errors(doc), key=lambda e: e.path)
    if errors:
        tag = f" (item {idx})" if idx is not None else ""
        for e in errors:
            print(f"ERROR{tag}: {list(e.path)} -> {e.message}")
        return False
    return True

with open("persons.json", "r", encoding="utf-8") as f:
    data = json.load(f)
ok = True
if isinstance(data, list):
    for i, item in enumerate(data):
        ok &= validate_doc(item, i)
else:
    ok &= validate_doc(data)
#  enforce uniqueness of personName (schema can’t do cross-record uniqueness)
if isinstance(data, list):
    names = [d.get("personName") for d in data]
    dups = {n for n in names if names.count(n) > 1}
    if dups:
        ok = False
        print(f"Error: duplicate personName values found: {sorted(dups)}")
print("Valid ✅" if ok else "Invalid ❌")

# validate_persons.py
import json
from jsonschema import Draft202012Validator

with open("person.schema.json", "r", encoding="utf-8") as f:
    schema = json.load(f)
validator = Draft202012Validator(schema)

def validate_doc(doc, idx=None):
    errors = sorted(validator.iter_errors(doc), key=lambda e: e.path)
    if errors:
        tag = f" (item {idx})" if idx is not None else ""
        for e in errors:
            print(f"ERROR{tag}: {list(e.path)} -> {e.message}")
        return False
    return True

with open("persons.json", "r", encoding="utf-8") as f:
    data = json.load(f)
ok = True
if isinstance(data, list):
    for i, item in enumerate(data):
        ok &= validate_doc(item, i)
else:
    ok &= validate_doc(data)
#  enforce uniqueness of personName (schema can’t do cross-record uniqueness)
if isinstance(data, list):
    names = [d.get("personName") for d in data]
    dups = {n for n in names if names.count(n) > 1}
    if dups:
        ok = False
        print(f"Error: duplicate personName values found: {sorted(dups)}")
print("Valid ✅" if ok else "Invalid ❌")

Turning an Array of Objects into a Database

While persons.json is technically just an array of objects, the application treats it as a lightweight database. A dedicated Python class, PersonDB, abstracts file access and provides indexing, search, and basic Create-Read-Update-Delete (CRUD) functionality.

This class also enforces semantic rules such as record identity and uniqueness, acting as the API between the flat-file database and the rest of the application.

Generating the Data-Entry Form

A separate Python component handles data entry. It parses the JSON Schema and dynamically generates a PyQt6 form, selecting appropriate widgets for each property: line edits for strings, spin boxes for numeric fields, checkboxes for booleans, and file dialogs for folder paths.

Required fields are marked automatically, and validation rules are enforced directly from the schema. This approach dramatically reduces UI boilerplate and makes the system resilient to future changes in the data model.

Here is a code fragment from the form builder illustrating how the different data types from the person.schema.json are translated into GUI widgets.

    def _widget_for_subschema(self, name: str, schema: Dict[str, Any], path: Tuple[str, ...], is_required: bool):
        stype = schema.get("type")
        enum_vals = schema.get("enum")
        # --- Enum -> QComboBox ---
        if enum_vals:
            combo = QComboBox()
            if not is_required:
                combo.addItem("")  # allow blank
            for val in enum_vals:
                combo.addItem(str(val), userData=val)
            def get_combo():
                data = combo.currentData()
                if data is None:
                    t = combo.currentText().strip()
                    return t if t else None
                return data
            def set_combo(value: Any):
                # Prefer matching userData, fallback to text
                for i in range(combo.count()):
                    if combo.itemData(i) == value:
                        combo.setCurrentIndex(i)
                        return
                # Fallback by text
                val_str = "" if value is None else str(value)
                idx = combo.findText(val_str)
                combo.setCurrentIndex(idx if idx >= 0 else 0)
            return combo, get_combo, set_combo

        # --- Object -> GroupBox (recurse) ---
        if stype == "object":
            gb = QGroupBox(name)
            fl = QFormLayout(gb)
            req = set(schema.get("required", []))
            self._add_object_fields(fl, schema, path, req)
            # group box itself has no getter/setter; its children are bound
            return gb, (lambda: None), (lambda v: None)
        # --- Numbers ---
        if stype in ("number", "integer"):
            minimum = schema.get("minimum", -1e9)
            maximum = schema.get("maximum", 1e9)
            if stype == "integer":
                sp = QSpinBox()
                sp.setRange(int(minimum), int(maximum))
                def get_num():
                    return int(sp.value())
                def set_num(value: Any):
                    try:
                        sp.setValue(int(value))
                    except Exception:
                        pass
                return sp, get_num, set_num
            else:
                dsp = QDoubleSpinBox()
                dsp.setDecimals(6)
                dsp.setRange(float(minimum), float(maximum))
                def get_dnum():
                    return float(dsp.value())
                def set_dnum(value: Any):
                    try:
                        dsp.setValue(float(value))
                    except Exception:
                        pass
                return dsp, get_dnum, set_dnum
        # --- Boolean ---
        if stype == "boolean":
            chk = QCheckBox()
            def get_bool():
                return bool(chk.isChecked())
            def set_bool(value: Any):
                chk.setChecked(bool(value))
            return chk, get_bool, set_bool
        # --- String (with special-case for files_path) ---
        if stype == "string" or stype is None:
            if path and path[-1] == "files_path":
                line = QLineEdit()
                browse = QPushButton("Browse…")
                row = QWidget()
                h = QHBoxLayout(row)
                h.setContentsMargins(0, 0, 0, 0)
                h.addWidget(line, 1)
                h.addWidget(browse, 0)
                patt = schema.get("pattern")
                if isinstance(patt, str):
                    rx = QRegularExpression(patt)
                    line.setValidator(QRegularExpressionValidator(rx))

                def on_browse():
                    d = QFileDialog.getExistingDirectory(self, "Select folder")
                    if d:
                        line.setText(str(pathlib.Path(d)))
                browse.clicked.connect(on_browse)

                def get_str():
                    s = line.text().strip()
                    return s if (s or is_required) else None
                def set_str(value: Any):
                    line.setText("" if value is None else str(value))
                return row, get_str, set_str

            line = QLineEdit()
            default = schema.get("default")
            if isinstance(default, str):
                line.setText(default)
            patt = schema.get("pattern")
            if isinstance(patt, str):
                rx = QRegularExpression(patt)
                line.setValidator(QRegularExpressionValidator(rx))
            def get_str():
                s = line.text().strip()
                return s if (s or is_required) else None
            def set_str(value: Any):
                line.setText("" if value is None else str(value))
            return line, get_str, set_str

    def _widget_for_subschema(self, name: str, schema: Dict[str, Any], path: Tuple[str, ...], is_required: bool):
        stype = schema.get("type")
        enum_vals = schema.get("enum")
        # --- Enum -> QComboBox ---
        if enum_vals:
            combo = QComboBox()
            if not is_required:
                combo.addItem("")  # allow blank
            for val in enum_vals:
                combo.addItem(str(val), userData=val)
            def get_combo():
                data = combo.currentData()
                if data is None:
                    t = combo.currentText().strip()
                    return t if t else None
                return data
            def set_combo(value: Any):
                # Prefer matching userData, fallback to text
                for i in range(combo.count()):
                    if combo.itemData(i) == value:
                        combo.setCurrentIndex(i)
                        return
                # Fallback by text
                val_str = "" if value is None else str(value)
                idx = combo.findText(val_str)
                combo.setCurrentIndex(idx if idx >= 0 else 0)
            return combo, get_combo, set_combo

        # --- Object -> GroupBox (recurse) ---
        if stype == "object":
            gb = QGroupBox(name)
            fl = QFormLayout(gb)
            req = set(schema.get("required", []))
            self._add_object_fields(fl, schema, path, req)
            # group box itself has no getter/setter; its children are bound
            return gb, (lambda: None), (lambda v: None)
        # --- Numbers ---
        if stype in ("number", "integer"):
            minimum = schema.get("minimum", -1e9)
            maximum = schema.get("maximum", 1e9)
            if stype == "integer":
                sp = QSpinBox()
                sp.setRange(int(minimum), int(maximum))
                def get_num():
                    return int(sp.value())
                def set_num(value: Any):
                    try:
                        sp.setValue(int(value))
                    except Exception:
                        pass
                return sp, get_num, set_num
            else:
                dsp = QDoubleSpinBox()
                dsp.setDecimals(6)
                dsp.setRange(float(minimum), float(maximum))
                def get_dnum():
                    return float(dsp.value())
                def set_dnum(value: Any):
                    try:
                        dsp.setValue(float(value))
                    except Exception:
                        pass
                return dsp, get_dnum, set_dnum
        # --- Boolean ---
        if stype == "boolean":
            chk = QCheckBox()
            def get_bool():
                return bool(chk.isChecked())
            def set_bool(value: Any):
                chk.setChecked(bool(value))
            return chk, get_bool, set_bool
        # --- String (with special-case for files_path) ---
        if stype == "string" or stype is None:
            if path and path[-1] == "files_path":
                line = QLineEdit()
                browse = QPushButton("Browse…")
                row = QWidget()
                h = QHBoxLayout(row)
                h.setContentsMargins(0, 0, 0, 0)
                h.addWidget(line, 1)
                h.addWidget(browse, 0)
                patt = schema.get("pattern")
                if isinstance(patt, str):
                    rx = QRegularExpression(patt)
                    line.setValidator(QRegularExpressionValidator(rx))

                def on_browse():
                    d = QFileDialog.getExistingDirectory(self, "Select folder")
                    if d:
                        line.setText(str(pathlib.Path(d)))
                browse.clicked.connect(on_browse)

                def get_str():
                    s = line.text().strip()
                    return s if (s or is_required) else None
                def set_str(value: Any):
                    line.setText("" if value is None else str(value))
                return row, get_str, set_str

            line = QLineEdit()
            default = schema.get("default")
            if isinstance(default, str):
                line.setText(default)
            patt = schema.get("pattern")
            if isinstance(patt, str):
                rx = QRegularExpression(patt)
                line.setValidator(QRegularExpressionValidator(rx))
            def get_str():
                s = line.text().strip()
                return s if (s or is_required) else None
            def set_str(value: Any):
                line.setText("" if value is None else str(value))
            return line, get_str, set_str

The form is presented alongside a searchable list of existing records, enabling both creation and editing of person entries. Buttons for saving, deleting, and validating records complete a standalone data-entry application.

Here’s is small example of a data-entry application dynamically created by the form builder from a very basic person.schema.json JSON Schema file.  

An inspection tool as a concrete example

Once both biometric embeddings and contextual data are present, new tooling becomes possible.

First, a consistency checker was implemented to compare knowledge-base statistics—such as the number of images and face/body encodings per person—with corresponding entries in the persons database.

Next a media checker was built. It answers questions such as: Does every person’s folder exist? How many images versus videos are present? Are there empty datasets? Are there silent mismatches between the database and the filesystem?

The tool provides sortable numeric analysis with visual health indicators, aggregate statistics and exportable CSV results and direct navigation into the filesystem (opening folders in Explorer).

This is not about UI polish. It is about operational awareness. Once you have objective embeddings, a structured people database, and schema-driven tooling, you can reason about your dataset—not just run models on it.

Beyond photo collections

A recognition system enriched with contextual data produces beyond identification useful meaning. From personal photo libraries, media archives, sports analytics, research datasets to surveillance footage review there are many applications in any domain where people are involved. The pattern scales even as the domain changes.

Recognition answers who is this? The database answers what do I know about them? The tooling answers how do I keep this knowledge consistent over time?

Ethical and Legal Considerations

Collecting richer data about people carries responsibility. Combining biometric data with biographical or contextual information raises ethical and legal concerns, particularly under regulations such as the GDPR and emerging European AI legislation.

Note. This section is not legal advice.

Closing Thoughts

Face recognition is powerful, but identification alone is not enough. By integrating contextual data, formal schemas and suitable tooling, recognition systems can evolve into knowledge systems—capable not only of recognizing people, but of supporting meaningful insight over time.

Download Code

The following files provide a minimal, self-contained reference implementation for developers interested in extending recognition systems with contextual data.

All examples shown here are available as a minimal, self-contained reference implementation of the new persons database, together with the person recognition application that has been updated.

The API of the new database and the inspection tooling has been added to the existing application. Because of its growing size the code base has been slightly refactored into further specialized source code files.   

People Recognize App with Database Modules

Size: 54 KB

Version: 0.8.5

Published: 2026-01-07

Download7 Downloads

Resulting Architecture

After completing the data entry module and the new functions the resulting system architecture is as follows.

A summery of the article has been posted on LinkedIn.