The first reproducible result in a computer-vision project should often be a dataset audit, not a training curve. If the same physical object appears in both training and test images, a model can look impressive while answering a much easier question than the one we intended.
This is especially relevant for heritage collections, where one object may have many photographs, crops, views or imaging sessions.
Research context
AmalthAI: An Open-Source Computer Vision Platform for Cultural Heritage was first posted on 13 August 2026. I am one of its coauthors; the platform is a collaborative TEXTaiLES contribution. The paper describes dataset management, classification, segmentation and detection, with training infrastructure and tools for expert interpretation. The official repository and documentation are the implementation references. This exercise audits a fictional dataset manifest before import. It is not an end-to-end installation or benchmark of the platform.
Decide what the test set should represent
If the intended use is classification of previously unseen objects, split by object identity. If it is generalization to a new collection, a collection-level holdout is more informative. If the challenge is a new acquisition setup, an imaging-session holdout may reveal problems that a random image split conceals.
Those are different experiments. A random split is not inherently dishonest, but its interpretation must match what it actually separates. “Unseen image” does not imply “unseen object.”
A useful manifest has at least an image ID, physical-object ID, label, acquisition context and split. Add rights information and annotation provenance where needed. The manifest should remain a versioned input to the experiment, rather than being reconstructed from filenames after training.
A leakage check that fails visibly
rows = [
{"image": "a-front", "object": "object-a", "split": "train"},
{"image": "a-back", "object": "object-a", "split": "test"},
{"image": "b-front", "object": "object-b", "split": "test"}
]
by_object = {}
for row in rows:
by_object.setdefault(row["object"], set()).add(row["split"])
leaks = {obj: sorted(splits) for obj, splits in by_object.items()
if len(splits) > 1}
print(leaks) # {'object-a': ['test', 'train']}
The problem is not that the two images are identical. They share an object identity that the intended evaluation was supposed to withhold. Duplicate-image hashes can catch some errors, but they do not replace this semantic grouping check.
The companion script demonstrates both the leaking manifest and a corrected split. It also checks that image identifiers are unique and that split names belong to a declared set.
Match the annotation to the task
For classification, specify the unit receiving the label: the whole image, one object, or an expert-defined category. For detection, define what counts as an instance and how overlapping or truncated objects are handled. For segmentation, document the meaning of each mask value and how ambiguous boundaries are treated.
Changing these conventions halfway through a dataset can create apparent model errors that are actually annotation disagreements. A small expert-reviewed sample, with written conventions, is often more valuable than silently extending an inconsistent annotation set.
Evaluate the claim you want to make
For a binary classification example, precision measures the fraction of predicted positives that were correct, while recall measures the fraction of actual positives recovered:
Neither replaces per-class inspection. A model can achieve high overall accuracy by performing well on the largest class while failing on a rare category that matters to the research question. Segmentation and detection also require task-specific metrics and clear matching conventions; a classification score cannot summarize them.
Keep interpretation separate from validation
A heatmap over a plausible region is useful for expert inspection. It does not prove that a model learned the intended archaeological distinction. I would inspect errors across backgrounds, lighting conditions and object groups, and check whether the decision changes when an irrelevant acquisition cue changes.
Before a training run, freeze the manifest and annotation version. After it, retain the configuration, model checkpoint and per-example predictions. That record allows a colleague to check whether the reported result corresponds to the intended split.
This is the practical entry point for using a platform well: make the experiment’s data boundary explicit before relying on the convenience of its interface.
Complete reproducible example
This standalone example uses fictional data or prescribed mathematical objects; it is not a reproduction of the source paper’s experiments.
"""Fictional object-level split audit; no AmalthAI installation required."""
import json
def audit(rows):
ids=[r["image"] for r in rows]
assert len(ids)==len(set(ids)), "Duplicate image identifiers"
assert all(r["split"] in {"train","validation","test"} for r in rows)
groups={}
for r in rows: groups.setdefault(r["object"],set()).add(r["split"])
return {obj:sorted(s) for obj,s in groups.items() if len(s)>1}
rows=[{"image":"a-front","object":"object-a","split":"train"},
{"image":"a-back","object":"object-a","split":"test"},
{"image":"b-front","object":"object-b","split":"test"}]
leaks=audit(rows)
assert "object-a" in leaks
corrected=[dict(r,split="train") if r["object"]=="object-a" else dict(r) for r in rows]
assert audit(corrected)=={}
print(json.dumps({"leaking_objects":leaks,"corrected_audit":audit(corrected)},indent=2))