CLEF HIPE-2026: Evaluating Accurate and Efficient Person-Place Relation Extraction from Multilingual Historical Texts

Technical Deep Dive: CLEF HIPE-2026 - Evaluating Accurate and Efficient Person-Place Relation Extraction

Problem Statement

HIPE-2026 is a CLEF evaluation lab focused on extracting semantic person-place relations from multilingual historical texts. The goal is to automatically detect where a person was located at the time of a document's publication, or anytime prior, based on evidence in the text. This enables reconstructing individuals' geographical and temporal trajectories to support digital humanities research.

Methodology

The task is formulated as a relation classification problem, where systems must label each person-place pair in a document with one of three possible "at" relations:

1. True: The text provides explicit evidence that the person was present at the place.
2. Probable: The text implies the person was likely present at the place, but does not state it explicitly.
3. False: There is no evidence, or contradictory evidence, that the person was at the place.

Additionally, systems must predict a binary "isAt" label indicating whether the person was present at the location around the document's publication time.

This requires temporal reasoning, geographic inference, and interpretation of noisy historical texts with sparse contextual cues. The task is designed to be approachable by both generative language models and more traditional classification approaches.

Data & Experimental Setup

The HIPE-2026 dataset consists of two test sets:

1. Test Set A: Historical newspaper articles in French, German, English, and Luxembourgish spanning the 19th-20th centuries.
2. Surprise Test Set B: French literary texts from the 16th-18th centuries, used to evaluate domain generalization.

A pilot study on the HIPE-2022 development data found:

• Inter-annotator agreement (Cohen's kappa) ranged from 0.7-0.9 for "at" and 0.4-0.9 for "isAt", indicating moderate to high consistency.
• GPT-4 achieved up to 0.8 agreement with the gold standard for "at", but lower and more variable results for "isAt" (0.2-0.7).
• The high inference cost and quadratic scaling of candidate entity pairs in historical documents highlight the need for scalable, efficient methods.

Evaluation

HIPE-2026 uses three evaluation profiles:

1. Accuracy Profile: Macro-averaged Recall, which ensures all relation labels contribute equally regardless of class imbalance.
2. Accuracy-Efficiency Profile: Balances predictive performance with model efficiency and resource usage.
3. Generalization Profile: Evaluates "at" relation accuracy on the Surprise Test Set B.

The dataset, baselines, and scoring tools are publicly released to promote reproducibility and further research.

Limitations & Uncertainties

• The task requires complex temporal and geographic reasoning over noisy, historical text, which remains challenging for current models.
• The dataset size may limit the ability to thoroughly evaluate domain generalization, particularly for low-resource languages and time periods.
• The "probable" label introduces subjectivity, as it depends on abductive inferences that can vary across annotators and models.

What Comes Next

HIPE-2026 aims to advance the state of the art in relation extraction for historical and multilingual domains. The results will inform future work on:

• Robust methods for mining person-place relations from digitized historical materials.
• Techniques for efficient, scalable relation extraction to handle the growing volume of historical data.
• Approaches for cross-lingual transfer and domain adaptation to improve generalization.
• Leveraging relation extraction to support downstream applications in digital humanities, such as biography reconstruction and spatial analysis.