Not all automatic annotations are created equal. A large-scale analysis of French newspaper headlines, powered by a three-LLM pipeline, reveals a two-speed reliability profile: solid when detecting structural frames like conflict or strategic calculation, far more uncertain on constructs involving moral judgment.

The team examined 28,592 headlines from 25 outlets between 2022 and 2025, focusing on how La France insoumise and Rassemblement National were represented. Three language models performed the annotation, validated against a stratified human audit. The core finding is not a valence asymmetry between left- and right-wing populists – who gets more negative coverage – but a role asymmetry: headlines about LFI tend to deploy a conflict register, while those about RN lean toward a strategic-electoral register. This split withstands bootstrapping, permutation tests, and most of the time windows.

Yet it is the varying reliability among frame types that carries the practical weight. Conflict framing and strategic-game framing achieved the highest human validation and cross-model stability. Actor role – who is aggressor, who is victim – proved direction-stable but with lower annotator agreement, leading the authors to treat it as corroborating evidence rather than a primary finding. Finally, normative constructs – legitimacy, blame attribution, victimization – were unstable and heavily shaped by each outlet’s editorial line. In short, the pipeline converges when describing a political mechanism and diverges when passing moral judgment.

For anyone building or managing in-house annotation tools, this is not an academic footnote. LLM pipelines are increasingly used to classify legal documents, analyze intelligence feeds, monitor compliance, or sift through vast text corpora for risk management. In all these scenarios, the temptation is to treat annotations as objective output. The study instead reminds us that the gap between “recognizable frames” and “latent judgments” is technical before it is political: a model can spot that a text describes a conflict, but it cannot reliably determine who is in the right.

Constructing a pipeline aware of this stratification means, concretely, splitting tasks into two tiers: a high-reliability tier for extracting descriptive indicators (e.g., presence of aggressive language or strategic calculations), and a low-reliability tier where human intervention or deterministic rules shrink the error margin. This is no minor detail: when data live on local infrastructure for sovereignty or confidentiality reasons, the ability to perform stratified auditing becomes an operational requirement. Knowing ahead of time what the model can do well and what it cannot enables review resources to be allocated only where they are needed, containing total cost of ownership while preserving data governance.

The framework the authors propose – construct-based validation rather than a single accuracy figure – transfers to any setting where automated annotations must be interpreted by human decision-makers. For those evaluating on-premise LLM deployments, the takeaway is immediate: pipeline quality is measured not only by aggregate numbers but by mapping its strengths and weaknesses, construct by construct.