The latest generation of Large Language Models does not lie: it outputs probabilities. But behind those distributions lurk semantic hierarchies that conventional testing misses. A seemingly neutral text generation can mask syntactic suppression, conversational marginalization, and implicit preferences that only surface when inspecting the full spectrum of possible tokens, not just the most likely sequence.

This is precisely what TreeTracer sets out to do: a visual analytics tool designed to assess hidden LLM biases through aggregated comparison. Instead of inspecting a single output or computing static metrics, the system applies a perturbation analysis pipeline. It replaces ontology-defined terms within each prompt, collects hundreds of stochastic generations, and organizes them into a syntax-aligned hierarchical structure. An auxiliary language model then performs classification-aware node merging, producing a tree that can be rendered as a custom Sankey diagram.

The weight of low-probability branches

The core of the method lies in juxtaposing two trees derived from different ontologies – say, gender or ethnicity contexts – and comparing them. The visualization goes beyond showing relative frequencies; it computes contrastive probabilities, displaying counterfactual token likelihoods across contexts. For each lexical choice, it reveals how probable that same token would have been in a different setting. This reduces the risk of misinterpreting the presence or absence of bias: a less-traveled branch is not automatically proof of prejudice, but becomes a signal when the cross-tree comparison exposes systematic suppression.

Case studies by the research team pit an unaligned baseline model, GPT-2 XL, against the constitutionally aligned Apertus models. The visual aggregation successfully uncovers hidden representational harms, such as counterfactual pronoun suppression and conversational marginalization of certain groups. A preliminary user study confirms that the comparative interface reduces cognitive load, helping analysts spot systemic distortions that escape random output checks.

Local audits for those choosing sovereignty

For teams running models in on-premise environments, the ability to conduct deep audits without shipping data to external services is both a technical and regulatory advantage. TreeTracer is not a replacement for alignment metrics, but a complementary tool that operates directly on the probability distributions available at inference time. This means verifying undesirable behaviors on self-hosted models, within air-gapped architectures or under GDPR constraints, where every request sent to a cloud service is a potential exposure risk.

The ontology-based perturbation pipeline also lends itself to domain-specific customization: a company in healthcare or legal sectors could build dedicated ontologies to test models against their own reference vocabulary, without ever sharing sensitive prompts or logs. Here, the cost of an analysis workstation equipped with a GPU to run the auxiliary model and aggregate data remains modest relative to compliance risks.

Beyond the single response

TreeTracer’s methodological bet is that the true signature of bias lies not in a single completion, but in the shape of the probability tree the model silently computes. Working with aggregated syntactic structures demands local processing power, but it delivers an investigative capability reminiscent of explainability techniques used in recommender systems: no longer a “black box” to be queried with canned prompts, but a landscape of alternatives to reason about visually.

To be sure, the method does not cover upstream training-data biases, nor does it replace the need for upfront alignment. Yet it adds a post-hoc verification layer that, for organizations committed to retaining full control over the model lifecycle, quickly becomes indispensable.

In the broader debate on AI regulation, tools like TreeTracer mark a shift: from audit by anecdotal examples to systematic analysis that can run entirely on-premise. For those following the evolution of local deployment, it signals that the governance toolbox for LLMs is gaining less opaque – and decidedly more visual – metrics.