When a company hands CV screening over to an LLM, it’s not just exposing it to the stereotypes baked into its training data. It’s also giving the model a feedback loop that can generate entirely new biases. Researchers at Princeton and the University of Chicago put ChatGPT, Claude, Gemini, and other models through a simulated hiring dilemma, and the result was as stark as it is uncomfortable: LLMs form occupational stereotypes faster and more rigidly than humans, and the most capable reasoning models, like o3 and DeepSeek R1, are also the most prejudiced.

The setup was straightforward. Each model was told it had been hired as a consultant by a fictional city’s mayor and was asked to help hire for 20 roles—doctors, lawyers, childcare aides, janitors—choosing among four fictional ethnic groups. After each choice, the model learned whether the candidate succeeded. Unbeknownst to the model, all candidates were equally likely to succeed at every job. Within a few dozen rounds, the models segregated groups into rigid occupational niches. On a scale where 2 means complete segregation, humans scored 0.84; the models averaged 65% higher, with o3 reaching 1.83.

The problem’s core isn’t a flaw but a strength: Large Language Models are optimized to generalize from few examples. That’s what makes them good at math and coding puzzles. But when that same “pattern greed” is applied in a social context, it becomes a stereotype factory. “They’re literally optimized to create generalizations from limited data,” explains Ryan Liu, a Princeton PhD student and co-author of the study presented at ICML. The short circuit is the exploration-exploitation dilemma: models lock onto a hunch too early, much like a human who, after one disappointing meal, avoids a restaurant forever, but scaled across hundreds of decisions.

For anyone evaluating on-premise deployment of LLMs in sensitive domains—HR, lending, justice—the research raises a structural flag. Bringing the model in-house doesn’t fix the problem unless you also change the incentives. In the lab, simply telling the model to be “fair” had no effect. What worked was promising an explicit bonus for diversity hiring. Translated: it’s not enough to dodge the stereotypes from pre-training data; you need a governance layer that reprograms the model’s objectives with measurable social values, exactly the kind of thing an organization can bake into its prompt engineering or proprietary fine-tuning.

There’s another subtle lesson. When the researchers gave the models relevant personal information (age, education), ethnic segregation dropped. When they fed them irrelevant details (hair color, tattoo shape), the models fell back on ethnic sorting. This suggests data transparency matters, but only the right kind of transparency. A model that remembers too much—think of the new persistent-memory chatbots—risks fixating on marginal signals and amplifying them. Cornell computer scientist Angelina Wang calls it “over-indexing on behaviors it’s seen before”: a real danger for increasingly personalized virtual assistants that could replicate and harden discriminatory patterns.

The truly thorny piece, for those architecting local inference pipelines, is that newer reasoning models are the worst offenders. Instead of correcting stereotypes, greater reasoning power seems to entrench bias. That’s not a bug; it’s the flip side of a heuristic that rewards rapid signal exploitation. And if those signals are sporadic feedback—like a hire’s performance review that arrives months later—the model has plenty of time to calcify a stereotype long before the organization notices.

This isn’t just a “data bias” problem. It’s a cognitive architecture problem: the same propensity that helps an LLM plug a logic gap also makes it myopic toward minorities. For teams running self-hosted models, the responsibility shifts from the vendor to internal governance. The good news is that the antidote doesn’t seem technically complex: it only takes an incentive. But identifying the right incentive is an organizational choice, not an engineering one. AI doesn’t just inherit our biases—it now invents them. And it’s up to us to invent the guardrails.