My friend and former colleague Raffi Krikorian, who recently became the CTO of Mozilla, in The Atlantic.
When validation is purchased rather than earned, we lose something vital. And when that validation comes from a system we don’t control, trained on choices we didn’t make, we should pause. Because these systems aren’t neutral; they encode values and incentives.
Values shape the worldview baked into their responses: what’s framed as respectful or rude, harmful or harmless, legitimate or fringe. Every model is a memory—trained not just on data but also on desire, omission, and belief. And layered onto those judgments are the incentives: to maximize engagement, minimize computing costs, promote internal products, sidestep controversy. Every answer carries both the choices of the people who built it and the pressures of the system that sustains it. Together, they determine what gets shown, what gets smoothed out, and what gets silenced. We already know this familiar bargain from the age of algorithmic social media. But AI chatbots take this dynamic further still by adding on an intimacy that fawns, echoing back whatever we bring to it, no matter what that person says.
Emphasis mine. I love that sentence.