AI 可解释性接近突破了吗？一篇新文章让研究者意见分裂

Neural networks aren't nearly as tangled as researchers assumed, according to Nick Cammarata, one of the people behind the superposition hypothesis. OpenAI's Roon calls it a historic moment. Google DeepMind researcher Neel Nanda warns against overhyping it. In a long essay titled "Neural Networks Are Already Clean," Nick Cammarata argues that the inner workings of large AI models are much easier to understand than previously thought. Cammarata is a researcher at the interpretability lab Goodfire. With the essay, he rejects an explanation he helped shape years ago. Mechanistic interpretability tries to do more than watch what an AI model outputs. It aims to trace how the model arrives at its answers internally. Ideally, each of the millions of artificial neurons would stand for a recognizable concept. In practice, many neurons seemed to fire on completely unrelated things, which researchers call polysemanticity. One early and famous example is a neuron in the image model InceptionV1 that responded to both cat faces and the fronts of cars. Cammarata used it as an example in his own early research. The standard explanation was superposition. The theory goes that when a network runs out of neurons, it packs unrelated concepts into the same neurons because they rarely get in each other's way. The result looks like a knot. That's why many labs train a second model, such as a sparse autoencoder, to pull the concepts apart. They then study that model instead of the original. Cammarata played a major role in shaping this view. With Chris Olah, he co-wrote the paper "Zoom In" , which introduced the first superposition circuit. In 2023, he started an automated interpretability project at OpenAI. At the time, only about 1.7 percent of neurons counted as understandable, even by the most generous measure. Cammarata now thinks the most famous examples were misinterpreted. He says the cat-car neuron is really a V-shape detector. Pointed shapes like fox snouts or flames activate it, while round shapes like the letter O push its activation negative. So it doesn't represent two completely unrelated concepts like cat faces and cars, which means it isn't an example of superposition. He also reinterprets the superposition circuit from "Zoom In," where a car seemed to hide among dog neurons. In his view, it's ordinary generalization, and the circuit simply groups cars and dogs by which way they face the camera. He blames the mess in InceptionV1's late layers on an outdated architecture and weak analysis tools. Cammarata explains the remaining ambiguity with a pattern he calls "recursive generalization." He says a neuron grows like a branch that sprouts side shoots. One neuron that recognized the letter sequence "iy" later responded to "diy" and eventually to "do it yourself," which doesn't contain "iy" at all. Unlike superposition, these groupings can teach you something about the network. For language models, he offers preliminary numbers. In Qwen3-32B, about 90 percent of neuron poles have one main job and about 7 percent have two related ones. Only 2 to 4 percent remain unexplained. Goodfire says it sees similar results in models with up to trillions of parameters. AI agents that analyze neurons in bulk did most of the work. Cammarata borrows from Rich Sutton's "Bitter Lesson" and concludes that in interpretability, too, simple investigation at massive scale beats clever decomposition methods. Cammarata doesn't introduce a fundamentally new method. He goes back to the field's original approach of studying neurons directly, just with better tools, larger models, and automated agents. If Cammarata is right, that would be good news, especially for AI safety. Right now, apart from monitoring chains of thought that keep getting less reliable , there are few ways to tell whether a model is deceiving people or behaving dangerously. Cammarata doubts that alignment, the work of steering AI toward human goals, can be solved without interpretabili