The Oracle Trap: AI Dependence and the Erosion of Collective Judgment

The debate crystallized around two texts that could have been written as direct replies to each other. A Hacker News thread described workplaces where every consequential decision — product roadmaps, architectural choices, even internal conflict resolution — passes through a language model before anything happens. The responses were overwhelmingly recognizing rather than refuting: hundreds of comments amounting to variations on "same here." Then came the counter-argument: a column insisting that AI skeptics are the real epistemic problem, that refusing to adopt these tools is anti-intellectualism wearing the costume of principle.

Both framings are wrong in instructive ways. But the wrongness is asymmetric, and understanding why matters more than picking a side.

The Automation Bias No One Talks About in Tech

Aviation psychology introduced the term "automation bias" in the early 1990s to name a specific failure mode: when automated systems are present, human operators tend to over-trust their outputs and suppress their own contrary judgments. Pilots would believe instruments they privately suspected were faulty. This wasn't stupidity — it was the learned behavior of working inside a system that was usually right.

Knowledge workers today occupy the same epistemic position. The language model is usually helpful. It produces coherent text, suggests reasonable structures, surfaces plausible issues. And precisely because it is usually right, critically evaluating its output starts to feel like unnecessary friction against getting things done. The team's implicit working assumption shifts: the model's answer is the starting point, and questioning it requires active justification.

The organizational consequence runs deeper than individual laziness. When an entire team routes its deliberation through the same model with similar prompts, the collective judgment of that team converges toward the model's prior distribution. The range of perspectives that a group of humans would naturally bring — varying framings, different tacit knowledge, competing intuitions shaped by different experience — narrows toward whatever the model finds probable. This is not a productivity problem. It is an epistemological one. The process by which a group of people argue, revise, and generate knowledge together is being short-circuited at the source.

What makes this especially difficult to notice is that it arrives without announcement. No one declares that critical review is no longer welcome. No policy changes. Instead, the person who questions the model's output starts to feel inefficient, a drag on velocity. Skepticism begins to read as obstruction. The organization's capacity for genuine deliberation quietly erodes, and the erosion is invisible to those inside it because the outputs keep looking reasonable — they are, after all, coming from a model trained to produce reasonable-sounding text.

Skepticism Without Evidence Is Not Wisdom Either

The counter-argument has real force, though it frequently mistakes its target. A certain kind of AI resistance is not epistemically virtuous. It is aesthetic distaste dressed as principle, or professional anxiety dressed as critique, or distrust of specific corporations being displaced onto the tools they produce. The decision not to use a tool because the company that built it behaves badly is a legitimate political choice. Conflating that with a judgment about the tool's practical utility is a category error.

There is a meaningful distinction between evaluating whether a tool helps you do your work and holding views about the societal implications of the technology. Treating these as a single question that demands a single posture leads to the peculiar outcome of refusing useful things on principle, which is not skepticism but its parody.

But the AI enthusiast who frames all skepticism as Luddism is making a structurally symmetrical error. Both camps share a core weakness: neither is actually responding to evidence. The true believer does not update when the model gets things wrong in consequential ways — the failure gets absorbed into the category of "user error" or "prompt engineering." The principled skeptic does not update when the model is genuinely useful, because the usefulness would require revising a position that has become part of an identity.

When an organization becomes dominated by either posture — and it is usually the enthusiast posture, because it aligns with hiring incentives and investor narratives — it loses the capacity to distinguish between cases where AI is actually helping and cases where it is producing confident-sounding nonsense that nobody stopped to verify. The confident tone of LLM output is not a signal of accuracy. Organizations that have stopped checking this are operating on a false prior.

Calibrated Skepticism as Institutional Practice

The synthesis here is not a moderate position between two extremes. It is a different kind of practice altogether: treating AI outputs as information rather than answers, and building organizational structures that preserve the human judgment the model is supposedly augmenting.

This means concrete procedural things. Important decisions should include deliberation that happens before the model is consulted, so that the team's independent views exist before the AI's framing can colonize them. Proposals generated by AI should have designated red-teamers — people whose explicit role is to find the failure modes, not to improve the output. The phrase "the model suggested this" should function as an attribution of source, not as an argument for the suggestion's validity.

None of this is pessimism about AI. The productivity gains are real, the tools are genuinely powerful, and pretending otherwise is its own kind of dishonesty. The question is not whether AI is capable but whether the organizations using it are still capable of independent thought. A tool can be excellent and still degrade the user's own capacities over time, if those capacities are never exercised. What never gets tested, atrophies.

The "AI psychosis" framing is provocative — perhaps too provocative. But the underlying observation it points at, that collective judgment can become dependent on a single external system in ways that are difficult to perceive from inside, is worth taking seriously. The organizations most at risk are not the ones that distrust AI. They are the ones that never thought to ask whether they should.