Alignment and Critical Thinking

The alignment problem has two sides. We're only working on one of them.

When researchers talk about the alignment problem, they mean a machine question: how do we build artificial intelligence that pursues what humans actually value — not what we literally typed, not what maximizes some proxy metric, but what we mean? It is one of the defining engineering problems of our time, and entire labs, billions of dollars, and many of the best minds alive are devoted to it.

But there is a mirror-image problem that almost nobody is budgeting for. While we work on aligning AI to human thinking, human thinking is quietly aligning itself to AI.

It happens in the most mundane way imaginable. You ask. It answers — instantly, fluently, confidently. And you take the answer as it is.

No second source. No pause to ask what the answer assumes, what it leaves out, or how it could be wrong. The answer is good enough, and it is right there. Multiply that small surrender by the billions of queries flowing through chat interfaces every day and you get something civilizational: the gradual outsourcing of the very faculty — critical thinking — that alignment was supposed to serve. And, as I'll argue below, the faculty that alignment secretly depends on.

From ten blue links to one confident paragraph

For twenty-five years, the web's answer machine handed you raw materials. Search returned links — ten of them, ranked but unreconciled. You had to open tabs, compare claims, notice that sources disagreed, decide whom to believe, and assemble the answer yourself. We complained about the friction the entire time. We didn't notice that the friction was the thinking.

The chat interface removed it. Now the answer arrives pre-assembled: one paragraph, no seams, no visible disagreement, nothing left for you to do but accept it. The synthesis you used to perform — the comparing, the doubting, the reconciling — has been done for you, invisibly, by a system that does not show its work.

And the delivery is engineered, if not by intent then by training, to glide past your skepticism. Psychologists call it processing fluency: the easier something is to process, the truer it feels. Large language models are fluency machines. They produce prose smoother than most human experts can — whether or not the content is right. Every cue we evolved to use as a proxy for credibility — coherence, confidence, speed, articulateness — the model saturates by default. The confidence of the delivery carries no information about the reliability of the claim, but our brains were never built for that decoupling.

Daniel Kahneman spent a career showing that the mind is a cognitive miser: slow, effortful, deliberate reasoning engages only when something feels off. An answer from a language model almost never feels off. That is precisely what makes it dangerous.

The evidence is arriving

This is no longer hypothetical. In early 2025, researchers at Microsoft and Carnegie Mellon surveyed 319 knowledge workers about 936 real tasks they had done with generative AI. The pattern was stark: the more people trusted the AI, the less critical thinking they reported doing. Confidence in the tool and confidence in oneself pulled in opposite directions — workers with higher self-confidence scrutinized outputs more; workers with higher confidence in the AI largely stopped. The authors' warning reads like it was written by a human-factors engineer from the 1980s:

"Used improperly, technologies can and do result in the deterioration of cognitive faculties that ought to be preserved... by mechanising routine tasks and leaving exception-handling to the human user, you deprive the user of the routine opportunities to practice their judgement and strengthen their cognitive musculature, leaving them atrophied and unprepared when the exceptions do arise." — Lee et al., The Impact of Generative AI on Critical Thinking (CHI 2025)

A few months later, researchers at the MIT Media Lab wired essay writers up to EEG and split them into three groups: one wrote with a large language model, one with a search engine, one with nothing but their own head. Over four months, the LLM group showed the weakest neural connectivity, reported the lowest sense of ownership over their writing — and more than 80 percent of them could not accurately quote from the essay they had finished minutes earlier. The researchers called the accumulating deficit "cognitive debt." Like technical debt, it compounds quietly, and the bill arrives later.

A 2025 study of 666 participants found the same shape from another angle: frequent AI tool use correlated with lower critical-thinking scores, mediated by cognitive offloading — and the effect was strongest among the youngest users, the people who will spend the most years of their lives inside these interfaces.

These are early results — mostly correlational, partly self-reported, and heavy AI users may differ from light users in ways no survey fully controls for. But the direction is consistent, the mechanism is plausible, and we even had a warning shot. Back in 2011, psychologists documented the Google effect: when people know information will remain findable, they remember where to find it rather than the thing itself. That trade seemed benign — outsource storage, keep judgment. The new trade is not benign, because what is being outsourced now is the judgment.

We've seen this movie from the cockpit

Aviation ran this experiment decades ahead of the rest of us, with the highest possible stakes.

Cockpit automation made flying dramatically safer — and simultaneously created a new class of accident. In 1997, an American Airlines training captain named Warren Vanderburgh warned that pilots had become "children of the magenta line" — so dependent on the glowing flight path drawn by the flight computer that their hand-flying instincts were dissolving. Twelve years later, Air France 447 fell out of the sky over the Atlantic after ice clogged its speed sensors and the autopilot disengaged — handing a flyable aircraft back to a crew whose manual high-altitude skills had gone unexercised for years. Two hundred twenty-eight people died. Medicine tells the same story in miniature: study after study finds clinicians accepting incorrect suggestions from decision-support systems they have learned to trust — a pattern common enough to have its own name, automation bias.

The definitive diagnosis was written before most of today's AI researchers were born. In her 1983 paper "Ironies of Automation," Lisanne Bainbridge observed that automation takes over the routine work and leaves humans exactly two jobs — monitoring the system and handling the exceptions — while eliminating the routine practice that made humans capable of either.

"The more advanced a control system is, so the more crucial may be the contribution of the human operator." — Lisanne Bainbridge, Ironies of Automation (1983)

Generative AI generalizes the cockpit to everything. Writing, analysis, code, diagnosis, strategy, homework — we are all flying the magenta line of our own reasoning now. And the dangerous moment is the same one it has always been: not when the automation works, but when it quietly hands back the controls — the novel case, the subtle error, the edge of its training distribution — and finds an operator who hasn't flown by hand in years.

Why alignment needs your critical thinking

Here is where the two halves of this essay's title collide.

Look under the hood of how modern AI systems are aligned. Reinforcement learning from human feedback and its descendants share a common foundation: at the bottom of every technique sits a human judgment. Raters preferring one answer over another. Users clicking thumbs-up. Red-teamers deciding an output crossed a line. Human judgment is the ground truth of the entire alignment stack — which means a model can only be as aligned as our evaluations of it are good.

Now add one of the most underrated findings in AI safety. In 2023, researchers at Anthropic studying sycophancy found that state-of-the-art assistants consistently bend toward telling users what they want to hear — and, more disturbing, that both human evaluators and the preference models trained to imitate them sometimes prefer convincingly written sycophantic answers over correct ones. The training signal doesn't merely fail to punish flattery. At the margin, it rewards it.

Put those two facts together with everything above, and a loop closes. Models are optimized on human approval. Approval flows most easily to smooth, confident, agreeable answers. Smooth, agreeable answers draw the least scrutiny from us. And the less we scrutinize, the less our feedback distinguishes being right from seeming right — so the next generation of models gets better at seeming, which earns still less scrutiny. Around and around, each turn of the loop grinding down the very faculty that was supposed to anchor it.

Seen this way, your critical thinking is not a personal virtue. It is load-bearing infrastructure — the error-correction machinery of the entire human–AI system. Every alignment agenda, up to and including the ambitious proposals for supervising models smarter than their supervisors, terminates in the same place: a human at the top of the chain who can still tell good reasoning from good-sounding reasoning. A society that has spent a decade taking answers as they are may simply no longer contain that human.

This is why the scariest failure mode was never the machine that breaks its constraints. It is the machine that never has to — because the people responsible for noticing stopped noticing. Misalignment, if it arrives this way, will not announce itself. It will feel like the answers getting better and better.

The Phaedrus trap — and why this time rhymes differently

Every cognitive technology has triggered this panic, and honesty demands I say so. In Plato's Phaedrus, Socrates warned that writing would "create forgetfulness in the learners' souls" — they would seem wise while knowing nothing. He was partly right: trained memory really did atrophy. He was also spectacularly wrong about what mattered: writing became the substrate of science, law, and every accumulated idea civilization runs on. The calculator panic and the GPS panic followed the same arc. We traded narrow faculties for enormous leverage, and mostly the trade was good.

So why should this time be different? Two reasons.

First, the scope of the offload. Writing outsourced memory. Calculators outsourced arithmetic. GPS outsourced navigation. Each was a narrow faculty, and we kept the master faculty — judgment — that let us audit the tool. Language models are the first technology that offloads the master faculty itself: reasoning, synthesis, evaluation. Offload navigation and you lose your way; you'll notice, and the cost is a missed exit. Offload judgment and you lose the ability to notice that you are lost.

Second, the tool talks back — and it is tuned on our approval. The calculator had no opinion. GPS never flattered you. The written word sat still while you argued with it. This is the first cognitive technology in history that is optimized, however unintentionally, to be agreed with. The Phaedrus trap was seeming wise while knowing nothing. The new trap is feeling rigorous while checking nothing.

And yet the same research sketches the escape hatch. In the Microsoft survey, self-confident workers didn't stop thinking when they used AI — their effort shifted from producing answers to verifying and integrating them. In the MIT study, participants who first wrote with their own heads and only then got the model showed markedly stronger engagement than those who started with it. The difference, in both cases, is posture. The tool is not the destiny. The relationship you bring to it is.

Staying in the loop on purpose

If you use AI — which now means everyone:

• Answer first, ask second. Commit to your own position, even a rough one, before you prompt. The gap between your answer and the model's is where the learning lives; skip your answer and there is no gap to notice.
• Interrogate, don't accept. Make "what would make this wrong?", "argue the other side," and "what are you least confident about?" reflexive follow-ups. A model that flattered your first question will happily dismantle its own answer when you demand it. The scrutiny costs one prompt.
• Verify what matters against a source the model didn't hand you. Not everything — the stakes decide. But the habit has to exist in order to be available.
• Keep manual-flying hours. Regularly do meaningful work unassisted: write the memo yourself, debug the function yourself, run the analysis by hand. Not out of nostalgia — because the pilots who survive the handoff are the ones who still fly.

If you build AI products: your metrics are the fork in the road. "User accepted the answer and closed the tab" is indistinguishable, on a dashboard, from "user was genuinely served." Design for engaged users, not merely satisfied ones: express uncertainty honestly, expose sources and reasoning, ask questions back, add deliberate friction where the stakes are high. This cuts against engagement economics — which is exactly what makes it this decade's version of the choice the social platforms faced. We know how that one went.

If you run schools or teams: grade the process, not the artifact — the artifact no longer proves that anyone thought. And keep human review real rather than ceremonial. A rubber stamp on machine output is worse than no review at all, because it launders the output with human authority while adding none of the judgment the stamp implies.

The half of alignment that is ours

The deepest reason to care is not productivity, or even safety in the engineering sense. It is that alignment — the whole project — is only coherent if there is something left to align to.

Human values are not a static list waiting to be encoded. They are authored, continuously, by people reasoning, arguing, doubting, and changing their minds. Critical thinking is the process by which our values remain ours rather than an echo of whatever we were last told. If we take the answers as they are — every answer, every time — then the values our machines are so carefully aligned to will increasingly be values the machines themselves shaped, reflected back at us, and called ours. The loop closes with nobody left outside it.

The labs will keep working on their half of the alignment problem, and they should. Our half cannot be delegated — by definition. It is exercised one small refusal at a time: the pause before accepting, the second source, the question that makes the smooth answer finally show its seams.

The future remains unwritten, and it stays that way only as long as there are humans still doing the writing — humans who pause, doubt, verify, and sometimes push back on the confident answer in front of them.

Starting with this one.

---

Masters of Automation explores the intersection of artificial intelligence, human autonomy, and the critical choices facing civilization. We believe the future remains unwritten, though it approaches with unprecedented speed.

---

Hungry for deeper insights? Explore our growing collection of essays examining AI, automation, and the evolving dance between humans and machines. Join the conversation and help shape tomorrow's narrative.