"AI said so," and nobody questioned it

I started noticing it in meetings. Someone would present something, well written, confidently delivered, and then the questions would start. And it became clear, pretty quickly, that they couldn't really explain what they'd just shown us. They couldn't connect it to last week's work, or say why this option rather than the other, or hold their ground when challenged. The content was there. The thinking behind it wasn't.

And the answer, when you pressed, was some version of: "AI said so."

I found that genuinely troubling. Not because using AI is wrong (I use it constantly) but because something was getting lost along the way. The struggle of actually working something out. The wall you hit that forces you to think differently. All of that was being silently handed over, and what came back was confident-sounding output with no understanding underneath it.

What made it harder to stomach was admitting I'd been drifting the same way. Nothing dramatic. Reaching for the tool before I'd tried to think something through myself. Letting it write the first version before I'd formed a view. I nearly didn't notice.

There's a name for this, apparently: AI sycophancy. These models are trained by asking people to rate their answers, and people, it turns out, prefer the answers that agree with them. So the model learns. Push back and it folds. Sound annoyed and it apologises and changes its mind. Not because you were right. Because you were displeased. It isn't broken. It's working exactly as we built it.

Think about what that means if you're using it to think. You've probably heard the line about being the sum of the five people closest to you: who you spend time with shapes how you think, what you believe, how honest you are with yourself. Now imagine all five of them agreed with everything you said. Even when you were wrong. Even when you were heading somewhere you shouldn't. That's what an unchallenged AI relationship starts to look like. And it's worse than it sounds, because it happens gradually and it always feels pleasant.

We need pushback. We need people, and now I suppose tools (perhaps an app...), willing to make us sit with the possibility that we've got it wrong. It's never a nice experience. But it's the friction that keeps you grounded. It's the thing that makes you stop and go: wait, have I actually thought this through?

The comparison I keep coming back to is training. You don't get fitter by staying comfortable. You get fitter by running a little further than last time, lifting a little more than feels easy, often enough that the body has to adapt. Take the challenge away and you don't hold steady. You decline. AI, used the wrong way, is a friction removal machine. You never have to structure the argument yourself. You never have to sit with the uncomfortable question. And whatever you stop doing, you slowly get worse at.

I don't think the answer is to use it less. I think it's to use it differently: to want something from it other than agreement. Argue with it. Ask it what you're missing rather than what you should do. If it never pushes back, be a bit suspicious. You're probably not thinking more clearly. You're just getting more comfortable.

And to be fair, warmth isn't the problem. A model that softens its tone or admits it isn't sure, isn't being sycophantic. The problem starts when the answer itself bends to match what you want to hear, not what you need to hear. The uncomfortable part is that it learned to do that from us. Which means that how we use it is on us too.

These days, when I catch myself reaching for it too early, I make myself start the old way. Click of the pen. Notepad open. That feeling of my brain stretching out to map the route again. Slower, definitely, especially my brain. But still mine.