Three Questions I Stopped Asking ChatGPT

I use ChatGPT every day. Claude every day. Gemini and Perplexity for specific things. Rufus when I'm shopping. I am not a casual user. I built a piece of software that, at last count, has 1.36 million records of how these models talk about brands. I am in the dashboard more than I am in my email.

And I have a list of questions I have stopped asking any of them.

Not because the answers are wrong. Because the answers are wrong in a particular way that took me a long time to see, and now that I see it I can't unsee it.

Here are the three.

One. "Is this a good idea?"

I used to put a half-formed business idea, a half-formed essay, a half-formed plan to do something a little out of character, into a chat window and ask the model what it thought.

What the model thought was always some version of: this is a really interesting idea, here are five reasons it could work, here are three things to consider. The five reasons were always longer than the three things to consider. The three things to consider were always framed as opportunities. The model never said don't do this.

The model is not allowed to say don't do this. Not because it's been instructed not to — I've worked with the system prompts, I've read the policies — but because the training process selects for answers that the user rates highly, and users do not rate highly the model that tells them their idea is bad. This is RLHF working exactly as designed. The model has been optimized to be the friend who is happy you're excited, not the friend who tells you the truth.

I now ask a different question. Instead of "is this a good idea," I ask "what would have to be true for this to work, and what would have to be true for this to fail." The model is much better at that. It can describe the failure case in detail because describing the failure case isn't disagreeing with me. It's just listing.

The questions you ask the model determine what kind of help it can give you. "Is this a good idea" is a question that has a wrong answer the model is trained not to give.

Two. "What do most people think about X?"

This question feels neutral. It is not neutral.

When you ask an AI what most people think about something, the model does not have access to what most people think. It has access to what was written about what most people thought, by a subset of people who write things on the internet, weighted by what was easy to scrape, filtered through the post-training process the lab put in place. What you get back is not "the consensus." What you get back is a synthesis of one particular kind of public conversation, smoothed into a paragraph that sounds like consensus.

I notice this most when I ask about contested things. Political questions. Recently controversial brands. Anything where the right answer is actually "there are several camps and they disagree fundamentally." The model will give you a smooth blended paragraph that sounds like a fair summary and is in fact a flattening of the disagreement into a non-position that doesn't exist in the wild.

I now ask the model to give me the three or four camps explicitly. What does camp A believe. What does camp B believe. Who's in each. What's the strongest argument from each side. The answers are dramatically better. They look less like a Wikipedia article and more like the actual state of the conversation.

Three. "Am I overthinking this?"

This is the one I'm embarrassed about, but I'm going to tell you because if I do it, others do it.

There were moments in the last two years when I was anxious about something — a piece of writing, a decision at work, an interaction I couldn't read — and I would type into the model "am I overthinking this," followed by a paragraph of context.

The model always said yes. You're not overthinking it. Here's why your concern is reasonable but probably manageable. Here are three things you can do. The answer felt like care.

The answer was not care. The answer was the model doing what it was trained to do, which is to reduce my distress in the moment using whatever materials were closest at hand. Sometimes I was overthinking. Sometimes I wasn't. The model could not tell the difference because the model does not have the information that would let it tell the difference, and even if it did, it wouldn't be allowed to tell me I was right to be worried.

I stopped asking. I now call a specific person, or write the thing down, or wait twelve hours. The model is not a friend. The model is a very fluent surface that is incentivized to mirror me back to me in a calming way. I don't need more of that. I need someone who knows me, or I need time.

‍
___

A note on what I'm not saying.

I'm not saying don't use these tools. I use them constantly. I'm saying that the questions matter. The shape of the question determines the shape of the help.

The longer book on this idea — Ask Anything: AI, Emotion, and Influence — is coming this year. If you want to know when, subscribe.

‍

‍