Does adding 'think step by step' to prompts actually work?
These days, prompt collection posts almost always include "think slowly, step by step." But I was curious whether this actually makes a difference with the latest models, so I ran a quick test.
I ran it on 20 math problems and 20 logic puzzles, and the results were as follows.
1. With older models, it clearly had an effect (accuracy increased by 12 percentage points)
2. With the latest reasoning models, there was almost no difference
3. If anything, it only made responses longer and increased token costs
The conclusion is that when models go through a generational change, prompt tricks become ineffective as well. Still, giving clear instructions and examples remains important.
9 answers
Oh, I didn't know this. I've been keeping it in out of habit, but I should try removing it.
True—with recent models, even if you put that in, they just give the same answer.
Since token costs came up—in practice, that's actually pretty significant lol. And when responses get long, parsing them gets annoying too.
But the last line seems the most important. It's better to make your instructions clear than to rely on tricks. Personally, I think giving examples is the most effective.
Could you give me more details on the test conditions? 20 per group seems like a small sample, so I’m curious whether a 12 percentage point difference is significant. It would also be good to include the problem difficulty distribution, model version, and temperature settings.
Well, that's a bit... Isn't saying there's no difference in reasoning models a different matter from saying "prompt tricks are ineffective"? Structuring instructions itself is often effective, too.
I feel the same way. I used to have to tack on all sorts of things, but these days I think it's better to just clearly write what you want.
Do you have a source? It’s a bit of a stretch to generalize from a single blog post—if you have papers or benchmark data, please share a link.
Honestly, posts testing this topic circulate like they're the standard, but most are cherry-picking, so I don't really trust them. Even if we let it slide as just their own environment, concluding that "it'll be invalid once the generation changes" seems a bit premature.