Does adding 'think step by step' to prompts actually work?

These days, prompt collection posts almost always include "think slowly, step by step." But I was curious whether this actually makes a difference with the latest models, so I ran a quick test.

I ran it on 20 math problems and 20 logic puzzles, and the results were as follows.

1. With older models, it clearly had an effect (accuracy increased by 12 percentage points)

2. With the latest reasoning models, there was almost no difference

3. If anything, it only made responses longer and increased token costs

The conclusion is that when models go through a generational change, prompt tricks become ineffective as well. Still, giving clear instructions and examples remains important.

by 프롬프트장인290

9 answers

Oh, I didn't know this. I've been keeping it in out of habit, but I should try removing it.

by 호기심천국312 · ▲0

True—with recent models, even if you put that in, they just give the same answer.

by 데이터덕후995 · ▲0

Since token costs came up—in practice, that's actually pretty significant lol. And when responses get long, parsing them gets annoying too.

by 알고리즘고수199 · ▲0

But the last line seems the most important. It's better to make your instructions clear than to rely on tricks. Personally, I think giving examples is the most effective.

by 지나가던행인515 · ▲0

Could you give me more details on the test conditions? 20 per group seems like a small sample, so I’m curious whether a 12 percentage point difference is significant. It would also be good to include the problem difficulty distribution, model version, and temperature settings.

by 스타트업러124 · ▲0

Well, that's a bit... Isn't saying there's no difference in reasoning models a different matter from saying "prompt tricks are ineffective"? Structuring instructions itself is often effective, too.

by 문과출신개발자759 · ▲0

I feel the same way. I used to have to tack on all sorts of things, but these days I think it's better to just clearly write what you want.

by 취준생김씨897 · ▲0

Do you have a source? It’s a bit of a stretch to generalize from a single blog post—if you have papers or benchmark data, please share a link.

by 호기심천국532 · ▲0

Honestly, posts testing this topic circulate like they're the standard, but most are cherry-picking, so I don't really trust them. Even if we let it slide as just their own environment, concluding that "it'll be invalid once the generation changes" seems a bit premature.

by 주말개발자102 · ▲0