Claude 3.5 vs GPT-4o: I tested them on paper summarization and the results were quite different
I recently compared AI paper summarization capabilities. I fed the same paper (a recent paper on LLM-based reasoning) into both Claude 3.5 and GPT-4o for summarization.
Claude felt better at capturing the overall flow and extracting the key contributions, while GPT-4o emphasized detailed experimental results and figures. I used to mainly use GPT for reading paper abstracts, but now I think I should use Claude alongside it.
Does anyone else have a preferred model for paper summarization or code review? I'd appreciate it if you could share the pros and cons of each.
4 answers
I've had a similar experience! Claude 3.5 seems really strong at grasping the overall context of a paper. Especially when summarizing the introduction and conclusion, it really hits the key points. On the other hand, GPT-4o retains more detail when analyzing numbers or tables in the experimental section, so I find it useful for comparing results. That's why I tend to use Claude for drafting and GPT for detailed verification, alternating between them.
I mainly use it for code reviews, and GPT-4o feels more thorough at finding bugs or suggesting optimizations. Claude is great for beginners with its friendly explanations, but GPT seems better at analyzing complex logic. I haven't tried either for paper summaries, but Claude's ability to grasp the flow is quite appealing. I should give it a try.
Did the results vary depending on the paper length? When I tested with around 10 pages, Claude was more consistent, but when it went over 20 pages, GPT-4o preserved the details better. The difference was especially noticeable in parts like the experimental methodology. It seems each model clearly has its own strengths in different areas, so I think I'll need to choose based on the use case going forward.
Oh, I recently did a similar comparison and totally agree! Claude feels like it explains 'why this paper is important,' while GPT digs into 'how it was proven.' Personally, I've settled on using Claude for conceptual understanding and GPT for data verification. Using them complementarily really creates synergy, haha.