I Tried Automating Paper Summaries with Claude, and It's Really Convenient
I'm a grad student, and I was losing my mind having to read 20 papers every week. So I wrote a script that parses PDFs and summarizes them section by section.
It's not perfect, but since I no longer have to skim abstracts to filter papers, it's cut my reading time in half. It's a whole new world 😭
That said, I still have to check equations and experimental setup myself. Still, even just this much is definitely worth it.
9 answers
Facts, once you get a taste of this, you can't go back lol
Could I ask which model you're running it with? I'm curious whether Sonnet is enough or if you have to go all the way to Opus—with 20 pieces, the cost would add up quite a bit.
Well, it does produce plausible summaries, but there are quite a few cases where they get subtly muddled into something different from the original... Especially with review papers, the tone shifts from section to section, so if you just feed them in in order, everything gets jumbled together. I don't trust automated abstract summarization, so I don't use it. The feeling that it saved you time is an illusion, and if you later find that it filtered something out incorrectly, you could end up taking a much bigger hit.
Could you share the script? ㅠㅠ
I use something similar, and PDF parsing is where things break the most. With two-column papers, the text order gets jumbled vertically, so sometimes the summary is completely off. So I extract block coordinates with pymupdf, split the columns based on x-coordinates, and then reorder them. Just doing that alone improves the quality dramatically. Tables... there’s really no good solution, so I just captured them as images and baked “look at them separately” into the prompt.
For arXiv papers, it's much better to grab the TeX source instead of the PDF and insert the sections as-is. You even get clean bibliographic info, so you don't have to go through parsing hell. Of course, for journal PDFs there's no choice, but...
It’s not that the abstract-skimming step disappeared—it’s that a step got added where you have to re-verify what the AI skimmed, right? lol Total time might be shorter, but I feel like it’s a bit risky to believe it automatically does the “filtering” for you.
I feel the same. When it comes to equations and experimental setups, you ultimately have to look at the original text. That part still seems to be a job for humans.
One tip: add this to your prompt—'Do not speculate about anything that is not in the text; explicitly state that it is not there'—and have each summary sentence include the corresponding quoted passage from the original. That will make verification at least twice as fast. Also, instead of having all 20 articles summarized, switch to running a relevance screening first and then only close-reading the ones that pass—you'll notice a much bigger difference.