Has anyone tried automating internal document summarization with Claude?
We've accumulated about 300 internal wiki documents, and it's too hard to search through them every time, so I'm planning to build a summarization pipeline.
I'm not sure whether I can just fetch them via API and summarize, or whether I definitely need to include embeddings + RAG. If anyone has tried this on a small scale, I'd appreciate some advice.
9 answers
For 300, it's fastest to just batch-summarize everything and paste it into the wiki. If you start with RAG, you'll spend a long time fumbling around lol
I did something similar, but in the end summaries alone weren’t searchable, so I added embeddings. That said, you don’t need to do that from the start—summary → keyword extraction → indexing with those is a realistic approach.
Just scrape it via API and summarize = this actually works. But with 300 documents, try calculating how much the token cost will be. Surprisingly, it gets cut off because of context length, so the summary quality ends up inconsistent.
Well, that's a bit... You first need to decide whether the purpose is summarization or search, but it feels like the two are mixed together.
Got a source? Didn't the security team give you any pushback for sending internal docs to an external API? We ran a local model because of that.
RAG is a must. If you only have summaries, when you think "where did that document go?" you won't be able to find it again. Embeddings are easy to add later, so I recommend starting with a summarization pipeline.
I disagree. At around 300 items, RAG is overengineering. Just make one index page with titles + tags + 3-line summaries, and people can search it themselves lol
Oh, I didn't know that.