Has anyone built an internal document summarization pipeline with the Claude API?
Hi everyone. I'm preparing to build a bot that automatically summarizes our internal wiki and meeting notes and posts them to Slack. The volume is around 300–500 items per day, and the documents average about 5,000 tokens.
There are three things I'm currently weighing.
- Whether to send long documents in one go, or split them into chunks, summarize each chunk, and then re-summarize
- Whether it's better to use the Batch API when factoring in cost
- How to do automated evaluation of summary quality, since people can't review everything manually
If anyone here is running something similar in production, I'd love to hear how you approach your chunking strategy or evaluation method. Thank you.
7 answers
I ran a chunk + re-summarization approach for about six months, and doing summaries of summaries wiped out all the details and made the writing style get stranger and stranger. Eventually, I settled on splitting document boundaries by headings and carrying the previous chunk’s summary into the next chunk’s prompt as context. This keeps quality stable and has no re-summarization step, so debugging is much easier.
The real trap is that nobody reads it even when you post it on Slack lol. I probably spent more time polishing the format (title, 3-line summary + action items) than on the quality of the summary itself.
If it's 500 items a day, there's no reason not to use the Batch API. It's not real-time summarization, and if it's just for posting to Slack, a few hours' delay doesn't matter. You can't ignore a 50% discount.
Wow, I need to save this.
For automated evaluation, we used a rubric (whether it covers key items, whether it hallucinates, and whether it meets length requirements), ran it through an LLM judge, and then had humans cross-check a 5% sample. It wasn't perfect, but it immediately caught score drops when we changed the prompt, so that was good enough.
If it's 5k tokens, you could probably just put it all in one shot—I don't see why you'd split it up. Isn't that overengineering? The context is plenty large, so I don't see why you'd need to go as far as a summary of a summary.
There are many opinions that running LLM judges is ultimately just self-grading and therefore meaningless, but I think it's well worth it if you use it only for regression detection. That said, keep the judge prompt fixed every quarter. Otherwise, you can't compare scores at all.