Has anyone built an internal document summarization pipeline with the Claude API?
Hi everyone. I'm preparing to build a bot that automatically summarizes our internal wiki and meeting notes and posts them to Slack. The volume is around 300–500 items per day, and the documents average about 5,000 tokens.
There are three things I'm currently weighing.
- Whether to send long documents in one go, or split them into chunks, summarize each chunk, and then re-summarize
- Whether it's better to use the Batch API when factoring in cost
- How to do automated evaluation of summary quality, since people can't review everything manually
If anyone here is running something similar in production, I'd love to hear how you approach your chunking strategy or evaluation method. Thank you.
7 answers
If it's 500 items a day, there's no reason not to use the Batch API. It's not real-time summarization, and if it's just for posting to Slack, a few hours' delay doesn't matter. You can't ignore a 50% discount.
If it's 5k tokens, you could probably just put it all in one shot—I don't see why you'd split it up. Isn't that overengineering? The context is plenty large, so I don't see why you'd need to go as far as a summary of a summary.
I ran a chunk + re-summarization approach for about six months, and doing summaries of summaries wiped out all the details and made the writing style get stranger and stranger. Eventually, I settled on splitting document boundaries by headings and carrying the previous chunk’s summary into the next chunk’s prompt as context. This keeps quality stable and has no re-summarization step, so debugging is much easier.
Wow, I need to save this.