Has anyone built an internal document summarization pipeline with the Claude API?

Hi everyone. I'm preparing to build a bot that automatically summarizes our internal wiki and meeting notes and posts them to Slack. The volume is around 300–500 items per day, and the documents average about 5,000 tokens.

There are three things I'm currently weighing.

  • Whether to send long documents in one go, or split them into chunks, summarize each chunk, and then re-summarize
  • Whether it's better to use the Batch API when factoring in cost
  • How to do automated evaluation of summary quality, since people can't review everything manually

If anyone here is running something similar in production, I'd love to hear how you approach your chunking strategy or evaluation method. Thank you.

by 밤샘코더9

7 answers

I ran a chunk + re-summarization approach for about six months, and doing summaries of summaries wiped out all the details and made the writing style get stranger and stranger. Eventually, I settled on splitting document boundaries by headings and carrying the previous chunk’s summary into the next chunk’s prompt as context. This keeps quality stable and has no re-summarization step, so debugging is much easier.

by 뉴비탈출796 · ▲0

The real trap is that nobody reads it even when you post it on Slack lol. I probably spent more time polishing the format (title, 3-line summary + action items) than on the quality of the summary itself.

by 월급루팡397 · ▲0

If it's 500 items a day, there's no reason not to use the Batch API. It's not real-time summarization, and if it's just for posting to Slack, a few hours' delay doesn't matter. You can't ignore a 50% discount.

by 프롬프트장인331 · ▲0

Wow, I need to save this.

by 무한도전러281 · ▲0

For automated evaluation, we used a rubric (whether it covers key items, whether it hallucinates, and whether it meets length requirements), ran it through an LLM judge, and then had humans cross-check a 5% sample. It wasn't perfect, but it immediately caught score drops when we changed the prompt, so that was good enough.

by 코딩하는곰698 · ▲0

If it's 5k tokens, you could probably just put it all in one shot—I don't see why you'd split it up. Isn't that overengineering? The context is plenty large, so I don't see why you'd need to go as far as a summary of a summary.

by 코딩하는곰862 · ▲0

There are many opinions that running LLM judges is ultimately just self-grading and therefore meaningless, but I think it's well worth it if you use it only for regression detection. That said, keep the judge prompt fixed every quarter. Otherwise, you can't compare scores at all.

by 데이터덕후306 · ▲0