I Asked ChatGPT to Handle the Meeting Minutes, and It Ended Up Taking Longer
I uploaded the recording file and asked it to summarize, but it made up things that weren’t there, so I ended up listening to the whole thing again from the start.
Do I need to write better prompts? How is everyone else automating meeting minutes?
8 answers
That’s not a prompt issue—it’s a characteristic of the model. In summarization tasks, feeding in long audio often causes hallucinations where it fills in content that isn’t there. I first extract the text with STT and have it summarize only that text; that way it’s rarely wrong. I don’t recommend feeding in the entire recording file.
Well, I'm not so sure about that. Isn't it just a matter of writing good prompts? For me, it dropped a lot just by adding this one sentence: 'Never include anything that isn't in the recording, and mark it if uncertain.'
I disagree. I don’t think it’s fair to pin this on the prompt. Summary quality ultimately comes down to the source audio quality and speaker diarization performance, so shouldn’t we start by changing the tool?
Oh, I didn’t know this. I’ve also been thinking about automating meeting notes—I’ll have to look into the approach of running STT separately. Thanks!
Looks like the answer is just to have a person write the meeting minutes... I spent half a day wrestling with automation and ended up doing them by hand again ㅠㅠ
Agreed—I also handed meeting minutes off to an AI and got startled when I saw it had made up an agenda item that didn’t exist lol
Do you have a source? Where does it say that the hallucination rate is that high?
I failed the same way at first, but it worked once I split the workflow into two steps. 1) Use STT (CLOVA Note or Whisper) to extract speaker diarization as well. 2) Feed only that output as text and use a fixed prompt asking it to organize it into just three categories: 'decisions/action items/pending.' The key here is not to ask for a summary all at once but to ask for each category separately. If you ask for everything at once, the model fills in the blanks with imagination. And always cross-check the output against the original text, with a human reviewing only the action items. This cut the time to organize a one-hour meeting to about 10 minutes. I think it's realistic to give up on full automation and go 'semi-automated.'