How I Tracked Down a Docker Container Memory Leak in 3 Days
After wrestling with this for three days straight, I finally found the root cause, so here's my write-up. Our company service kept dying once a day, so I dug through the logs and plotted a memory graph—it spiked sharply around 3 AM.
The culprit was the log collection container. It sends logs to Elasticsearch, and when an exception occurred, if the stack trace was very long, indexing failures would repeat and the buffer kept growing. There was no limit on buffer size either, so it ate up to 2GB of memory and the container died.
The fix was simple:
- Truncate log messages to a maximum length of 10KB.
- Change the buffer settings to drop logs when backpressure kicks in.
After that, it stopped dying. If you're seeing similar symptoms, check your log pipeline first.
9 answers
Yeah, that's common if you don't set a limit on the log buffer count lol.
Oh, I didn't know that. We should check our logging pipeline as well.
But isn't that more of a buffer setting issue than a memory leak? Anyway, congrats on getting it solved.
I also experienced the same symptom due to fluentd retry last time, and I solved it by setting a max_buffer_size limit and merging multiline logs. Truncating logs to 10KB is also practical.
Thanks for the great article. We've also been troubled by our log collection container dying periodically, so we'll start by checking the buffer size.
If Elasticsearch indexing failures keep occurring, it's better to route them to a DLQ. Just piling them up in a buffer isn't a real solution. Using Kafka also makes backpressure management easier.
Well, I don't think truncating the log length is the real solution. Wouldn't it be better to dump the stack trace to a file and send a summary in the message instead?
Three days would be nothing lol. It took me a month, and it turned out I just had log_level set to debug...
There's plenty of disk space, why did you keep it in memory? You could've just written it to a file lol