After wrestling with this for three days straight, I finally found the root cause, so here's my write-up. Our company service kept dying once a day, so I dug through the logs and plotted a memory graph—it spiked sharply around 3 AM.
The culprit was the log collection container. It sends logs to Elasticsearch, and when an exception occurred, if the stack trace was very long, indexing failures would repeat and the buffer kept growing. There was no limit on buffer size either, so it ate up to 2GB of memory and the container died.
The fix was simple:
- Truncate log messages to a maximum length of 10KB.
- Change the buffer settings to drop logs when backpressure kicks in.
After that, it stopped dying. If you're seeing similar symptoms, check your log pipeline first.