Sharing How I Solved AWS Lambda Python Package Size Limit Issues
At work, I was moving a simple crawling function to Lambda. The problem was that including requests and bs4 pushed the compressed size well over 50MB, making deployment impossible.
After looking around, I concluded that using Lambda Layer wasn't worth it—it was better to minimize dependencies. I implemented HTTP requests using only urllib.request and used regex for HTML parsing, and the total size came to under 1MB. Sure, the code got a bit messier, but the operational burden clearly decreased.
If anyone is struggling with the same issue, I recommend first checking whether the standard library is sufficient instead of using unnecessary libraries.
9 answers
Yeah, looking into the standard library first is the right move.
I've had a similar experience, and I solved it with Lambda Layers. When creating a layer, you can install the Python packages and zip them up. The 50MB limit is based on the uncompressed size, so you can go up to 250MB. I don't think you needed to go all the way to regex.
Parsing HTML with regex becomes hell every time the site structure changes. From a maintenance perspective, wouldn't it be better to use a Layer or container image instead?
I struggled with Python packages in Lambda before, but eventually deployed it as a Docker image. It's so much more convenient without the package restrictions. Thanks for sharing.
Oh, I didn't know this.