Sharing How I Solved AWS Lambda Python Package Size Limit Issues
At work, I was moving a simple crawling function to Lambda. The problem was that including requests and bs4 pushed the compressed size well over 50MB, making deployment impossible.
After looking around, I concluded that using Lambda Layer wasn't worth it—it was better to minimize dependencies. I implemented HTTP requests using only urllib.request and used regex for HTML parsing, and the total size came to under 1MB. Sure, the code got a bit messier, but the operational burden clearly decreased.
If anyone is struggling with the same issue, I recommend first checking whether the standard library is sufficient instead of using unnecessary libraries.
9 answers
Yeah, looking into the standard library first is the right move.
I've had a similar experience, and I solved it with Lambda Layers. When creating a layer, you can install the Python packages and zip them up. The 50MB limit is based on the uncompressed size, so you can go up to 250MB. I don't think you needed to go all the way to regex.
Parsing HTML with regex becomes hell every time the site structure changes. From a maintenance perspective, wouldn't it be better to use a Layer or container image instead?
I struggled with Python packages in Lambda before, but eventually deployed it as a Docker image. It's so much more convenient without the package restrictions. Thanks for sharing.
Oh, I didn't know this.
Hmm, but crawling with regex? If the site is simple, maybe, but if it's complex, it gets really tough. Wouldn't it be better to just put BeautifulSoup in a layer? It doesn't take up that much space.
Where's the source?
Anyway, since Lambda now supports containers, you don't need to worry too much about package size. Especially when you have heavy external library dependencies like in crawling, just using a container is the answer rather than cramming everything into the standard library. Still, trimming it down to urllib is refreshing.
Haha, I, on the other hand, deployed it as a layer and got a ton of hassle with permission issues.