How to Solve the 15-Minute Timeout in AWS Lambda
I'm running batch jobs on AWS Lambda, but I'm hitting the 15-minute limit and can't process large files. I'm considering chaining them with Step Functions or switching to ECS Fargate—which one would be more efficient? I also need to consider costs, so I can't make a decision. Would appreciate advice from anyone who's faced a similar issue.
8 answers
I had a similar concern, and if the task is simple, going with Fargate rather than Step Functions felt cleaner. Lambda is fundamentally optimized for short processing. As for cost, Fargate charges even during idle time, but for longer tasks, it can actually be cheaper than Lambda.
I solved it by chaining multiple 15-minute Lambdas together using Step Functions. Since the files are processed in chunks, there was no timeout issue, and I was able to take advantage of Lambda's free tier, which kept initial costs low. However, if the task is complex, management can become cumbersome.
I recommend Fargate. Lambda has a 15-minute limit, making it unsuitable for processing large files from the start. Fargate is container-based, so you can flexibly adjust memory and CPU, and costs are fairly predictable. However, while there's no cold start, you need to consider the always-on running costs.
Is the task truly a periodic batch job? Or is it triggered by events? If it's the former, Fargate is better; if it's the latter, chaining Lambdas with Step Functions would fit an event-driven architecture better. Costs are similar for both, so choose based on the design pattern.
I switched to Fargate, and it felt great to escape Lambda's 15-minute limit. But the cost turned out to be a bit higher than expected, so if the job runs about once a day, bypassing the timeout with Step Functions might be more economical. How large are the files?
Try using Step Functions. Even the official AWS documentation recommends it for handling long-running batch jobs. By chaining multiple Lambdas together, each can run for up to 15 minutes, effectively giving you unlimited execution time. You can also easily add retry logic for failures. The cost is low since you're only charged per state transition.
I've used both, and Fargate was more stable. Step Functions becomes a debugging nightmare when state management gets complex. With Fargate, it's just a single Docker image, and viewing logs via CloudWatch is convenient. As for cost, Fargate is more advantageous the longer the task runs.
I've been thinking about the same thing lately lol. For now, I'm planning to test it lightly with Step Functions. Fargate seems a bit heavy to set up. If you're reading files from S3, have you considered using Lambda with S3 batch operations? That might not have the 15-minute limit.