Do you recognize this view when looking into your CloudWatch log groups? Each AWS Lambda function has an associated CloudWatch log group. However, there is no cleanup process available as soon as a relationship between a CloudWatch log group and Lambda function expires. In that case it’s necessary to remove these old log groups manually. In this post I’ll show you an easy way to always have a clean set of CloudWatch log groups by automatically removing old log groups.
Before you start: use the following script with care! I do not recommend to use this in any AWS production account. Instead, you could for example use it in your own developer account.
Removing old CloudWatch log groups which do not belong to a Lambda function anymore is a tedious process. I guess you don’t like to remove them manually, right? Same for me. Therefore I’ve posted a small script on GitHub which is removing such old CloudWatch log groups. The are multiple reasons why old and unused CloudWatch log groups exist. One of reason is that a Lambda function got a new name or simply does not exist anymore. Like in the screenshot above, the name simply changed from “MyFunction” to “NewFunction”.
But there are ways to automate this process. My script works basically like this:
- Get all Lambda function names from your target CloudFormation stack, e.g. my-stack-MyFunction-1A2B3C
- Get all CloudWatch log group names, e.g. /aws/lambda/my-stack-MyFunction-1A2B3C
- Loop over all CloudWatch log group names
- Retrieve the Lambda function name by removing /aws/lambda/ from the log group name
- Remove all log groups which do belong to the CloudFormation stack, but do not match with any function
As you can see, this script only works if you use CloudFormation to manage your Lambda functions. However, if you adjust the code to your needs, this should also work in different situations. You only have to make sure that you don’t accidentally remove too many log groups. It’s up to you how you identify that. Eeeasy!
Another recommendation from my side: Put this cleanup Lambda function into a separate maintenance stack and let the function execute each night. For example, use a Rate Expression to schedule your Lambda function with ‘rate(1 day)‘ for a daily execution. Then you’ll have a clean set of CloudWatch log groups every day. Furthermore, such a maintenance stack is a nice thing, because it gives you also further options like shutting down certain AWS resources over night which will save you money as well.
[mailjet_subscribe widget_id=”3″]