How to reduce your CloudFormation template size - Sebastian Hesse

Recently, I came across a limit which I haven’t known before: CloudFormation just allows a maximum size of 51,200 bytes per template. If you have ever reached this limit, you might have encountered this error message:

[YOUR_TEMPLATE_CODE] at 'templateBody' failed to satisfy constraint: Member must have length less than or equal to 51200

So, what are the possible solutions to reduce a CloudFormation template size? In my opinion, there are two relatively easy solutions to overcome this problem: using a preprocessor and/or using AWS::Include.

Use a Preprocessor

There are some CloudFormation template preprocessors available. One of them is cfn-include (a CLI tool) which does a good job in my opinion: it reads your template file (JSON or YAML) and can produce a minified JSON template file. Using this tool, it can reduce your template size e.g. from 50 kb to 40 kb (Note: it always depends on the content you have). Here is a small example how to use it:

# -m => minify JSON output
cfn-include path/to/cfn.yml -m > output.json

This is a first and easy step to come around the size limitation, but won’t save you forever. Other preprocessor alternatives are StackFormation or awsboxen. You might want to check them out as well!

Use AWS::Include

Another option would be to use the new AWS::Include command. Using such a simple snippet like below, you can import another template:

###################################
#### main template's content ######
###################################

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: A AWS::Include test.

Resources:
  MainFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: index.handler
      Runtime: nodejs6.10
      CodeUri: ./
      Policies:
        - AWSLambdaBasicExecutionRole

  'Fn::Transform':
    Name: 'AWS::Include'
    Parameters:
      Location: s3://BUCKET_NAME/key/to/included-template.yml



############################
#### included template #####
############################

SNSTopic:
  Type: AWS::SNS::Topic
  Properties:
    DisplayName: MySNSTopic

It’s important to note that you must only enter the actual parts in the included template which you want to include. Nothing else like parameter declarations, outputs, etc. You have to add such things to the main template! As you also might derive from the code example, all included templates must be uploaded to S3 first. Uploading it first adds some more work around a stack deployment. This can be automated in a deployment script, so it shouldn’t be a big deal. Here is an example on how to achieve this using a certain parameter to hand over the S3 Url:

# these commands expect that your main template file is cfn.yml and
# your included template file is cfn-include1.yml;


# upload included template file to S3 bucket; bucket name is hold in LAMBDA_BUCKET,
# because I also have a Lambda SAM function in the template above.
# (of course you can the bucket if you want, just replace LAMBDA_BUCKET)
aws s3 cp cfn-include1.yml s3://${LAMBDA_BUCKET}/templates/cfn-include1.yml


# save the URL of the uploaded template
INCLUDE_URL="s3://${LAMBDA_BUCKET}/templates/cfn-include1.yml"


# now package the main template, upload SAM artifacts to S3 and deploy it;
aws cloudformation package --template-file cfn.yml --s3-bucket ${LAMBDA_BUCKET} --output-template-file cfn.packaged.yml


# important here: you have to hand over the INCLUDE_URL;
# later in the template, you can reference it using "!Ref IncludeUrl"
aws cloudformation deploy --template-file cfn.packaged.yml --stack-name ${STACK_NAME} --capabilities CAPABILITY_IAM --parameter-overrides IncludeUrl=${INCLUDE_URL}

Now, when deploying the CloudFormation stack, CloudFormation resolves the included templates by downloading them from S3 and inserts them directly into your main template. Each of the included ones (and of course the main template as well) may not exceed the aforementioned byte size limit, otherwise this will fail again.

Disadvantages of AWS::Include

Although it seems to be nice that such kind of import is possible, it comes with a few drawbacks:

In the beginning, I did not know that the included template may not contain the short version of an intrinsic function. This was an issue for me, because I had to change all functions to use the full notation.
If you’re using a code completion plugin for CloudFormation templates like I do for IntelliJ, then you’ll probably see many errors in your templates, because it can’t resolve parameters or other stack resources anymore. That’s not a problem of AWS::Include directly. But the point is you’re bringing in some more complexity into your stack setup. So keep in mind that you structure your code and include templates in a good way.
You can’t export parameters into an include template.

For a more detailed view on this, please read this excellent blog post: https://thomasvachon.com/articles/making-modular-cloudformation-with-includes/

Interesting to know: AFAIK you can not outsource Lambda functions using Serverless Application Model (SAM) into separate templates. The reason is that they need to be transformed first. In that case you’d need to add other lines into your template, but then it’s not an include template anymore.

Summary

Finally, I want to say that I prefer the AWS::Include solution, because I think you just postpone the problem by using a preprocessor. At some point, you will reach the limit again and then you need to change an even bigger stack. So try to be prepared for this problem by building a modular stack from the beginning! Also take a look at my GitHub repository aws-cloudformation-templates where I have included an example using AWS::Include to reduce your CloudFormation template size.