Creating a CFN Template using Pyplates

As simple as it gets

Here’s an example of the simplest pyplate you can make, which is one that defines a CloudFormationTemplate, and then adds one Resource to it. Let’s say that this is “template.py” (a.k.a python template; a.k.a pyplate!)

CloudFormation won’t let you make a stack with no Resources, so this template needs one. Notice how cft.resources is already there for you. In addition to CloudFormationTemplate, common things that you’ll need are available right now without having to import them, including classes like Resource and Properties, as well as all of the intrinsic functions such as ref and base64.

You can see what the required properties of an AWS::EC2::Instance are here:

http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ec2-instance.html

template.py

# Start with a template...
cft = CloudFormationTemplate(description="A very small template.")

cft.resources.ec2_instance = Resource('AnInstance', 'AWS::EC2::Instance',
    Properties({
        # This is an ubuntu AMI, picked from http://cloud-images.ubuntu.com/
        # You may need to change this if you're not in the us-east-1 region
        # Or if Ubuntu deregisters the AMI
        'ImageId': 'ami-c30360aa',
        'InstanceType': 'm1.small',
    })
)

Now, on the command-line, we run cfn_py_generate template.py out.json. Here’s what out.json looks like:

out.json

{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Description": "A very small template.",
  "Resources": {
    "AnInstance": {
      "Type": "AWS::EC2::Instance",
      "Properties": {
        "InstanceType": "m1.small",
        "ImageId": "ami-c30360aa"
      }
    }
  }
}

Upload that to CloudFormation to have it make you a stack of one unnamed instance.

But now let’s do something useful

Okay, we’ve made an instance! It would be nice to actually hand it a user-data script, though. There are a few ways to go about that. Fortunately, the pyplate is written in python, and we can anything that python can do.

Note

Properties args can be either a ‘Properties’ object, or just a plain old python dict. For this example, we’ll use a dict, but either way works.

template.py

cft = CloudFormationTemplate(description="A slightly more useful template.")

user_data_script = '''#!/bin/bash

echo "You can put your userdata script right here!"
'''

cft.resources.add(Resource('AnInstance', 'AWS::EC2::Instance',
    {
        'ImageId': 'ami-c30360aa',
        'InstanceType': 'm1.small',
        'UserData': base64(user_data_script),
    })
)

Alternatively, because this is python, you could put the userdata script in its own file, and read it in using normal file operations:

user_data_script = open('userdata.sh').read()

The output certainly makes a mess of the script file, but that’s really a discussion between the JSON serializer and CloudFormation that we don’t need to worry ourselves with. After, we’re here because making proper JSON is not a task for a human. Writing python is much more appropriate.

cfn_py_generate template.py out.json

out.json

{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Description": "A slightly more useful template.",
  "Resources": {
    "AnInstance": {
      "Type": "AWS::EC2::Instance",
      "Properties": {
        "UserData": {
          "Fn::Base64": "#!/bin/bash\n\necho \"You can put your userdata script right here!\"\n"
        },
        "InstanceType": "m1.small",
        "ImageId": "ami-c30360aa"
      }
    }
  }
}

Adding Metadata and Other Attributes to Resources

Cloudformation provides extensive support for Metadata that may be used to associate structured data with a resource.

Note

AWS CloudFormation does not validate the JSON in the Metadata attribute.

Adding Metadata to an S3 bucket

s3.py

cft = CloudFormationTemplate(description="A slightly more useful template.")

cft.resources.add(
    Resource('MyS3Bucket', 'AWS::S3::Bucket', None, Metadata(
        {"Object1": "Location1", "Object2": "Location2"}
    ))
)

out.json

{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Description": "A slightly more useful template.",
  "Resources": {
    "MyS3Bucket": {
      "Type": "AWS::S3::Bucket",
      "Metadata": {
        "Object1": "Location1",
        "Object2": "Location2"
      }
    }
  }
}

Adding Metadata to an EC2 instance

ec2_instance.py

user_data_script = '''#!/bin/bash

echo "You can put your userdata script right here!"
'''

cft = CloudFormationTemplate(description="A slightly more useful template.")
properties = {
    'ImageId': 'ami-c30360aa',
    'InstanceType': 'm1.small',
    'UserData': base64(user_data_script),
}
attributes = [
    Metadata(
        {
            "AWS::CloudFormation::Init": {
                "config": {
                    "packages": {},
                    "sources": {},
                    "commands": {},
                    "files": {},
                    "services": {},
                    "users": {},
                    "groups": {}
                }
            }
        }
    ),
]

cft.resources.add(
    Resource('MyInstance', 'AWS::EC2::Instance', properties, attributes)
)

out.json

{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Description": "A slightly more useful template.",
  "Resources": {
    "MyInstance": {
      "Type": "AWS::EC2::Instance",
      "Properties": {
        "UserData": {
          "Fn::Base64": "#!/bin/bash\n\necho \"You can put your userdata script right here!\"\n"
        },
        "InstanceType": "m1.small",
        "ImageId": "ami-c30360aa"
      },
      "Metadata": {
        "AWS::CloudFormation::Init": {
          "config": {
            "files": {},
            "commands": {},
            "users": {},
            "sources": {},
            "groups": {},
            "services": {},
            "packages": {}
          }
        }
      }
    }
  }
}

Practical Metadata example for bootstrapping an instance

ec2_instance_attribs.py

user_data_script = '''#!/bin/bash

echo "You can put your userdata script right here!"
'''

cft = CloudFormationTemplate(description="A slightly more useful template.")
properties = {
    'ImageId': 'ami-c30360aa',
    'InstanceType': 'm1.small',
    'UserData': base64(user_data_script),
}
attributes = [
    Metadata(
        {
            "AWS::CloudFormation::Init": {
                "config": {
                    "packages": {
                        "rpm": {
                            "epel": "http://download.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm"
                        },
                        "yum": {
                            "httpd": [],
                            "php": [],
                            "wordpress": []
                        },
                        "rubygems": {
                            "chef": ["0.10.2"]
                        }
                    },
                    "sources": {
                        "/etc/puppet": "https://github.com/user1/cfn-demo/tarball/master"
                    },
                    "commands": {
                        "test": {
                            "command": "echo \"$CFNTEST\" > test.txt",
                            "env": {"CFNTEST": "I come from config1."},
                            "cwd": "~",
                            "test": "test ! -e ~/test.txt",
                            "ignoreErrors": "false"
                        }
                    },
                    "files": {
                        "/tmp/setup.mysql": {
                            "content":
                                join('',
                                     "CREATE DATABASE ", ref("DBName"), ";\n",
                                     "CREATE USER '", ref("DBUsername"), "'@'localhost' IDENTIFIED BY '",
                                     ref("DBPassword"),
                                     "';\n",
                                     "GRANT ALL ON ", ref("DBName"), ".* TO '", ref("DBUsername"), "'@'localhost';\n",
                                     "FLUSH PRIVILEGES;\n"
                                ),
                            "mode": "000644",
                            "owner": "root",
                            "group": "root"
                        }
                    },
                    "services": {
                        "sysvinit": {
                            "nginx": {
                                "enabled": "true",
                                "ensureRunning": "true",
                                "files": ["/etc/nginx/nginx.conf"],
                                "sources": ["/var/www/html"]
                            },
                            "php-fastcgi": {
                                "enabled": "true",
                                "ensureRunning": "true",
                                "packages": {
                                    "yum": ["php", "spawn-fcgi"]
                                }
                            },
                            "sendmail": {
                                "enabled": "false",
                                "ensureRunning": "false"
                            }
                        }
                    },
                    "users": {
                        "myUser": {
                            "groups": ["groupOne", "groupTwo"],
                            "uid": "50",
                            "homeDir": "/tmp"
                        }
                    },
                    "groups": {
                        "groupOne": {
                        },
                        "groupTwo": {
                            "gid": "45"
                        }
                    }
                }
            }
        }
    ),
    UpdatePolicy(
        {
            "AutoScalingRollingUpdate": {
                "MinInstancesInService": "1",
                "MaxBatchSize": "1",
                "PauseTime": "PT12M5S"
            }
        }
    ),
    DeletionPolicy("Retain"),
    DependsOn(ref("myDB"))
]
cft.resources.add(
    Resource('MyInstance', 'AWS::EC2::Instance', properties, attributes)
)

out.json

{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Description": "A slightly more useful template.",
  "Resources": {
    "MyInstance": {
      "Type": "AWS::EC2::Instance",
      "Properties": {
        "UserData": {
          "Fn::Base64": "#!/bin/bash\n\necho \"You can put your userdata script right here!\"\n"
        },
        "InstanceType": "m1.small",
        "ImageId": "ami-c30360aa"
      },
      "Metadata": {
        "AWS::CloudFormation::Init": {
          "config": {
            "files": {
              "/tmp/setup.mysql": {
                "content": {
                  "Fn::Join": [
                    "",
                    [
                      "CREATE DATABASE ",
                      {
                        "Ref": "DBName"
                      },
                      ";\n",
                      "CREATE USER '",
                      {
                        "Ref": "DBUsername"
                      },
                      "'@'localhost' IDENTIFIED BY '",
                      {
                        "Ref": "DBPassword"
                      },
                      "';\n",
                      "GRANT ALL ON ",
                      {
                        "Ref": "DBName"
                      },
                      ".* TO '",
                      {
                        "Ref": "DBUsername"
                      },
                      "'@'localhost';\n",
                      "FLUSH PRIVILEGES;\n"
                    ]
                  ]
                },
                "owner": "root",
                "group": "root",
                "mode": "000644"
              }
            },
            "commands": {
              "test": {
                "test": "test ! -e ~/test.txt",
                "ignoreErrors": "false",
                "command": "echo \"$CFNTEST\" > test.txt",
                "cwd": "~",
                "env": {
                  "CFNTEST": "I come from config1."
                }
              }
            },
            "users": {
              "myUser": {
                "uid": "50",
                "groups": [
                  "groupOne",
                  "groupTwo"
                ],
                "homeDir": "/tmp"
              }
            },
            "sources": {
              "/etc/puppet": "https://github.com/user1/cfn-demo/tarball/master"
            },
            "groups": {
              "groupTwo": {
                "gid": "45"
              },
              "groupOne": {}
            },
            "services": {
              "sysvinit": {
                "nginx": {
                  "files": [
                    "/etc/nginx/nginx.conf"
                  ],
                  "sources": [
                    "/var/www/html"
                  ],
                  "ensureRunning": "true",
                  "enabled": "true"
                },
                "sendmail": {
                  "ensureRunning": "false",
                  "enabled": "false"
                },
                "php-fastcgi": {
                  "ensureRunning": "true",
                  "packages": {
                    "yum": [
                      "php",
                      "spawn-fcgi"
                    ]
                  },
                  "enabled": "true"
                }
              }
            },
            "packages": {
              "rubygems": {
                "chef": [
                  "0.10.2"
                ]
              },
              "yum": {
                "httpd": [],
                "php": [],
                "wordpress": []
              },
              "rpm": {
                "epel": "http://download.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm"
              }
            }
          }
        }
      },
      "UpdatePolicy": {
        "AutoScalingRollingUpdate": {
          "PauseTime": "PT12M5S",
          "MaxBatchSize": "1",
          "MinInstancesInService": "1"
        }
      },
      "DeletionPolicy": "Retain",
      "DependsOn": {
        "Ref": "myDB"
      }
    }
  }
}

Referencing Other Template Objects

This is where things start to really come together. The instrinsic functions ref and get_att are critical tools for getting the most out of CloudFormation templates.

What if you’re using CloudFormation to describe a stack of Resources, but the goal is to try a bunch of different AMIs? Entering the AMIs at stack creation is a good way to tackle that situation, and parameters are how you do it. Use ref to refer to the parameter from the Instance properties.

While we’re here, we’d also like to prompt the user for what instance type they’d like to spawn, as well as a friendly name to put on the instance for EC2 API tools, like the AWS Console. These are also good uses for ref

We also want to put the instance’s DNS name in the stack outputs so we see it when using CFN API tools, and maybe act on it with some automation later. In this case, get_att is right for the job.

Here are some examples:

template.py

cft = CloudFormationTemplate(description="A self-referential template.")

cft.parameters.add(Parameter('ImageId', 'String',
    {
        'Default': 'ami-c30360aa',
        'Description': 'Amazon Machine Image ID to use for the created instance',
        'AllowedPattern': 'ami-[a-f0-9]+',
        'ConstraintDescription': 'must start with "ami-" followed by lowercase hexidecimal characters',
    })
)

cft.parameters.add(Parameter('InstanceName', 'String',
    {
        'Description': 'A name for the instance to be created',
    })
)

cft.parameters.add(Parameter('InstanceType', 'String',
    {
        'Default': 'm1.small',
    })
)

# Now the Resource definition is all refs, totally customizeable at
# stack creation
cft.resources.add(Resource('AnInstance', 'AWS::EC2::Instance',
    {
        'ImageId': ref('ImageId'),
        'InstanceType': ref('InstanceType'),
        'Tags': ec2_tags({
            'Name': ref('InstanceName'),
        })
    })
)

cft.outputs.add(Output('DnsName',
    get_att('AnInstance', 'PublicDnsName'),
    'The public DNS Name for AnInstance')
)

cfn_py_generate template.py out.json

out.json

{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Description": "A self-referential template.",
  "Parameters": {
    "ImageId": {
      "Default": "ami-c30360aa",
      "AllowedPattern": "ami-[a-f0-9]+",
      "Type": "String",
      "Description": "Amazon Machine Image ID to use for the created instance",
      "ConstraintDescription": "must start with \"ami-\" followed by lowercase hexidecimal characters"
    },
    "InstanceName": {
      "Type": "String",
      "Description": "A name for the instance to be created"
    },
    "InstanceType": {
      "Default": "m1.small",
      "Type": "String"
    }
  },
  "Resources": {
    "AnInstance": {
      "Type": "AWS::EC2::Instance",
      "Properties": {
        "Tags": [
          {
            "Value": {
              "Ref": "InstanceName"
            },
            "Key": "Name"
          }
        ],
        "InstanceType": {
          "Ref": "InstanceType"
        },
        "ImageId": {
          "Ref": "ImageId"
        }
      }
    }
  },
  "Outputs": {
    "DnsName": {
      "Description": "The public DNS Name for AnInstance",
      "Value": {
        "Fn::GetAtt": [
          "AnInstance",
          "PublicDnsName"
        ]
      }
    }
  }
}

Using the Options Mapping

In the introduction, there was some talk of using one pyplate to easily describe similar stacks. For example, let’s say you have a static website being hosted on any number instances running in an auto scaling group behind a load balancer. This website has a few known roles, including development, testing, and production.

Using the options mapping, you can specify different options for each of those roles, and then plug them into the pyplate when the stack template is generated, giving you a custom template for each stack role.

Options mappins are defined in YAML. Here are some examples of the options for each stack role:

mappings/development.yaml

# A development stack, so very few and very small instances
StackRole: development
AppServerAvailabilityZones:
    # us-east-1b and us-east-1c smell funny, skip those...
    - us-east-1a
    - us-east-1d
    - us-east-1e
# micros are cheap, woo!
AppServerInstanceType: m1.micro
# No more/no less than 1
AutoScalinggroupMinSize: 1
AutoScalingGroupMaxSize: 1
# Don't scale based on CPU alarms...
# Even though we can't because the max group size is 1
CloudWatchAlarmActionsEnabled: false
# SSH keypair to use, this is one that developers have
KeyPair: developers

mappings/testing.yaml

# A testing stack, so small instances but now it'll scale a little
StackRole: testing
AppServerAvailabilityZones:
    - us-east-1a
    - us-east-1d
    - us-east-1e
# We can test on smaller instances than we need in production
AppServerInstanceType: m1.small
# We're testing, so we'd love to make sure scaling works
AutoScalinggroupMinSize: 1
AutoScalingGroupMaxSize: 4
CloudWatchAlarmActionsEnabled: true
# Let's boot a buntu...
ImageId: ami-c30360aa
# SSH keypair to use, developers can get in here, just like before
KeyPair: developers

mappings/production.yaml

# A production stack, so big instances, lots of scaling!
StackRole: production
AppServerAvailabilityZones:
    - us-east-1a
    - us-east-1d
    - us-east-1e
AppServerInstanceType: m1.large
AutoScalinggroupMinSize: 4 # Need at least 4 running all the time...
AutoScalingGroupMaxSize: 100 # Really? 100? Also YAML lets you put comments here.
CloudWatchAlarmActionsEnabled: true
ImageId: ami-c30360aa
# Only admins have this SSH keypair
KeyPair: production

And here’s the pyplate:

Notice that ‘ImageId’ is absent from the development options mapping. This will trigger a prompt for the user to fill in the blanks. This does two things:

  • It gives you the ability to easily add options at runtime where appropriate
  • It helps you spot typos in options names

In our case, the former reason is what we’re after. Our developers love to boot all sorts of different AMIs and mess around, so it’s easiest just to put in a new ID every time we generate a template. Here’s what that prompt looks like:

Key "ImageId" not found in the supplied options mapping.
You can enter it now (or leave blank for None/null):
>

Generate the development template:

  • cfn_py_generate template.py development.json -o mappings/development.yaml

The ami flavor du jour is ami-deadbeef, which I entered in the prompt. You can see how it was inserted into the development.json below.

Now, generate a new stack template based on each remaining role:

  • cfn_py_generate template.py testing.json -o mappings/testing.yaml
  • cfn_py_generate template.py production.json -o mappings/production.yaml

And here are the generated templates for CloudFormation:

Go forth, and pyplate

As you can see, things with pyplates can escalate quickly. Fortunately, with the help of the python interpreter, a little bit of YAML, and CloudFormation itself, crazy templates like the above don’t have to be written purely in JSON, with no comments.

See any room for improvement? Fork this on GitHub!

https://github.com/seandst/cfn-pyplates