Reusable CloudFormation Snippets with ERB

Amazon’s CloudFormation is a wonderful and flexible tool for provisioning and managing resources in an EC2 VPC. It really takes the concept of infrastructure-as-code and helps make it a reality. For all its flexibility though, it sacrifices intuitiveness and ease. It is also limited by the rigidity of JSON, which isn’t a full-fledged language so it doesn’t support variables (although Parameters, Mappings, and References to them are a long-winded and difficult to parse approach that comes close) or easily referencing reusable external libraries. It also isn’t possible to define arbitrary functions, iterate over lists, or define anything but the most rudimentary conditional sections. This is by no means a criticism of CloudFormation, as it has certainly done a lot to turn a serialization format into a pseudo scripting language, but these are my observations that might frustrate other people when using it.

That’s where ERB comes in. As a big advocate of Ruby, whenever I think of templating the first thing that comes to mind is ERB. A while back, I put together a super simple script that generates templates from JSON “layouts” and “snippets” (think views and partials from Rails), both of which fully support ERB and all of its Ruby goodness.

To get started, let’s look at the code:

Seriously, that’s all there is to it. It is really, really simple. All standard library ruby, so it’ll run on most any Ruby interpreter (even on Windows). To run it, create a directory called “layouts”, then drop a JSON file in there with the extension “.json.erb” rather than just “.json”, then run the ruby script, passing an argument of the name of the JSON file without the “.json.erb” part. If you haven’t done anything with Ruby in the file, all this tool will do is act as a really horrible lint tool for your JSON (it errors out with bad JSON, but can’t tell you where the problem). It always produces human-readable output too, including proper indentation and spacing, so it can clean up ugly JSON for you.

But, if you want to actually get some real value out of the tool, start using some Ruby in your JSON. To do this, there’s one simple helper baked-in to allow splitting up those unruly, monolithic CFN templates: snippet(). With minimal effort, you can quickly take large chunks of JSON from your template and move it to a snippet, just by putting it in a file under a “snippets” directory. In your main file (called a “layout”), replace the original JSON text you moved with this:

This assumes you moved that JSON to a file called “name_of_snippet.json.erb” under the “snippets” directory. This simple step of breaking out chunks of JSON into snippets is immensely helpful, because it allows you to easily reuse the same boilerplate (like AMI id lists, IAM roles and policies, and instance types, just to name a few). It also makes it clearer what a “layout” does, by reducing the junk you need to sift through and parse.

Some additional advantages that come for free with ERB are things like wrapping sections of the template (like resources, output, etc) in conditionals in Ruby, iterations to produce multiple items that are nearly identical (like multiple ELBs, Subnets, etc.), comments in the ERB layout (though these won’t make it to the template output, they’re only useful for the anyone viewing the layout file), variable definitions that don’t prompt the person running the CFN template, and more.

To make snippets more powerful, I added the ability to pass a Hash of parameters to a template, allowing you to produce dynamic templates that behave differently based on how they’re called. As an example, CloudFormation Parameters allow you to set a default, but for some CloudFormation templates, it makes sense for that default to differ. While the overall choice list should be the same, most of the time you’ll want to have sane defaults so running the template as-is will produce something useful. Well, the overall list of instance types in AWS is long and is exactly the kind of reusable cruft I like to pull out of my templates, but I like the default value to be what I would expect for a given template. A java app server might make sense to default to “r3.large”, while a NAT instance might be fine with “t2.small”. So, to make the same list of instance types usable as a snippet for both these scenarios, we can pass a parameter to the snippet helper like so:

Now, in the snippet called “instance_type_parameter.json.erb”, I can do this:

Notice that my ERB code looks like this:

This is a Ruby way of saying if there’s a parameter passed called “default”, use it, otherwise go with “r3.large”. This way, “r3.large” is my default if you don’t do anything but call the snippet, but you can pass your own value when you call the snippet if you’d like as well.

One thing to watch out for that might be even more frustrating when using this tool is commas. To be the most flexible and safe, never ever end a snippet with a comma, and always aim for snippets to be valid JSON in their own right (meaning as-is, they should pass a JSON lint test). Pay attention to quotes as well. If you’re used to writing Ruby and letting ERB return a String, you usually don’t have to worry about wrapping your ERB in double-quotes, but this is required when using the params Hash, as JSON requires strings to be enclosed in double-quotes.

Overall, this approach allows the actual compiled template output by the tool to be ephemeral, especially if you version control this layouts and snippets you create (which I highly recommend). It allows for cleaner and more readable CloudFormation templates, quicker turnaround on similar changes to lots of templates (if they share a snippet), and lots more power to generate just the content you need.

I recently decided to commit it to a bitbucket public repo, so feel free to check out the source and the README there. I’ll be adding lots more examples in that README soon.

Leave me a comment if you have any questions or if you’re interested in the tool.