In our recent post “Reducing Infrustration”, we described how Radify adopted immutable infrastructure and what the benefits have been. In this post, we talk about how we integrated immutable infrastructure with our feature branch workflow.

Automatically deploying feature branches

A commit to a feature branch are automatically picked up by our Continuous Integration server, Jenkins. If the build passes, then it is automatically deployed via Ansible to a brand new EC2 node and given a CNAME so that it is reachable at {branch}.demo.{client}.{tld}:

Automated workflow for standing up demo nodes

A developer pushes code to a branch. If it passes the build on Jenkins, an Ansible playbook is kicked off, which:

  1. stands up a new micro instance based on our Radify base box
  2. installs the application
  3. creates a CNAME
  4. notifies the team that fresh code is available for testing
  5. finally, updates our minimalist open source branch manager, StationMaster

If a commit is made to an existing feature branch, this means that the existing micro node is thrown away and a new one is stood up in its stead with the latest code on it and that all the team know when a feature branch has been successfully updated and deployed.

Automatically removing expired feature branches

It’s important to keep the branch count in check - after all, every node costs money! Therefore, on every commit to any project, we run a script that goes through and checks for any expired branches. If it finds any, it deletes the CNAME, burns the node, and removes it from StationMaster. Here it is in pseudocode:

FOR EACH branch on demo
    DOES IT still have an active branch?
        YES
            leave it alone
        NO
            delete CNAME {branch}.demo.{client}.{tld}
            terminate node
            remove from StationMaster

Why do this?

For us, this replaced the way we used to work, which was having a medium instance with multiple workspaces on there with an Nginx configuration for each one. This meant that the branches performed well, but there were several problems:

  1. We couldn’t throw away the node. Because the node was shared, we couldn’t just bin it in the same way as we do with the rest of our infrastructure. This meant we had to keep on top of manual patching.
  2. Deployment method was not identical to production. See our post, 4 Principles of DevOps, on why this doesn’t match our principles.
  3. The Nginx config was SLIGHTLY different to accommodate multiple workspaces. Only ever so slightly, but in my experience, we want to be absolutely as close as humanly possible to production.
  4. Multiple workspaces on the same box makes environment variable configuration impractical, which was restricting us and overcomplicated our Ansible playbook.

So now, we have demo nodes that are handled exactly the same way as production. Great! There are, of course, at least four drawbacks.

Firstly, performance: micro instances do not perform nearly as well as the others. That said, it’s actually quite good to test your application on smaller hardware than it is targeting, it’s sometimes a useful way of getting an idea of where bottlenecks may be.

Secondly, they are not behind load balancers in the same way as production nodes are.

Thirdly, the cost. Micro instances are as cheap as it gets in AWS, but if you have a lot of branches that are long-running, it can mount up. To mitigate this, being disciplined about closing any unused branches is essential.

Finally, having demo nodes does obviously slightly increase the surface area of your application.

Benefits to the client

All the tools we’ve built to do this are fairly generic, extended slightly per-client. It means that:

  • Features can be tested before integration, meaning the dev team can check one another’s work
  • Integrations can be tested in an integration branch
  • Greater confidence that testing on demo is representative of production
  • Full automation - no need for dev team to spend time poking about on boxes
  • Ability to quickly push out alternative branches for client A/B testing

What do you think?

We've found that for demonstrating features, these small, cheap, disposable environments are ideal for us. Do you automatically push out feature branches for testing? Would you like to? Not sure where to start? Think it’s a daft idea? Let us know in the comments below or feel free to get in touch with us directly if you'd like some help setting this up!