Title

Drupal 8 DevOps: Automation for happier teams and clients

November 15, 2017| by Tim Broeker

Body

How the DevOps movement is pointing the way forward to higher quality Drupal projects, faster delivery, happier team members, and satisfied clients for projects of any scale

Over the past several years, the best practices surrounding Drupal DevOps have expanded and improved, including continuous integration, continuous delivery, continuous deployment, and automated testing. In turn more organizations are discovering the benefits of these automated tools and deployment methods:

Faster development – developers can deliver more work in less time and reduce project costs (or deliver more work for the same budget)
Higher quality – with repeatable builds and automated tests against live production content for every change we reduces stress and uncertainty surrounding deployments of new features and bug fixes
Transparent development cycles – frequent builds mean any team member can check on the progress of any task at any time
Better collaboration – team members and stakeholders can focus on the features that matter instead of collaborating over the mundane details of builds and deploys

Whether a site is under active development or already in production, these benefits result in less stress, higher quality projects, happier developers, and satisfied stakeholders.

Paragraphs

Our team hesitated before fully embracing a true DevOps strategy, for reasons similar to others who haven't taken the leap. Especially in the early days of the DevOps movement most DrupalCon presentations and DevOps-related blog posts tended to share a few things in common:

Enterprise projects – many DevOps advocates come from very large agencies working on large, complex projects with large, distributed teams
Dedicated teams – many have one or more full-time staff in a dedicated DevOps role
Supersized budgets -– many regularly work on projects with seven figure budgets that easily allow for complex DevOps setups

Drupal DevOps seems great in concept and promise. Yet historically it's perceived as difficult to implement and prohibitively costly unless you have a large enough team (or a big enough project budget) to justify the extra complexity.

But as our team discovered, even the smallest of projects can benefit from the advantages of a solid DevOps strategy. It no longer takes a Herculean effort or a large financial investment to implement a modern workflow, one that helps you deliver better projects, a happier team and satisfied clients.

When hosting platforms such as Acquia (2007) and Pantheon (2011) appeared, they immediately helped solve some significant DevOps pain points. These platforms were light years ahead of the FTP deploys of old, or manual deploys from a disconnected source repository. They provided affordable, out-of-the-box solutions for what used to be cumbersome and complicated setups financially out of reach for most clients.

Suddenly there was a base level of development tools, deployment tools, and strategies that “just worked,” and guided the way toward best practices for our agency and many others.

We determined that the advanced deployment tools available on these cloud-based platforms, combined with a solid git-based workflow, was “good enough” in terms of DevOps. It was reliable, and met both our client’s needs and our internal needs with minimal effort and cost. (For some time, these tools and best practices were a big enough deal that we could use them as a significant selling point for potential clients.)

In the case of Pantheon we automatically spin up a new environment after every Circle CI build for additional QA (if necessary) or automated deploys to any environment

Like most agencies who have adopted even a basic level of best DevOps practices (such as dedicated development and staging environments and a git-based workflow), we used a reliable system consisting of local development environments for each team member, often followed by:

Frequent git pulls from development to keep work in sync
Frequent database downloads and imports from production for fresh data
A push to a development environment after completing a fix or feature (automated deploys)
A manual deploy to a staging server for additional QA, testing, and approval
Another manual deploy to the production server
Frequent clones of production data and files back to development or staging
Rinse and repeat (and add in a few complicated Git conflicts along the way!)

This process works perfectly well most of the time. We write our own scripts to automate various processes, we communicate internally over what each team member is currently working on, the project manager keeps track of the current state of any given environment, and we coordinate the timing for things like QA and production deploys.

Clients are happy to have dedicated environments for QA and testing, and our team is happy with one-click deploys to staging or production and easy rollbacks if necessary. In fact, we have been so satisfied with this process that it prevented us from trying to improve things even further.

While our workflow was solid and mostly pain free, we were still doing things in a manual way. This can be error prone, lead to unnecessary delays, and is heavily reliant on internal communication. This sometimes led to other troublesome issues that made our work harder than it should be:

No true automation – automated git deploys to dev are nice, but our team spent a lot of time manually managing processes and moving environments around
QA breakdowns – with no automated testing, sometimes simple module updates or theme changes would periodically break things that are difficult to reliably test after deploys
Traffic management – work in progress might be sitting on the DEV environment, preventing easy deploys of other unrelated work and leading to unnecessary slowdowns
Communication challenges – as our team grows and number of projects increases, it becomes more difficult to keep everybody on the same page for any given development environment, taking away time for other more important collaboration

Our toolset proved better than adequate in most cases, but there is a much better way in the form of a true DevOps strategy–using continuous deployments, continuous delivery, and automated testing.

While DevOps is not a silver bullet that will automatically solve every problem (and is certain to introduce some new ones) we realized quickly how dramatically it can improve internal processes, ensure quality, and speed up the overall velocity of our team.

Our thinking began to change after Drupal 8 was released. We began learning how to leverage the power of Composer and the new configuration management system. After seeing some exciting demos by the Pantheon team at DrupalCon Baltimore, it became clear that Drupal 8 was especially well-suited for automation and that it was time to level up our DevOps game.

By the time our first big Drupal 8 project was underway, we found ourselves with a reliable set of tools that provide continuous integration, automated deployments, automated tests, and a customizable build pipeline for each client that delivers all of the benefits DevOps advocates have been praising for years:

Continuous integration – all work on any bug fix or new feature is done independently on stand-alone instances and frequently merged and automatically tested against the master branch. This allows us to catch problems early to prevent us from stepping on each other’s toes.
Continuous delivery – we can now develop, test, and deliver new features or bug fixes reliably and rapidly, independent of any other work being done on the site and without risk to the production site.
Continuous deployments – for sites that have launched, we can now automatically deploy things like module updates or approved site changes, all backed by automated tests prior to an automated production deployment.
Automated testing – automated Behat testing and visual regression testing to dramatically improve the QA process and site reliability.
Improved collaboration and happier teams – embracing DevOps means adopting new ways of thinking and a culture that increases transparency and collaboration between developers, designers, QA teams, and clients.

This new approach allows us faster project starts, easier collaboration, faster resolution to bugs or other site problems, faster delivery of new features, fail safe deploys, and a process that makes both our team members and our clients happier and more productive. And best of all, these tools and systems are no longer exclusive to clients or projects with large budgets or a dedicated DevOps team.

Run automated tests on a Continuous Integration server after every new feature or bug fix. Each CircleCI build installs and tests a new version of the site using live production data and the latest code

Your own approach will differ depending on your hosting platform, your budget, and previous experience of your team members but there are several key pieces to address as you embark on the DevOps journey. The basic setup below outlines the key pieces to a reasonable, DevOps-based workflow, regardless of your hosting platform or the various tools you decide to use.

Drupal install profile

We maintain a pre-configured Drupal 8 install profile that lives on Github and is mirrored on Packagist. This allows us to begin all new projects at "high altitude," with a working theme, preconfigured content types, Paragraphs bundles, Media bundles, and other elements that are common to almost all of our site builds. Note: This step is not a requirement, but it lets us hit the ground running with a continuous integration server, configured right out of the gate. There is nothing stopping you from spinning up a vanilla, Composer-based Drupal 8 install instead.

Local development setup

Having a solid local development workflow is a key piece of any continuous workflow environment. Developers create new features or fix bugs on their local machines, and push changes to Github in order to trigger various actions. At Electric Citizen, we have been experimenting with Drupal VM and more recently with Lando. Both of these tools provide easily repeatable processes that allow developers and contractors to easily spin up a local environment that is a close (or even exact) match to the production environment.

Git repository and Git-based workflow

Most modern Drupal hosting platforms these days come with a bare minimum, Git-based workflow and multiple server environments. Sometimes that is enough. But a more common technique involves using Github or a similar service to manage your canonical codebase in order to take advantage of their tools and their ability to easily interact with a continuous integration server.

In our case, the build code for each project (e.g. composer.json and any custom modules or theme work) resides on Github. On every single pull request, our code is automatically deployed both to a continuous integration server and ultimately back to one of our live web environments.

Modern hosting platform

In order to truly leverage modern DevOps techniques you will need a “programmable” hosting platform that allows your developers and other systems (such as a continuous integration server) to automate and interact with your platform. Pantheon (Terminus/Quicksilver), Acquia (Acquia Cloud Hooks), and Platform.sh (Platform.sh CLI) all provide powerful tools that allow you automate tasks within your hosting environment.

Electric Citizen has worked extensively with both Acquia and Pantheon hosting. While their tools are different, the same basic approach can be implemented on either platform (or even your own setup, though a completely custom solution will certainly entail more heavy lifting.)

Continuous Integration server

You will also need some type of continuous integration server to complete the puzzle. Essentially a continuous integration server allows you to automatically spin up and test a new version of the site each time a developer pushes a new feature or a bug fix to the git repository. In our case we are using Circle CI for this but there a variety of other popular options such as Travis CI or >Jenkins

Automated functionality tests

Automated testing is another key component to any solid DevOps strategy. Each time we push a new commit, a complete version of the site spins up on CircleCI and runs through a series of automated Behat tests to verify key functionality. If the tests pass, CircleCI automatically notifies our hosting environment, and spins up a new branch and a new copy of the site ready for any final QA or additional client feedback. When we submit a Github pull request on that branch, a final CircleCI build is triggered. If the tests succeed, the code is automatically merged into the production site.

Visual regression and load tests

We have also introduced visual regression testing to our builds using Backstop JS. Similar to our process for Behat testing, the CircleCI server automatically runs a series of predefined visual regression tests. These notify our team if even a single pixel has changed between one deployment and another. Our team has yet to integrate load or performance testing into this process but the opportunities are endless in terms of what and how you want to test.

Use GitHub pull requests to trigger builds on your Continuous Integration server and to ultimately push working code to your production environment

In the past, wiring all of these tools together was a complicated and time consuming step (and it still can be) but as the DevOps movement continues to grow, it has become much easier to leverage some of the tools provided by each platform.

Pantheon, for example, now provides a series of build tools and example scripts that make it relatively painless to have a Circle CI setup fully integrated with their hosting platform in a matter of hours. Acquia provides similar tools and starting points in the form of Acquia BLT or in the fairly new Acquia Cloud CD platform.

There is certainly no one-size fits all approach to DevOps, but regardless of your specific situation, we believe it is now possible and desirable to apply a modern DevOps approach to every project, independent of project scale or scope. Your team and your clients will both reap the benefits:

Faster development of new features
Quicker deployments for bug fixes
A happier QA team
Stress-free deployments
More productive developers
Higher quality projects
Happier clients and teams!

Here are some additional resources that might help you on your path to a better devops strategy:

Local development

Lando: https://github.com/lando/lando
Drupal VM: https://www.drupalvm.com

Git

Hosting platforms

Continuous integration

Automated testing

Behat: http://behat.org
Backstop JS: https://garris.github.io/BackstopJS
Backtrac: https://backtrac.io
Loadimpact: https://loadimpact.com

Footnotes:

About the Author:

Tim manages the technical side of our web projects, and is heavily involved with strategic planning and architecture decisions throughout the course of our projects. More about Tim »

Drupal 8 DevOps: Automation for happier teams and clients

About the Author:

Join the Discussion +

Interested?

Drupal 8 DevOps: Automation for happier teams and clients

About the Author:

Join the Discussion +

More Blog Posts

When to upgrade from Drupal 7

Layout in Drupal 8, Part 1

Twig for Drupal 8 Development: Twig Templating Part 1 of 2

Interested?