Automated Drupal 7 deployments with Atlassian Bamboo

Update 2016/06/18: I finally fixed the markup of this post after migrating from WordPress to Hugo. I also fixed some typo’s and updated this post with some current information. Even though almost a year has passed since writing it, this post is still relevant for the current 5.12 version of Bamboo.

If you are reading this post you probably already know what automated deployment is and why it’s important. I’ll probably write a blog post about that subject in the near future but first I’m going to write this one about doing automated Drupal 7 deployments with Atlassian Bamboo.

This blog post is not going to be a discussion what the best deployment system is. Some people like capistrano, some people like jenkins, some people like Bamboo. For us at Nascom Bamboo works pretty well because it integrates perfectly with JIRA and Bitbucket, allowing us to view linked JIRA issues when making a build and see what on wich environment a JIRA issue has been deployed.

This is a pretty big blog post, so take your time to go through it.

Bamboo in action

Before I dive into the details about the setup I’ll first show you a 4 minute screencast with some minimal comments how the whole setup works. This gives a nice global picture so you have a better understanding of the steps that come next.

Prerequisites

For this blog post I’m going to make some assumptions about your development setup and Drupal site structure:

Right, now let’s dive into the setup.

Bamboo concepts

First you need to get used with some Bamboo concepts (also see the official Atlassian Bamboo documentation):

Build plans and artifacts

A build plan is the process that generates a Bamboo artifact. An artifact is something that can be deployed later, most of the time an executable or a jar file when you are talking about software that compiles, but for our Drupal site this will simply be a compressed tar file called drupal.tar.gz that contains the Drupal source code.

Build plans are composed of three pieces: Stages, Jobs and Tasks. If you look at the graph below it should be clear how those three fit inside each other:

Bamboo Build Plan Anatomy

As I wrote above, the result of a build plan will be an artifact that we can use for deployment later on.

Releases, deployment plans & environments

Now that we have a build plan that produces an artifact, our drupal.tar.gz file, we need to get that deployed to our servers. We can use releases and deployments plans to achieve that:

A real life example project

I’m going to take my own Narfum IT Services website as an example deployment project. It’s a Drupal 7 site that will be deployed to a staging and a production environment.

Update 2016/06/18: This narfum.eu site is now offline, but the example is still valid.

A Drupal site can always be split into three pieces:

If you follow my env.settings.php setup structure for Drupal 7, it’s easy to keep these 3 separated.

Our Bamboo deployment plan will only handle the first item, the Drupal PHP source. This codebase will be stored in our version control system.

The database will most of the time be deployed one time, and then updated via update hooks in Drupal. These update hooks will be run by our deployment plan (via drush updatedb), so it’s not needed to include an automated database deployment.

The user uploaded content is located outside of the Drupal PHP directory, so we can just leave that alone during deployment and just make a new symlink to it. Sidenote: you never commit this content to your version control software!

The build plan

What a build plan comes down to in practice is simply put:

  1. Download the source code
  2. Do some local modification to those files
  3. Package the result as an artifact.

This means that whatever you put in git, is not necessarily going to end up on your deployment environment. For a Drupal website this means we can do a lot of handy things during the build phase:

It will take you some time to setup all of this and make it error proof, but after that you have a fully automated build system that will never forget a single thing!

Creating a build plan

Ok, let’s start by creating a Build plan. From the “Create” menu at the top chose “Create new plan” and fill in the fields like in the screenshot below. Do not chose a version control system here yet, we’ll add that later. (If you add it here, it will be a global repository and we don’t want that).

(Click the image for a larger version)

Create a new Bamboo build plan

On the next screen just check “Yes please!” to enable the plan and click Create. We will add the tasks later, we just want an empty build plan for now. When we have our empty build plan, go to “Actions” on the right side and chose “Configure plan”. You will get the screen below (click the image for a larger version):

Configure a Bamboo build plan

As you can see Bamboo has made some default items for us: A “Default stage” stage with one job called “Default Job”. We will use these defaults for this example and just add tasks inside this one job.

Connect our git repository

As we need to do git checkouts in more than one task, we will add our Drupal git repository as a local repository for this project. On the build plan configuration page go to the “Repositories” tab and click “Add repository”:

(Click the image for a larger version)

Add a source code repository

It should be pretty obvious what you need to fill in here.

The easiest way to connect to Bitbucket is with the “Bitbucket” option, but that requires entering a password and I don’t like that. So I always chose “Git”, enter the ssh location for the Bitbucket repository and use an ssh private key to authenticate. But chose whatever method works for you.

It’s important that you chose the “master” branch here as that will be the main branch for our builds. Master should always be the code that goes onto production, so try to keep that best practice for your projects too.

If you want to read about a proper git branching model for your development, be sure to checkout the Git branching model.

Add build tasks

The last thing we have to do now is add tasks that will actually do things for us. Below is a screenshot of the real Narfum project (that has 2 stages instead of 1 but we will ignore the test stage for now) where I’m currently showing you the “Package Drupal” job.

(Click the image for a larger version)

Package Drupal job

There are 3 jobs:

The first step will always be a source code checkout. Remember that jobs can run in parallel and they are sandboxed in their own directory. So one job does not know about another job’s files, meaning you always have to checkout files (or important an artifact) as the first task in a job.

The 3 tasks in detail:

Task 1: Checkout source code from our repository.

These files will be downloaded in the root directory of our job and will be available to be modified for the remaining tasks.

(Click the image for a larger version)

Task 1: Source Code Checkout

Task 2: Magic

Once we have the Drupal code, we can do a lot of things to modify this code. We keep it simple here and just do a production compile of the SASS files for our theme:

(Click the image for a larger version)

Task 2: Magic

Notice the “Working sub directory” at the bottom! This points to the main theme directory.

Task 3: Create Drupal tarball

The last task is always creating a Drupal tarball of our files now that we’re done with modifying them:

(Click the image for a larger version)

Task 3: Create tarball

I prefer to exclude files like the “node_modules” directory rather than removing them so a new build won’t have to download them all again. (I know they are cached in the bamboo user’s homedir yes, but it’s the idea that counts here: we don’t want to re-do too many things for new builds).

After those 3 tasks are done, we will have a drupal.tar.gz file in our root directory. We now need to make this available as a shared artifact so our deployment plan can use it.

The last build step: create the artifact

In the “Package Drupal” stage, go to “Artifacts” and add a new artifact definition:

Artifact overview page:

(Click the image for a larger version)

Artifact overview

Artifact detail page:

Artifact definition

Make sure the “Shared” box is checked, otherwise it will not be available to our deployment project!

And that’s all there is do to for a Drupal build plan. If you run this build now from the “Run” menu and then “Run plan” you should get a green page saying the build was successful. You will also be able to download the artifact manually at the bottom of the page.

This next screenshot is an example build result page from the Narfum website project. You can ignore the right upper box for now, in your project that will be empty as you don’t have a linked deployment project yet:

(Click the image for a larger version)

A successful Bamboo build

The deployment plan

Still with me after the build plan setup? Good. Because now it’s time to deploy our code to an actual environment.

Create a deployment plan

A deployment plan is nothing more than a container for multiple deployment environments. From the “Create” menu chose “Create deployment project” and fill in the screen like in the screenshot below. Make sure you select the right build project you are attaching to this deployment plan:

(Click the image for a larger version)

Create a deployment plan

After that you will see the configuration page of our empty deployment plan.

Creating enviroments

Now chose “Add environment” and simply give it a clear name. I always go for the structure “$hostingprovider - $env_type” so this could be “AWS - Production”. Click on “Create and configure tasks”.

This will be our production environment but you can of course add a staging environment and testing environment too. Using the “Clone environment” option after our production environment is finished this is very fast to setup.

You should now see the screen below. This is a similar list of tasks you saw on the job configuration page for a build plan:

Create deployment tasks

There are 2 tasks made for you already, which you should always leave as first two tasks for your deployment project: the clean working directory and the artifact download. These tasks make sure you have an empty work directory with just our drupal.tar.gz artifact file.

The next tasks will then be using this drupal.tar.gz file and get it on our target environment. The exact tasks in our deployment will be:

This is what it will look like when we’ve set up all these tasks (I’ve switched to my Narfum website deployment plan again now for these screenshots):

Completed deployment environment tasks

Environment variables

Before we continue with the tasks, we first need to setup some variables. End your task setup process and go back to the deployment plan page. You will get an incomplete task warning but just ignore that for now.

Unfinished deployment environment

Click the “Variables” button at the bottom and add the variables “deploy.hostname” and “deploy.username” with the values needed for your server:

Deployment environment variables

We can now go back to configure our environment tasks.

Environment tasks

Remember that tasks inside a job can halt the deployment process if they fail? That’s the main reasons we split up all these things into separate tasks.

Task 1: Copy the artifact to the remote server

This is adding a “SCP Task” where you simply copy the artifact to the remote server. We can use the variable “deploy.hostname” as “${bamboo.deploy.hostname}” inside tasks, the same goes for “deploy.username”.

I’m also not using a password but ssh keys to login to the remote server. Sadly you have to upload the private key in every task, this is one of the few shortcomings Bamboo still has.

Task 1: Copy the artifact to the remote server

Task 2: Extract the tarball on the remote server

This tasks uses the “SSH Task” type we will be using for the rest of the tasks. It simply allows you the enter shell commands that will be executed on the remote server over an SSH connection.

This task makes a new release directory inside the “releases” directory on the server, extracts the tarball there and then deletes it again.

Task 2: Extract the tarball on the remote server

In this task we add the symbolic links to our env.settings.php file and our sites/default/files content. See this blog post how and why we do this.

Task 3: Update symbolic links

Task 4: Databases updates

This task is currently not present for my project, but you can easily add it here yourself. Make the same SSH Script as above and use whatever drush commands you would like.

The idea for this task is:

Task 5: Set this new version as the live version

This simply makes the “www” folder, which is the Apache or nginx document root, a symbolic link to the newly uploaded release folder:

Task 5: Set the new uploaded version as the live version

Task 6: Cache clear

Because this is always a good thing to do.

For most of my project I also do a sudo php-fpm reload here, to make sure the PHP opcache is cleared, but permission to execute that command needs to be set up on your server first and is outside the scope of this blog post.

Task 6: Cachec clear

Task 7: Clean up older releases

This is a nice to have task. For production environments we mostly do this manually when the server raises a disk space warning, but for testing and staging environments this can be automated.

This script only keeps the last 5 recent deployments (determined by the timestamp of the release folder) and deletes the rest. The chmod command is needed because Drupal removes the write flag from the sites/default folder:

Task 7: Clean up older releases

And that’s all for the environment tasks.

Running a deployment

Now that we have a working build plan with a linked deployment plan we can run a deployment. The steps we have to do are always:

You probably made some errors in your config along the way. Luckily Bamboo will show you a nice big log file where you can debug your problem, so go ahead and test with your own projects now. Your automated Drupal deployment setup is now finished!

Sidenote: Using Triggers it’s possible to automate deployments whenever a build runs successful. That might be a good thing to do for automated deployment to a dev or test environment, but for a production environment you still want to keep that a manual task.

Room for improvement

This blog post of course only shows a very simple deployment setup. To keep this blog post somewhat short I only covered the basic steps in creating the whole deployment setyp. It’s up to you to extended these build and deployment plans for your own project.

Here are a few pointers what can still be improved:

There is also a big marketplace of plugins for Bamboo, free and commercial ones, that make your life easier.

Newer versions of Bamboo will most likely add more useful features, so make sure you keep upgrading your Bamboo installation to the latest version.

The end.

That’s all folks, I hope you learned something useful from this post. Use the comments section if you have any questions or remarks!

comments powered by Disqus