Home / Comment permalink

A pragmatic guide to the Branch Per Feature git branching strategy

[Update 09/09/2013] I mentioned below that we'd make our scripts available for setting up BPF. Well, in collaboration with the awesome team at Affinity Bridge, we've gone one better than that: there's now a ruby gem available for this at http://rubygems.org/gems/git_bpf. Use this gem to initialize a repo for BPF, including setting up the shared rerere cache (which is actually in the same repo as your main repo, using an orphaned branch) and the hooks that will push to and pull from the shared cache. It also provides a command for recreating the QA branch from scratch.
-----------------------------------------------------------

When the Drupal Gardens project first switched from Subversion to Git, we adopted the popular git-flow branching model. While this model clearly works for a great many projects, it does not suit every workflow, and not long after we adopted it we decided it didn't suit ours. The branching strategy we ended up switching to is the Branch Per Feature (BPF) model described in this post by its originator, Adam Dymitruk. We were drawn to this model because of the level of control it gives us over what we do and do not include in a given release, and it ties in well with our use of Jira for issue tracking. Here is BPF in a nutshell:

  • All work is done in feature branches
  • All feature branches start from a common point (branched from master at the point of the last release)
  • Merge regularly into a throw-away integration branch to resolve merge conflicts and ensure tests aren't breaking
  • QA branch is created off master and gets all completed features merged into it; it is recreated every time a new feature becomes ready, or when a feature that was being QA'd has been deemed not release-ready (so it gets recreated without that feature branch in it).
  • Manual testing is done on QA branch, whatever doesn't pass gets left out next time it is recreated
  • When ready to release, QA branch is recreated with all features that passed testing, merged into master & tagged

At the heart of this branching strategy is the idea that your feature branches are completely unadulterated by anything else happening in the repo - their history should be a series of commits, each one of which is related solely to the feature itself (i.e. no back merges). Here's an illustration of this clean commit history:

Branch Per Feature commit history
(Feature branches are in pink, QA in green, and the master branch in blue.)

The blog post by Adam Dymitruk linked above does a great job of explaining the rationale for adopting this strategy, but we found it a little short on details with regard to how to implement it. There are certain key pieces, such as the use of a shared rerere cache, that took some figuring out before we could make the switch. We've now been up and running with this model very successfully for several months, and this post aims to complement Dymitruk's by filling in some of these practical details.

But first, some cautionary words, as it is easy to end up Doing It Wrong with this branching strategy.

1. Do not deploy the integration branch

The integration branch will always contain the latest commits from all features, because the developers working on those features will have been merging into it continuously as they go. It can therefore be tempting to deploy the integration branch to a test environment to see how things are coming along. This is an abuse of the integration branch and can lead to an endemic misunderstanding of what the branch is for - it is purely for ensuring that commits are able to merge with each other and are not breaking tests. It might therefore contain all kinds of cruft from features that were never completed*. At any time during the sprint if you need to test out a feature in a test/staging environment, recreate the QA branch with that feature in it and deploy that. You can always recreate it again without that branch if it proves itself not really QA ready yet.

2. Do not make creation of the QA branch the task of one single person

The QA branch should get recreated from scratch every time a new feature becomes ready for QA. Don't leave this until the end of the sprint and make it the responsibility of a dedicated release manager. The merging of a feature branch into QA is the responsibility of the developer who worked on that branch. This is important because even though merges into integration will catch most merge conflicts with other features (which are resolved and then the resolution shared using the rerere cache as explained below), it is possible that the order of commits is different when merging into QA so conflicts can easily arise for which no existing rerere will work. The best person to deal with a merge conflict is the person who worked on the feature being merged in. When they resolve conflicts at this point, the shared resolution will work for all future iterations of the QA branch recreation (because the branches get merged in in the same order each time)**. Once we realized that each developer would need to be able to recreate the QA branch, we put together a script that can be set up as a git alias so that this process amounts to running one simple command.

The Process

Here's the complete process of working on and completing a feature during a sprint:

$ git checkout master && git pull origin master // Make sure to be working off the latest code in the master branch
$ git checkout -b J-1234 // Where J-1234 is the Jira ticket this feature corresponds to
// Hack hack hack
$ git commit -am "My first commit on my feature branch"
$ git checkout integration && git pull origin integration // Make sure I have an up-to-date local integration branch
$ git merge --no-ff J-1234
// Resolve any conflicts that arise, they will automatically be pushed to the shared rerere cache
$ git push origin integration
// Hack hack hack and repeat merges to integration
// Feature is ready for code review, submit pull request, get feedback etc., hack some more if necessary
// Now my feature is ready for QA
$ git newqa // This is the alias to the script for recreating the QA branch from scratch
$ git merge --no-ff J-1234
$ git push --force origin QA // Force-push the new QA branch
$ git push origin J-1234 // Make sure my branch is on the remote for future QA branch recreation by other devs

The nitty-gritty

And now for the nitty-gritty on how to set this up. Bear in mind that the set-up described below is in use for a single product codebase, i.e. our team is working on one ongoing project, not a sequence of new client projects. While this strategy may still be suitable for teams working on multiple client projects, some adjustments might need to be made to some of these implementation details.

Dealing with merge conflicts

When your feature branch is merged into the QA branch immediately prior to the QA branch being merged into master, this will be the nth time it has been merged in, where n is the number of times the QA branch has been recreated from scratch since your feature branch became ready for QA. If n = 1, the person doing the merging is you, the developer of that feature, and so any merge conflicts that arise can be easily dealt with by you. However if n is greater than 1, the chances are it is another developer doing the merging, and this developer likely won't know how to resolve the conflict (and certainly won't want to be dealing with it anyway.) So we use git's rerere feature to reuse recorded resolutions.

There's a great explanation of the rerere feature here. In short, "it allows you to ask Git to remember how you've resolved a hunk conflict so that the next time it sees the same conflict, Git can automatically resolve it for you". To set it up, run
$ git config --global rerere.enabled true
And to make sure that whenever a conflict is resolved in this way it is automatically staged and ready to commit, run
$ git config --global rerere.autoupdate true

Rerere in action

As the developer of a feature branch, you have been dutifully merging into the integration branch on a regular basis and resolving merge conflicts as they arise. With the rerere feature enabled, you record these resolutions. When your feature is finally ready for QA you recreate the QA branch to include your branch, and your recorded resolutions are used to resolve the conflicts (or there may be new conflicts, due to a difference in the order of the commits between the integration branch and the QA branch, in which case you resolve them and record the resolutions).

Later on, your co-worker finishes a feature she's been working on and now she needs to recreate the QA branch to include her branch. As she goes through the previously merged in branches, she comes to yours which has a conflict - how does she have access to your previously recorded resolution? This is where the idea of a shared rerere cache comes in. Normally, git's rerere cache is not something that gets shared between developers because it doesn't get included when you clone a repo - it's your own special cache in the .git directory of your local tree. The solution we implemented to this problem was along the lines of the suggestion here, i.e., to share the rereres in a git repository.

We created a separate "tools" repo that contains, among other things I'll get to later on, a rerere directory. As a one-off configuration step, each Gardens developer clones this tools repo and runs the following commands from within their regular Gardens repo:

$ rm -R .git/rr-cache
$ ln -s /path/to/my/clone/of/tools_repo/rerere/gardens .git/rr-cache

Now when a developer resolves a conflict in the gardens repo and a new rerere is recorded for it, it will be placed in the rerere directory of the tools repo, ready to be added, committed and pushed.

Shared hooks

The obvious question then is, how do we make that "add, commit and push" part happen automatically as soon as the resolution happens? And the answer to this is git hooks. You can read up on git hooks here, but in short, they are just executable scripts that get run when particular actions happen. You can have pre-commit, pre-rebase, post-merge, etc. hooks. You can write them as simple bash scripts or use Ruby or Python for example. They reside in the hooks directory inside the .git directory of your local repo. Again, they are a local thing, not shared when a repo is cloned. So again, we use our tools repo to share these hooks. From within their regular Gardens repo, each developer runs:

$ rm -R .git/hooks
$ ln -s /path/to/my/clone/of/tools_repo/hooks/gardens .git/hooks

as another one-off configuration step.

Inside the hooks/gardens directory of our tools repo, we have a very simple post-commit script that checks whether the commit was a merge commit (the post-merge hook does not get triggered if there was a merge conflict, even if it was successfully resolved using rerere) and if so, goes to the rerere directory to see if there are any new rereres. If it finds one, it adds it, commits and pushes up to the tools repo so that everyone else will get it when they pull from the remote***.

Recreating the QA branch

I mentioned a script for recreating the QA branch, which developers have set up as an alias. All they have to do is run
$ git newqa
and the branch gets recreated with all of the feature branches merged into it that had been in the previous QA branch. So, what does this script do exactly? We will be making our script, which is written in Ruby, available once we figure out the best way to package it up for broader consumption, but in the meantime, here's what it does:

  • If you don't specify a branch name, it assumes a branch name of "QA" and looks for such a branch in the "origin" remote of your repo
  • If such a branch exists it parses its history to come up with a list of branches that were merged into it since it was branched off of master
  • It creates a new local "QA" branch off master (having deleted any previously existing one, but not without prompting you for approval) and merges in the remote branches from the list, one by one, resolving conflicts using shared rereres
  • Alternatively, if you specify a file containing an explicit list of branches to merge in, it will use this list instead, but can warn you if any of the branches was not merged into the previous QA branch
  • If you have specified any branches to exclude using the -x switch, it will not merge them in, regardless of which list it is using

Once you have run the newqa command, you are ready to just merge in your own new feature branch and force-push the new QA branch to the remote.

Some clarification around long-running feature branches and hotfixes

With the BPF model, any features that don't make it into the release need to be rebased against the master branch after the release, so that they are then starting from the same point as all new features. We initially thought this meant that when we released a hotfix all feature branches would need to get rebased against master, but this doesn't really make any sense. Hotfixes are unlikely to be relevant to the feature branches in progress - the QA branch, when it gets recreated next time around, will have the hotfix, so there's no need for feature branches to be concerned with it. And besides, rebasing carries with it some dangers, especially if more than one developer has been working on a branch, so insisting that all feature branches be rebased against master after a hotfix simply isn't worth the hassle.

Interested in trying it?

This is a powerful branching strategy that may well be a good fit for your project - I hope the information presented here is helpful in determining that. We intend to make all the scripts and hooks I've mentioned available in the near future, but in the meantime for anyone happy to have a go at writing their own, I've tried to make it clear exactly what they need to do. I cannot stress enough, however, how important it is that every member on the team understand the fundamentals of this strategy. I highly recommend reading Dymitruk's blog post about it and also this Google plus thread which has a huge amount of discussion and some interesting insights on it.

---------------------------------------------
* It can happen that the integration branch ends up containing a lot of cruft and it makes sense to recreate it from scratch off master - in this case all in-progress feature branches need to get re-merged into it.
** There are exceptions to this but they are fairly edge-casey - e.g. if a hotfix goes into master and is therefore in QA when the QA branch is recreated off the lastest master, if that hotfix touches code that is also touched by one of the features, a new conflict can arise. In that case the owner of the feature affected needs to resolve the conflict and recreate the QA branch.
*** This is one step we haven't yet automated, i.e. the updating of your local tools repo, but it just means that each developer has to ensure they have an up-to-date tools repo with all recent rereres before they recreate the QA branch.

Comments

Posted on by Robin Puga (not verified).

Good read Katherine.

Thanks for turning us onto this powerful model. It's been super helpful in making our deployments sane. :-)

Posted on by David_Rothstein (not verified).

Very timely - I've been looking for a new way to handle branches to recommend to clients/colleagues and this is the first branch-per-feature concept I've ever seen that actually made sense to me. Nice job!

One question:

> QA branch is created off master and gets all completed
> features merged into it; it is recreated every time a new
> feature becomes ready, or when a feature that was being QA'd
> has been deemed not release-ready (so it gets recreated
> without that feature branch in it).

I can see why you would need to recreate the QA branch when a feature gets pulled out (or whenever plans change otherwise), but why would you need to recreate the whole branch when a new feature is ready? Couldn't you just merge the feature into the QA branch directly?

And one comment:

For the particular use case I'm thinking for this, I'd actually consider a simplified version that leaves out the integration branch and git rerere. The reason is, this would be for clients who basically have a site they are happy with but are on a maintenance plan, so (a) the "features" are often bugfixes and relatively small additions to the site (not major changes) and therefore very unlikely to have merge conflicts, and (b) some of the clients have their own developers working on the site occasionally, and I think the integration branch and git rerere concept might be too complicated to explain to a large and diverse group of people.

Perhaps the first time a merge conflict and a "change of release plans" event do arise at the same time I'd regret that decision :) But otherwise seems like it might be a simpler way to do this for certain use cases.

Posted on by Katherine.Bailey.

why would you need to recreate the whole branch when a new feature is ready?

The reason for doing this is to ensure that the QA branch is completely up-to-date. Suppose there had been a hotfix since the last time it was recreated - recreating QA from scratch ensures it starts out with the latest code from master. It's also entirely possible that developers have pushed minor changes to their feature branches in the meantime (even if they're not supposed to) so this ensures the latest commits from those branches get pulled in as well.

I think the simplified version you mention could definitely work, especially if the project is not doing any continuous integration. It would mean that merging a feature into QA is potentially much more difficult because each commit could have conflicts needing to be resolved. But if the "features" are more like bugfixes as you say, they are probably only one or two commits anyway so this doesn't matter so much.

Posted on by David_Rothstein (not verified).

Thanks!

Hm, I assumed that whenever a hotfix goes out, a new QA branch would be generated in response to that directly. I see what you are saying about developers committing code to a feature branch after it was merged in, though. (Although if the last feature to be merged in is the one where someone does that, it seems like you'd miss it anyway.)

I think in our case, it would not be too likely for developers to do that since we'd be planning to do as much QA as possible on the feature branch itself (and only use the actual QA branch for "final" QA, sanity checking, making sure all the features play nicely together, etc). This means than when a feature is merged in, the ticket in which it was implemented would have its status changed to something approximating "closed". So the developer would hopefully know they were doing something wrong by continuing to work on a closed ticket :)

Posted on by Yuriy Gerasimov.

Katherine, thank you for the nice explanation.

Please advise how do you organize creating virtual hosts for newly created branches?

I am looking for the system that will automatically create a virtual host, clone database, copy files when we create a new git branch. And of course does complete cleanup after we delete the branch.

Thanks

Posted on by Katherine.Bailey.

Hi Yuriy,
that's not actually part of our current workflow. I guess maybe you could do something with a post-receive hook..? See http://git -scm.com/book/ch7-3.html#Server-Side-Hooks.

Kat

Posted on by David_Rothstein (not verified).

We've been thinking about the exact same kind of setup...

Yeah, a Git hook combined with something like Apache VirtualDocumentRoot is probably the way to go. (The latter is key, since it means you just need to create a directory in the filesystem with the code in it and Apache should automatically start serving the new copy of the site from there.)

Posted on by Diego Carlton (not verified).

Chicago says Hello to Katherine,

First thanks for the time and energy to share this. I am adopting Git and researching workflows to manage development and builds of 20 plus and growing different client releases of a handful of batch applications I need to support.

First, when you say you merge the final QA branch to master, are you rebasing QA with master prior to merging? then deleting the feature branches which made it into the release?

Second, I understand the need to have developers merge after a commit to a shared integration branch. Do I correctly assume the integration branch is created from origin/master from the point of the last release? And, do I assume that the integration branch remains until the next release? Meaning where as the QA branch is recreated when the completion of a new feature is ready for testing, the Integration branch remains until the final QA branch is merged back onto master.

Third, what happens to the integration branch when a feature is dropped (as in it was a requirement but now VP says to forget about it until I tell you otherwise ) mid-development from the upcoming release? Meaning other features are still undergoing development for the upcoming release. Do I assume to recreate the Integration branch like you do the QA except to leave out the dropped feature?

Lastly, is the submit pull request executed, or is it just for the purpose of a code review? I do not see how the pull request would actually be executed since the developer would have already pushed to origin/Integration and origin/QA.

Thanks and I am definitely hope you publish the $ git newqa.

Diego

Posted on by Byron Sommardahl (not verified).

I'm working on implementing your plan for one of my projects. I'm stuck on the git hook piece. Can you share your post-commit hook that adds/commits/pushes the rr-cache folder? Great article!

Posted on by Katherine.Bailey.

Hi Byron,
there's now a gem available for doing all this set-up. See my update at the top of the post.
Kat

Posted on by Ivo (not verified).

Hi Katherine,

Thanks for a very informative post on this workflow. It really helped in understanding the finer details.

I've created two (python) scripts which add a 'release' and 'integrate' command to Git. It is mostly based on the fine work by Acquia but I've hopefully removed the need for hooks and sharing of the rerere cache. Instead, I've used the fact that Git is able to learn the conflict resolutions from history. This means that the script should be able to fix most (all?) conflicts by itself. The scripts and link to the accompanying blog post can be found here: https://github.com/ivov/bpf. I would love some (positive or negative) feedback if you decide to use it.

Posted on by Steve Martini (not verified).

Hi Katherine,

You mention you were drawn to the BPF model because of the level of control it gives you regarding what you do and do not include in a given release, and it ties in well with your use of Jira for issue tracking. Were these the reasons why git-flow did not suit your workflow, or were there other reasons why git-flow was not right for you?

Was there some apprehension when you started using the BPF model about merging into the QA branch and not running automated tests?

Do you feel like you've given up any capabilities moving away from git-flow or added any unexpected consequences?

Thanks

Posted on by Kevin L. Pauba (not verified).

In the section titled "Some clarification around long-running feature branches and hotfixes", you only talk about hotfixes.

Is there any more that you have to say about long-running feature branches?

Thanks!

Posted on by Ralf Stephan (not verified).

They are the problem of that strategy: if there are several authors collaborating over the time of several releases, every time someone takes a look at that old ticket (by checking it out) touches many of his source files and has to pay a huge fee of recompiling for just the look. After that, going back to her own branch: again. So you end up applying patches as in the pre-git era, unless you are sure to commit yourself. Do I miss something?

Posted on by Oyunokata (not verified).

You say that moved to this model and away from git-flow because "We were drawn to this model because of the level of control it gives us over what we do and do not include in a given release, and it ties in well with our use of Jira for issue tracking"

Just curious as to how git-flow did not provide this functionality and control. We are using git-flow and Jira at my company and I have not had any issues of this sort. Can you please elaborate?

How to bug fixes (not hotfixes) get incorporated. Are bugs treated the same as features? If someone is in the middle of a feature and finds that they need a certain bug fix how are you handling this? Cherry-picking? merge, integrate, re-branch?

Posted on by Vinicius Oyama (not verified).

Hi, great article. I'm trying to understand this model but have some questions and I will be gratefull if i you can help me:

In this GiT model all feature branches start from the same commit and never receives any other code.

Let's say that I'm working in a Ruby On Rails Project and there are two features need to use the User model. The file /models/user.rb will be created twice? And the conflict must be resolved in the future?

Doesn't this branch model introduce to much time resolving merge conflicts?

Add new comment

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.

Filtered HTML

  • Use [acphone_sales], [acphone_sales_text], [acphone_support], [acphone_international], [acphone_devcloud], [acphone_extra1] and [acphone_extra2] as placeholders for Acquia phone numbers. Add class "acquia-phones-link" to wrapper element to make number a link.
  • To post pieces of code, surround them with <code>...</code> tags. For PHP code, you can use <?php ... ?>, which will also colour it based on syntax.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <blockquote> <code> <ul> <ol> <li> <h4> <h5> <h2> <img>
  • Lines and paragraphs break automatically.
By submitting this form, you accept the Mollom privacy policy.