Drupal Migrations: Pains and Good Practices

Drupal Migrations: Pains and Good Practices

Migrations are a difficult subject; they are often full of challenges, problems and unforeseen surprises (sometimes good ones, sometimes bad ones).

The Acquia Professional Services team has a lot of experience with migrations. We wanted to cover some principles and ideas that can come handy when dealing with all types of migrations, so I spoke to some of my colleagues in EMEA Professional Services to get their feedback and best practices. Here are some of the findings, successes and challenges that we’ve experienced during the last few years working on all kinds of migrations projects of varying complexities and sizes.

Make a Plan

You surely don’t want to have unexpected surprises. The best thing you can do to avoid surprises is to make a plan. It seems simple enough, but it doesn’t always happen. Obviously every project, every site, every migration is different, but an attack plan like the one below should fit most of the migration projects you would encounter.

  1. Discovery: Identify the scope of things to migrate in terms of features and data. This is also a good time for a clean up exercise.
  2. Design: Decide the approach, whether it’s all at once or progressive.
  3. Implement: Implement the prerequisites: access to source data, new data structure, architecture changes. In addition, implement the migration scripts.
  4. Migrate: Rehearse the migration and migrate production data.

Each of the steps above is critical for a successful migration.

“Investing time on planning and documenting each steps of the plan can save a lot of time in the end,“ said Sylvain Delbosc, senior technical services manager, Acquia Professional Services in EMEA.

Migrating a Monolith

When the time comes to migrate a monolith, sometimes it’s easier to start small instead of waiting for a whole huge project to finish. Commence deploying small bits of your software and present it to your users. It’s always more difficult if you have a monolith on your hands, but you still can do some things like routing to different codebases depending on paths.

Let’s imagine that you have a recipes website, with a user area, a registration form and obviously a recipes page. You could start building your new recipe page in your new platform (we recommend Drupal 8 😉). When you are ready, you can start routing your requests to recipe pages to your new codebase. There are several tools that you could use for this, but the simplest approach would be to leverage in third-party DNS, where you would only have to worry about writing the appropriate rules.

Another idea could be to start decoupling some areas. Simply switch the underlying API from one version to another.

Let’s say that you have a dedicated area where you show information to your users. Why not improve the speed of a normally slow area (all authenticated areas are prone to slowness) by adding a layer in your favourite JavaScript framework and loading just the small bits of information that you need from the API? At some point, you should be able to simply point to your new back end with no interruption at all for your users.

“A good CDN can be a salve to migrating from (Drupal) 7 to 8 if a site or application can be served from the old D7 back end on some paths, and the new on others, doesn’t work for everything, but can be a nice way to do a slow transition rather than have to be a big-bang change.“ said George Hazlewood, technical account manager, Acquia Professional Services in EMEA.

Think Micro(services)

Get Updates

Receive the best content about the future of marketing, industry shifts, and other thought leadership.

Before people went all crazy about decoupling, the same or very similar concept was already being applied in the industry; it is called microservices. Stop building massive monoliths. They are a pain to maintain long term, they are a pain to scale, and they will be a pain to migrate.

There is a saying in software development: divide and conquer. The same principle should apply at larger scale, where we are talking not about code, functions, methods and objects, but about whole pieces of business functionalities.

Discovering which pieces of those functionalities can live in isolation would be key, and communication between the different “islands” can be easily done via APIs. That means that whenever you need to update or even do the next migration, you can tackle the small parts without causing much disruption (or any at all) on the whole ecosystem.

This also applies to Drupal, you just need to be identify which bits can live in isolation. You’ll have different problems, like single sign-on, but most of the time small issues compared with the benefits of the reduced impact on the whole platform that this architecture brings.

Automate All the Things

Migration day has arrived. Would you prefer your team to have a lot of things to think and action during a stressful process, or would you prefer to simply press a button and just watch in case something goes wrong?

To me, the answer is clear. Your mind will be in a more readied state if you are relaxed, not having to concentrate on the next steps but simply watching and reacting in case something needs attention.

A process wIth manual steps, lots or not, is also prone to error (sorry, humans). The more steps the more things that can go wrong. On the other hand, the less steps your team needs to execute the easiest will be to exactly reproduce the whole process, and hence the more confident and prepared you'll be for the big day.

Of course there is also the always-present subject of budget and resources. If automating something that is not really risky and runs in seconds takes you several hours or days, maybe it’s not really worthy, so common sense also needs to be applied here. Automate all the things … if it makes sense.

Rehearse, Several Times and in Different Environments

Automation is key, but unfortunately not everything can be automated. Budget, DNS updates, manual deployments or whatever else, your infrastructure or resources might not allow to fully automate everything, or as much as you’d like to.

Have a clear script, and reduce the demo effect (things going wrong just when you are doing the live demo) practicing and repeating. Practice makes the master, and demos and big migration days are no exception. The more you practice, the more assurance you’ll have that your big day will go as expected. So, reduce the risks simply repeating the migration.

It may take time, but you must decide between the risks you are taking and the resources needed for ensuring everything goes smoothly. This risk mitigation will tell you how many times you’ll need to repeat the process, and how well (or bad) the process itself works will also tell you if you are good to go or if you may need more rehearsal until every possible (small or big) hiccup has disappeared. Maybe small hiccups are allowed, again, every project is a different world.

Rehearsal Is Not Just for Those in the Trenches

Rehearsing also goes for your business or site owners, editorial and any other roles involved in the site.

Your rehearsal should include those roles at some point in the process. They should be able to have a version of how the site will look after the migration some time before the big day, and they should test and check so everyone is happy with the results.

Having different roles also gives your a different perspective. Different people with different jobs and/or responsibilities jump directly into things that you wouldn't always test and you assume the will be “just fine.” It will also give you ideas to test that you didn’t even think about.

This could also raise false positives, but it's really worth having in mind that could open your vision to areas that you thought would be safe.

Do not underestimate the value of everyone’s review, whatever their role. They will give you a lot of valuable information.

Use the Force … of the Regression Suites

Maybe you are doing language version migration (hello php7), not a framework or CMS. With these cases, your system is not going to change, but still some things could go wrong and break. If you are not already using a regression suite maybe this is the perfect moment to do it. It does not necessarily have to cover hundreds of features and areas of your application, but maybe just those that are most critical.

TDD practitioners are master of this technique. Refactoring is fun, it allows you to improve your software, learn from past lessons and apply new concepts and ideas. But it may also be stressing as you may be touching areas that you didn’t thought it would break. A migration is not too far from those fears and concepts. If you haven’t already, invest some time on having a good regression suite. It will not just be worth for your migration, but for the rest of the life of the project.

Remember: Monitor, Monitor and Monitor Again

A migration can be complex enough, but add to the cocktail something like a platform migration, something like PHP5 to PHP7. You know things can go wrong, but you cannot be completely sure what’s going to blow. What do you do on these cases?

Well, when you are not sure what is going to bite you back (which is most of the time, most of the projects), for sure monitor all the things. Monitor everything. Monitor all the time. Monitor as much as you can.

Remember: Monitor, Monitor and Monitor Again

“Monitor everything and be extra careful with memcached. It truly bites,” said Henk Beld, technical account manager, Acquia Professional Services in EMEA.

Another way of avoiding nasty surprises is, again, to automate things. There are some tools that will help you to identify which parts of your code will break or cause problems. It is the case for example of php7mar, which will generate reports for codebases migrating from php5 to php7 with warnings of things that could go wrong, that need change, etc.

Conclusion

Every project will be different, and the pains and problems you’ll find will also differ wildly. That doesn’t mean that you cannot approach things in a methodological and structured way, so you can (and should) try to reduce risks as much as possible.

Some other random thoughts and knowledge pills from my colleagues in Professional Services:

“Media, paragraphs and field collections... it's a trap,” said Lubomir Culen, technical architect, Acquia Professional Services in EMEA.

“Migrating to an ‘enterprise’ data center: Don’t assume you’ll get things like DNS, or internet connectivity out of the box,” said Nick Downton, technical architect, Acquia Professional Services in EMEA.

What do you think? What’s your experience regarding migrations? Do you think that we have covered most of the problems? Leave your opinions in our social networks: Twitter, Facebook, or Linkedin.

Thanks to Henk Beld for his insights, expertise and time on this subject, George Hazlewood for his recommendation on tools like php7mar, my colleagues in PS EMEA, especially Sylvain Delbosc for his contribution on the “make a plan” section, Lubomir Culen, Graham Arrowsmith and Wouter Adem for their feedback and contributions.

Alejandro Moreno Lopez, Software Engineer / Technical Architect, Acquia

Alejandro Moreno López

Software Engineer / Technical Architect Acquia

Alejandro Moreno Lopez is a software engineer/architect who has worked in different roles, from pure development, to leading small and medium teams, to co-founding three different companies with different partners in technology, content management, marketing online and tourism. He's also been lucky enough to work and lead some of the biggest enterprise projects in Drupal like Royal Mail, Parcelforce, and BBC.

Before joining Acquia, Alejandro was the technical architect at BBC, leading two teams responsible for the delivery and innovation of two of the biggest brands in the corporation: BBC GoodFood and BBC TopGear.

He also worked with one of the biggest consultancies in the world, Capgemini, where he enjoyed the delivery and learning curve of complex tools and architectures of the enterprise world.

Born in sunny Alicante, Alejandro moved to London exchanging sun, good weather and exceptional food for interesting projects, incredible teams and amazing challenges. He still doesn’t regret that choice... most of the time.

He loves learning languages, everything boarding (skateboarding, snowboarding, kitesurf, …), cycling, and spending time with his daughter, his wife, and his French Bulldog.