Home / Comment permalink

How to test 20 000 Drupal 7 core patches

Summary: 20 000 patches submitted by 1200 contributors for Drupal 7 will require:
* testing tools development with patch testing automation
* 1200 core contributors to be trained on how to write and maintain core tests

The tantalizing goal The Drupal project lead has presented an interesting challenge to the Drupal project's core development contributors. If contributors add 100% testing coverage to Drupal 7, they can reap the benefits of automated testing by having four more months of feature improvements in the development cycle. Dries has left the Drupal community drooling for more development time.

The state of Drupal testing There are four kinds of testing the Drupal community is working on: Functional simple tests, Unit simple tests, Javascript testing, and user experience testing.

* Functional simple test status.
* Unit simple test framework plan for core. Karoly has a slimmed down version being prepared for core.
* Jquery test framework. We've identified that there is a framework for testing jquery, but we've not yet found leadership to begin jquery testing for Drupal core.
* User experience testing. We recently conducted usability testing at the University of Minnesota. We are also investigating how we can work closely with core developers to have human readable test cases. The pool of people who know exactly how HEAD works is very small at any given time. Real quality assurance requires ingenious humans to be able to keep up with how Drupal 7, a.k.a. HEAD, is supposed to work at any time in the development cycle.
* Patch testing automation. In Drupal 6 development process there were 9425 patches submitted by 741 individuals. Many of those patches are broken, stale, or would break functionality or APIs in Drupal. Patch test automation can help by reducing the number of malformed, stale, or broken patches that collaborators have to test, and the patches that core maintainers have to review. If you want to learn more about how software for testing software is helping with project success, read Software that makes software better.
* Test tool development. As we are increasing our efforts in testing, it becomes necessary to modify, and create tools, like simpletest automator, to make the process of testing more effective.

The problem with Drupal testing

The Drupal community has already begun to embrace testing for Drupal 7. Core contributors are beginning to provide additional tests, or improvements to existing tests along with their patches. If the Drupal 6 development cycle grew to 9425 patches submitted by 741 unique developers, then it's possible the Drupal 7 development cycle will have 20 000 patches with 1200 or more core contributors based on growth rates between previous releases. This presents two challenges for the Drupal community. First, test creation tools, patch testing automation, functional test creation, unit test creation, and maintenance of those tests against the HEAD branch present serious systems integration and collaboration challenges. Second, assuming this complex infrastructure is working flawlessly then we must go about training 1200 potential core contributors. Contributors will need to make a huge initial investment in test driven development and change the way they contribute to Drupal. This may takes years. We are going to need a full suite of development best practices and techniques to succeed.

The primarily method for developer communication is the project module that runs on Drupal.org. We had approximately 200 000 issues and follow-ups in 2007. Thanks to great maintenance by Derek Wright, Chad Phillips, Adam Light, and Jimmy Berry the project module should scale to handle significant growth. Our infrastructure team lead by Gerhard Killesreiter with lots of leadership by Narayan Newton and backed by reviews from Jeremy Andrews and David Strauss should allow the Drupal infrastructure to scale to meet increases in both patch submissions and testing.

Steven Peck and the documentation team continue to make great progress on the huge challenge of gardening Drupal's documentation. They will continue to provide a solid framework for patching and test creation documentation to be created. The popular Drupal dojo is a great model for trainin. Charlie Gordon and as well as others will step up to bootstrap the Drupal communities test creation and test maintenance instruction with Drupal dojo. Don't get me wrong, none of these efforts are without their own great challenges, and we are relying on dozens of contributors to perform as admirably as they have in the past. The beating heart of Drupal is the 24/7 banter and friendly collaborative discussion that go into IRC channel #drupal, now one of half a dozen specialized Drupal channels. The #drupal IRC channel will continue to be the friendly introduction for person to person support, and test development training, the community does so well.

However, there one is technique the Drupal community has relied on to make great progress on in the past that we can use again. That's the Drupal developer sprint. We've done sprints at:

* Drupalcon Amsterdam 1 day bug-fix sprint for 4.7
* OSCMS Yahoo had a sprint
* St. Charles, Illinois lead to Forms API III
* At Drupalcon Barcelona, the CCK redesign efforts are proving to be successful
* The Data architecture sprint in Chicago led to Database TNG and the registry patches

A testing sprint could be critical technique for ensuring the complex infrastructure we are building is ready to support 100% test coverage.

The solution for Drupal testing

* Documentation
* Drupal testing dojo
* Project module support
* Infrastructure support for Drupal.org and patch testing on http://testing.drupal.org
* Drupalcamp testing training
* Teach customers to demand testing with their Drupal site
* Simpletest auto
* Framework for unit testing
* Written test cases so non-programmers can help test Drupal 7 during development
* Check Drupal planet to see how you can help the testing project leads collaborate in person

Comments

Posted on by dmitrig01 (not verified).

In my opinion, the existing JavaScript testing frameworks are both inadequate for our needs.
I have written a patch, http://drupal.org/node/237566 (reviews, please), that can do this better.

Additionally it's Steven Peck not Stephen Peck ("Stephen Peck and the documentation team continue to make great progress on the huge challenge of gardening").