Home / Comment permalink

Upal - a proposed test framework for Drupal 8

Meet Upal. UPal is a PHPUnit based test framework for Drupal. I hope it can replace Simpletest as the test runner in Drupal 8. It aims to run all existing tests with little or no modification. To do so, it provides DrupalWebTestCase and DrupalUnitTestCase classes which are built upon PHPUnit instead of Drupal’s Simpletest.

Motivation

  1. In my opinion, there is an intractable problem with Simpletest. Namely, even experienced developers get confused between the two Drupal sites that are in use during a test run. Those two sites are the Tester and ‘System Under Test’ (SUT). Simpletest takes heroic measures to isolate the SUT but we still are fighting fatal leakages between the two sites (e.g. all tables get deleted from tester, tester database can't be reached. Note that Simpletest 7.x-2.0 has Remote Test but that still involves two Drupals. With UPal, there is only one Drupal site - the SUT. I believe that this confusion severely reduce the pool of people who are willing and able to contribute patches to Drupal core.
  2. PHPUnit is the de facto standard for unit testing PHP apps. It is integrated into all the major IDEs (PHPStorm, NetBeans, Komodo, Eclipse), CI platforms like Jenkins, and has Selenium support built in. Most folks run PHPUnit from the command line, but VisualPHPUnit also does the job in a browser. PHPUnit has a great manual. We'll attract more talented PHP developers by moving to PHPUnit.

Next Steps - Coders needed

PHPUnit (sensibly) assumes that each test file only contains one test class. So it is not running subsequent test classes when Drupal has more than one in same file. We either need to compose our own test suite, or move subsequent classes to own in order overcome this limitation. Moving classes is probably more sensible, given our PSR-0 efforts. Let's discuss in that issue.

Upal is currently much faster than Simpletest because it restores an archived, empty Drupal site for each class, instead of installing from scratch. We can use drush’s archive-dump and archive-restore here instead of hard coding a DB dump from Drupal 7.4. That is how Drush’s test suite works. If we need to, we can just reinstall from scratch like Simpletest. I'm happy to take on this piece. Simpletest also do the ‘reinstall from sql file’ technique, so this isn’t an argument for Upal.

We also need to scope a plan for using Upal at http://qa.drupal.org. I think this isn’t too hard. We can get the test bots to send back the same XML that they send today. All the dispatching to test bots and communicating with drupal.org remains the same.

History

Drupal old timers might recall that I started the Simpletest project and maintained it for a four years. Since then, I feel like the father who watches his kid go out on a big bender. Simpletest has exceeded my dreams in many ways (30,000 assertions in Drupal core, automated testing of core and contrib patches), but it also broke some bones and sufferred head trauma along the way. Upal is my proposal for preserving the tests, while modernizing the framework.

Comments

Posted on by Devin Zuczek.

Glad to see some Drupal futureproofing.

Posted on by Isaac Sukin.

Clever name, but it sure doesn't lend itself well to searching...

Posted on by Mark Theunissen (not verified).

Awesome stuff Moshe, SimpleTest is definitely a big barrier to entry, and confusing as hell... ahh, takes me back... ;)

Posted on by Anonymous (not verified).

Elaborate please on the web runner. It runs outside of Drupal apparently and then how it finds the tests...?

Posted on by mweitzman.

The most common way to use PHPUnit is to point at a given directory and PHPUnit discovers classes that extend their PHPUnit_Framework_TestCase. Thats what Upal's DrupalUnitTestCase and DrupalWebTestCase extend. This technique works for web runner and CLI runner alike.

Posted on by Anonymous (not verified).

How do you separate tests? Drop a Drupal for one test, install another for the next...?

Posted on by mweitzman.

Upal emulates what simpletest has done for a while; it starts with a fresh Drupal at the start of each test class (not each test method)

Posted on by Anonymous (not verified).

You got rid of the test runner inside Drupal, added an outside runner which a Drupal 4 era DB dump to reload a given profile. If I get this correctly then tests can't use any Drupal API to check whether the test succeeds.

Why are we not focusing on having clean CRUD APIs so that we can use the existing SimpleTest remote test with proper cleanup? That would have the advantage of being able to test the current site as it is.

Posted on by mweitzman.

Your first sentence is pretty correct, though the 'Drupal 4 era' comment is non-constructive and inaccurate. Database backups and restores have not changed in decades. They are quite a valid to quickly restore a fresh site.

Since a huge portion of the current tests pass, you can be sure that we are not taking away capabilities that are in the current test framework. You can still run pure unit tests that derive from DrupalUnitTestCase; then the whole Drupal API is available to you.

I explained in the original post that SimpleTest Remote does not solve the two Drupals problem. Reusing an existing site is a valid use case for testing existing websites but I've only focused on the use case of Drupal core testing. FYI, Jimmy berry, the maintainer of Simpletest is on board with Upal.

Lots of people are focusing on cleaner APIs. Feel free to join us.

Posted on by Larry Garfield (not verified).

I fully support this concept, for one simple reason: Less Drupal-specific infrastructure code. Most of the PHP world has standardized on PHPUnit. It's extremely rare to find a project that isn't using it. It's a much better horse to hitch to than SimpleTest, plus we can benefit from people who are not us adding features to it. (Assuming we don't fork-and-hack like we did for SimpleTest. Let's not do that.)

That said, this won't fix the root problem: That most of Drupal is not unit testable, only system-testable. The DB-dump approach makes system testing faster, but won't help with unit testability. That's something we have to solve separately, and no testing framework is going to solve that for us.