Faster PHP code coverage

Or how we switched to a powerful new test framework without rewriting any code!

Inspired by both Behaviour Driven Development (BDD) and fluent test libraries like JSpec or RSpec, Kahlan is an exciting new test framework for PHP which allows tight, expressive tests. Kahlan allows you to stub or monkey patch your code directly like in Ruby or JavaScript, without needing any PECL extensions!

Kahlan on Github

To make testing faster, easier, and - let’s be honest - more enjoyable, we wanted to switch our Li3 (AKA Lithium) projects to use Kahlan rather than the native test utilities. Rewriting all our tests in a new framework would be crazy; no client is going to pay us to sit there doing that! We therefore decided to keep the existing tests as they were and simply write any new tests in Kahlan. Simple, right? Just one gotcha - how would we merge the code coverage statistics from both the Kahlan and Li3 tests? We needed to see our overall test coverage so that we could see where we needed to improve our testing. To that end, we needed to work out how to run our tests in one go and have the union of the coverage reported. Kahlan author Simon Jaillet got to work on implementing a wrapper!

Implementation

The first idea was to create a single spec in the Kahlan environment to encapsulate all existing Li3 legacy tests. The main drawback of this approach is that it would mix both the Li3 & Kahlan console reporting, which was not ideal. Thankfully, the Kahlan workflow is highly configurable and so we decided to run all of our Li3 legacy tests after the Kahlan specs had finished to keep console output clean and the two suites separate. To get coverage of our legacy tests, we needed only to enable the code coverage reporter again during Li3 test execution.

Here’s how we did it, step by step!

Step 1: Creating a custom config file

Kahlan uses a simple PHP file as config file, so you don’t need to learn any new funky syntax to configure the workflow the way you want. By default, Kahlan looks for a file called kahlan-config.php, so let’s create one:

touch kahlan-config.php

Step 2: Registering some namespaces

Kahlan works in tandem with Composer and uses its autoloading mechanism. Unfortunately, we were using a legacy installation of Li3 which uses a dedicated loader and some libraries that were not Composer compatible. So, the second step here was to dynamically add some namespaces and a class map to Composer so that all autoloading was done by Composer:

/**
 * Adding custom namespaces to Composer
 */
Filter::register(‘app.namespaces', function($chain) {
    // PSR-0
    $this->_autoloader->add('spec\\', __DIR__);
    $this->_autoloader->add('lithium\\', __DIR__ . '/libraries');

    // PSR-4
    $this->_autoloader->addPsr4(app\\', __DIR__);

    // PSR-0 Li3 Libraries
    $this->_autoloader->add('li3_access\\', __DIR__ . '/libraries');
    $this->_autoloader->add('li3_mailer\\', __DIR__ . '/libraries');
    // etc .

    // AWS class paths
    $awsPath = __DIR__ . '/libraries/li3_aws/libraries/aws-sdk-for-php';
    $this->_autoloader->addClassMap([
        'AmazonAS' => $awsPath . '/services/as.class.php',
        'AmazonCloudFormation' => $awsPath . '/services/cloudformation.class.php',
        'AmazonCloudFront' => $awsPath . '/services/cloudfront.class.php',
        'AmazonCloudSearch' => $awsPath . '/services/cloudsearch.class.php',
        // etc.
    ]);
});


Filter::apply($this, 'namespaces', app.namespaces');

Filter::register() & Filter::apply() are the two necessary methods to customize the spec execution workflow. The first one aims to attach an name (in this case, ’app.namespaces’) to a piece of logic (i.e the closure). Once you have registered your logic, you only need to attach it to a workflow entry point (in this case, ’namespaces’).

Note: Kahlan doesn’t use an event based approach to customize the execution workflow; rather it uses an AOP (Aspect Oriented Programming) approach. For more information on AOP, check out https://github.com/crysalead/filter.

Step 3: Configuring the coverage reporter to only scan a couple of directories

To generate accurate code coverage, we needed to exclude some parts of the tree from being parsed; for example, we don’t need coverage for the tests folder. So in the same way as we did for namespaces, we provided a filter to customize how the code coverage reporter should be initialized:

use kahlan\reporter\Coverage;
use kahlan\reporter\coverage\driver\Xdebug;

/**
 * Initializing a custom coverage reporter
 */
Filter::register('app.coverage', function($chain) {
    $reporters = $this->reporters();

    // Limit the Coverage analysis to only a couple of directories only
    $coverage = new Coverage([
            'verbosity' => $this->args()->get('coverage'),
            'driver' => new Xdebug(),
            'path' => [
                'controllers',
                'models',
               // etc..
            ]
    ]);
    $reporters->add('coverage', $coverage);

    return $reporters;
});

Filter::apply($this, 'coverage', 'app.coverage');

Step 4: Creating the logic to run Li3 legacy tests

Once the coverage reporter configured, we needed to reproduce how Li3 tests are executed inside the Li3 test layer. This specific task has been integrated in the $runLi3Suite closure and $runLegacyTests just adds some coverage logic around to encapsulate the whole execution of legacy tests to be able to retrieve some code coverage data for these tests:

use lithium\console\Request as ConsoleRequest;

$runLegacyTests = function($kahlan) {


    // If the coverage option is enabled, manually enable it for Li3 tests
    if ($this->args()->exists('coverage')) {
        $coverage = $this->reporters()->get('coverage');
        $coverage->before(); // Starts to log coverage
    }

    $runLi3Suite = function($path) {
        $test = new Test([
            'request' => new ConsoleRequest(['input' => fopen('php://temp', 'w+')])
        ]);
        return $test->run($path);
    };


    // Run the Li3 `’tests’` suite
    $success = $runLi3Suite('tests');

    if (!$success) {
        echo "Lithium tests failed\n";
        $kahlan->suite()->status(-1); // Force Kahlan to fail (i.e. exit(-1))
    }


    // Stop Legacy Code coverage
    if ($this->args()->exists('coverage')) {
        $coverage->after(); // Stops to log coverage
    }
    return $success;
};

Step 5: Creating the test execution workflow

This is the final step and actually the most obvious. Here we simply added a filter around the 'run' endpoint which runs all the logic together. In this filter we also added a couple of initializations related to the Li3 framework and declaring a constant which will be checked in the li3 bootstrap file to disable the Li3 built-in autoloader:

/**
 * Boostrapping & running the specs
 */
Filter::register('app.run', function($chain) use ($runLegacyTests) {
    // Used to bail out the Li3 autoloader initialization in test environment.
    // Indeed Composer will do all the job here.
    define('KAHLAN_ENVIRONMENT', 'true');

    // Lithium bootstrap file
    require __DIR__ . '/config/bootstrap.php';

    // Run Kahlan specs
    $result = $chain->next();

    // Run li3 legacy tests
    $runLegacyTests($this);

    return $result;
});

Filter::apply($this, run, app.run);

Step 6: Adding a --legacy option for making the transition easier for developers

During development, sometimes we want to run the legacy tests, and sometimes not. Thankfully, Kahlan makes adding custom CLI (Command Line Interface) arguments easy!

We added a new option, --legacy, to the command line. If this parameter is supplied, the li3 legacy tests are executed and if not they are simply ignored. We just needed to add at the top of the Kahlan config file:

// Adding a, custom option for legacy tests
$args = $this->args();
$args->argument('legacy', ['type'  => 'boolean']);

and then we simply added at the top of the $runLegacyTests closure:

$runLegacyTests = function($kahlan) {

    if (!$kahlan->args()->get('legacy')) {
        return true;
    }
    // ...

Results

This article is not so much about the wrapper to Li3 tests itself, but more to demonstrate that no matter how complex your test setup, Kahlan can be integrated into your workflow.

Switching to another test framework is not an harmless decision, but we started to use Kahlan a couple of months ago on one of our team’s projects and we already see huge benefits using it. We write specs a lot faster and easier and the created specs are a lot more reader friendly for the rest of the team. Furthermore, being able to focus the code coverage report right on a single class or method provide a new spec writing experience for better code coverage.

Even though Kahlan requires some extra cycles for “Just In Time” code patching, it actually ends up making our test suite faster overall for 2 main reasons. Firstly, because we were able to rewrite time consuming integration tests to simple unit tests. Secondly, due to the Kahlan code coverage heuristic, we can make builds with code coverage stats an order of magnitude faster.

Benefits

GREAT SPEEEEEEEED! Both in writing and running tests.
Coverage stats on every build - we saw 40 minutes come down to 2 minutes!
No wastage - we didn’t have to throw away or rewrite our old tests!
Easier test writing with Kahlan

Get involved

Kahlan is open source and MIT licensed! Feel free to get involved by using and contributing to this exciting PHP test framework!

There’s also no reason that Kahlan couldn’t be integrated with tools like Behat or PHPUnit - the sky’s the limit!

Have you come up against slow code coverage or test writing? What did you do about it? Let us know below!

Faster PHP code coverage