Standalone unit- and functional tests for ExtBase extensions in TYPO3 6.2

The TYPO3 core team did a lot of work in the last weeks to simplify unit- and functional testing in TYPO3, so now everything about testing TYPO3 core and TYPO3 extensions has become much more clear and straightlined. In this article I will describe how I switched the unit- and functional tests in one of my TYPO3 extensions from cli_dispatch to standalone phpunit tests.

Initial situation

I am working an an event management and registration extension for TYPO3. Since I always work with unit- and functional tests and a CI environment, I've setup my IDE PHPStorm to execute unit- and functional tests as described in my old blogpost and I also integrated the extension in Travis CI and Scrutinizer CI. Since the extension only should be compatible with TYPO3 6.2+, I already used the new functional tests introduced in TYPO3 6.2 instead of the functional testing framework which comes with the TYPO3 extension phpunit.

Standalone unit tests for PHPStorm

Helmut Hummel wrote an excellent article about how to execute TYPO3 Unit tests in PHPStorm. I've setup my unit tests as descibed in Helmut's article, but with one difference. Instead of configuring the unit test bootstrap file as described in step 5, I created a PHPUnit configuration file, which holds all PHPUnit configuration and which later on also will be used on Travis CI for test execution.

Unit tests use an alternative configuration file

The configuration file for PHPUnit is like shown below.

<phpunit
        backupGlobals="false"
        backupStaticAttributes="false"
        bootstrap="../../../../../typo3/sysext/core/Build/UnitTestsBootstrap.php"
        colors="true"
        convertErrorsToExceptions="true"
        convertWarningsToExceptions="true"
        forceCoversAnnotation="false"
        processIsolation="false"
        stopOnError="false"
        stopOnFailure="false"
        stopOnIncomplete="false"
        stopOnSkipped="false"
        verbose="false">

    <testsuites>
        <testsuite name="EXT:sf_event mgt">
            <directory>../Unit/</directory>
        </testsuite>
    </testsuites>
</phpunit>

After I had setup everything, I executed my unit tests, which actually did not run. With my old setup and within the TYPO3 extension PHPUnit, the tests executed successfully. After some hours of debugging, I fixed the failing tests. I had to do more mocking inside my tests, so they really could run independent from a working TYPO3 installation.

Standalone functional tests for PHPStorm

The configuration for functional tests in PHPStorm is similar to the unit test setup with some small differences. You must switch on PHPUnit process isolation and you must configure database settings, if you do not run the tests in an working TYPO3 CMS installation. Documentation about functional tests in TYPO3 can be found in the TYPO3 wiki about functional testing.

Since I want to run the functional tests without a working TYPO3 CMS installation and also on Travis CI, I configured a PHPUnit configuration file and added environment variables for database access to the PHPStorm Run/Debug configuration.

Environment variables for functional tests

The PHPUnit configuration file is like shown below:

<phpunit
        backupGlobals="false"
        backupStaticAttributes="false"
        bootstrap="../../../../../typo3/sysext/core/Build/FunctionalTestsBootstrap.php"
        colors="true"
        convertErrorsToExceptions="true"
        convertWarningsToExceptions="true"
        forceCoversAnnotation="false"
        processIsolation="true"
        stopOnError="false"
        stopOnFailure="false"
        stopOnIncomplete="false"
        stopOnSkipped="false"
        verbose="false">

    <testsuites>
        <testsuite name="EXT:sf_event mgt">
            <directory>../Functional/</directory>
        </testsuite>
    </testsuites>
</phpunit>

Note, that I've set processIsolation to true.

After I had setup the functional test configuration, my functional tests also did not work. Again this was caused incomplete mocking and also by direct usage of the ExtBase objectManager in my tests. After I completed the mocking and removed all direct usages of the objectmanager in the tests, the functional tests were executed successfully in PHPStorm.

Unit and functional tests on Travis CI and code analysis on Scrutinizer CI

My TYPO3 extension was already integrated in Travis CI and Scrutinizer CI. Since I switched from test execution through cli_dispatch to standalone unit test execution, I had to modify my Travis CI configuration to execute tests directly through phpunit.

Both unit- and functional tests can be executed on commandline with the following commands:

phpunit -c typo3conf/ext/sf_event_mgt/Build/UnitTests.xml
phpunit -c typo3conf/ext/sf_event_mgt/Build/FunctionalTests.xml

Please note, that the test execution as shown above only works if the commands are executed from within a directory with a full TYPO3 CMS directory structure. In case of the functional tests, you must also set environment variables for Database access.

To get the test execution working on Jenkins CI, you just have to clone the TYPO3 core and create the TYPO3 directory structure (fileadmin, uploads and typo3conf/ext) and finally move your extension to typo3conf/ext/. After that, you're ready to execute both unit- and functional tests on Jenkins CI.

My final .travis.yml file is like shown below:

language: php
php:
  - 5.3
  - 5.4

env:
  - DB=mysql TYPO3_BRANCH=master COVERAGE=0

matrix:
  include:
    - php: 5.5
      env: DB=mysql TYPO3_BRANCH=master COVERAGE=1

notifications:
  email:
    - [email protected]

before_script:
  - cd ..
  - git clone --single-branch --branch $TYPO3_BRANCH --depth 1 https://github.com/TYPO3/TYPO3.CMS.git typo3_core
  - mv typo3_core/* .
  - sudo apt-get install parallel
  - composer self-update
  - composer install
  - mkdir -p uploads typo3temp typo3conf/ext
  - mv sf_event_mgt typo3conf/ext/

script:
  - >
    if [[ "$COVERAGE" == "0" ]]; then
      echo;
      echo "Running unit tests";
      ./bin/phpunit --colors -c typo3conf/ext/sf_event_mgt/Tests/Build/UnitTests.xml
    fi
  - >
    if [[ "$COVERAGE" == "1" ]]; then
      echo;
      echo "Running unit tests";
      ./bin/phpunit --coverage-clover=coverage.clover --colors -c typo3conf/ext/sf_event_mgt/Tests/Build/UnitTests.xml
    fi
  - >
    if [[ "$COVERAGE" == "1" ]]; then
      echo;
      export typo3DatabaseName="typo3";
      export typo3DatabaseHost="localhost";
      export typo3DatabaseUsername="root";
      export typo3DatabasePassword="";
      find . -wholename '*/typo3conf/ext/sf_event_mgt/Tests/Functional/*Test.php' | parallel --gnu 'echo; echo "Running functional test suite {}"; ./bin/phpunit --colors -c typo3conf/ext/sf_event_mgt/Tests/Build/FunctionalTests.xml {}'
    fi
  - >
    if [[ "$COVERAGE" == "1" ]]; then
      echo;
      echo "Uploading code coverage results";
      wget https://scrutinizer-ci.com/ocular.phar
      cp -R typo3conf/ext/sf_event_mgt/.git .
      php ocular.phar code-coverage:upload --format=php-clover coverage.clover
    fi

In order to get functional tests to execute successfully, you must provide TYPO3 database credentials and the TYPO3 database name as shown in the config above.

The Travis CI configuration file already includes code coverage analysis through Scrutinizer CI. In the last part of the configuration, the code coverage results are uploaded to Scrutinizer.

Merging unit- and functional test code coverage data

[EDIT 28.06.2014] It seems I have missed one thing regarding code coverage. With cli_dispatch test execution, both unit- and functional tests run at the same time resulting in one file with code coverage data (if code coverage is switched on). With standalone unit- and functional tests, tests are seperated in at least 2 runs. The config for Travis CI I posted above just handles code coverage for unit tests and not for funtional tests. Also, the Travis CI config runs functional tests in parallel to speed up functional test execution. To include code coverage data from both unit- and functional tests, I changed to Travis CI config as following for the functional test execution.

find . -wholename '*/typo3conf/ext/sf_event_mgt/Tests/Functional/*Test.php' | parallel --gnu 'echo; echo "Running functional test suite {}"; ./bin/phpunit --coverage-clover={}functionaltest-coverage.clover --colors -c typo3conf/ext/sf_event_mgt/Tests/Build/FunctionalTests.xml {}'

With this change, phpunit creates one file with code coverage data for each functional test suite. Keep in mind, that this will really slow down functional test execution.

And for the upload of code coverage data to Scrutinizer CI I changed the following:

find . -wholename '*/typo3conf/ext/sf_event_mgt/Tests/Functional/*Test.php' -exec php ocular.phar code-coverage:upload --format=php-clover {}functionaltest-coverage.clover \;

This uploads each file with code coverage data for functional tests to scrutinizer CI. The Scrutinizer CI configuration must now be changed as shown below.

external_code_coverage:
    timeout: 700
    runs: 3

With this change, Scrutinizer CI waits for 3 code coverage data uploads and merges them, so finally the code analysis end up with code coverage data from all unit- and functional test. There is one downside with this settings. The number of runs must be changed manually depending on the amount of test suites with code coverage data you have in your project. But finally, Scrutinizer CI shows the same code coverage as PHPStorm.

Code coverage in PHPStorm and Scrutinizer CI both show 92%

Conclusion

The switch from cli_dispatch to standalone test execution for ExtBase extensions is'nt really complicated. Test setup for unit- and functional tests has become much more easy and clear now and I think (and hope) that standalone unit- and functional tests will reduce the barriers for TYPO3 extension developers to actually create TYPO3 Extension with good test coverage.

I would like to thank all people in the TYPO3 community who made standalone tests possible. I really appreciate the work that has been done.