Running Tests

General

Tests are a very important part for the development process for MaidSafe. In an attempt to encourage co-development with our peers we have developed many tests. These tests allow developers to attempt fixes, optimisations and additions with some degree of comfort. If you work on code and the tests are still passing, it is usually a good sign. There is nothing that will grab a code reviewer's eye quicker than tests that have been altered. Such alterations should be well documented.

MaidSafe-supplied tests are prefixed with either TEST (indicating that it uses the Google Test framework) or test_ (indicating the Catch Test framework). A major limitation of the Catch framework at the moment is that its assertions are not threadsafe, so if you plan to write new tests and these need to be tested in a multi-threaded way, prefer GTest.

To check options on any of these tests, use -h or --help as the command line switch (e.g. ./TESTcommon -h). The initial portion of the help output relates to logging arguments documented elsewhere in the wiki.

In addition there are some third party tests (not provided by MaidSafe) which are also included in the test suite. These currently include tests for Crypto++ and leveldb.

Running Tests in a Specific Submodule/Subproject

Each submodule has an Exper<submodule name> target (e.g. ExperCommon). Building an Exper target causes all the targets in that submodule to be built, but also uses CTest to invoke the test executables (i.e. runs the test suite for that submodule) and publishes the results to the MaidSafe dashboard. So, for example, to run the Experimental tests for the Common submodule, you can do:

make ExperCommon

or on Windows:

msbuild /P:Configuration=Debug src\Common\ExperCommon.vcxproj

Running the Full Test Suite

We also provide CTest scripts to allow for simple running of Continuous, Nightly or Weekly tests. They execute the full suite of tests for all submodules including third-party tests. These should not be run from a repository which you use to develop the MaidSafe code, since they may revert commits or checkout to a different branch from your current working one!

The Continuous tests will only execute the tests if an update is pulled from the remote repository, so if no code is committed and pushed, these are effectively no-ops.

The Nightly tests revert the code to the commits in place at midnight GMT, then executes all the tests.

The Weekly tests are similar to Nightlies except for including some exceptionally long-running tests which are normally excluded from test runs.

The scripts are produced when you first run CMake and appear in the root build folder. Their names should be self-explanatory, and to invoke them simply use the -S command line arg with CTest while in your build root; e.g:

ctest -S CI_Nightly_Debug.cmake

By default, all of these scripts checkout all projects to the next branch. This is generally the most bleeding-edge branch which should usually compile on all supported platforms. To use a different branch, you can set the variable BRANCH using the command line -D arg, e.g:

ctest -S CI_Continuous_Release.cmake -DBRANCH=master

Running the Suite Continuously

The Continuous, Nightly and Weekly tests are designed to be run continuously. This can be achieved by setting up a cron job on Linux or OS X, or adding a scheduled task on Windows:

Example cron Job

Create a script called /home/Me/NightlyTesting.sh containing:

#!/bin/bash
cd /home/Me/build_maidsafe
ctest -S CI_Nightly_Debug.cmake

Add the following cron entry:

#Nightly CI Tests
30 1 * * 0-6 /home/Me/NightlyTesting.sh

Example Scheduled Task

Create a batch script called D:\Tests\NightlyTesting.bat containing:

@echo off
%comspec% /C "cd "D:\Tests\build_maidsafe" && D: && "%VS120COMNTOOLS%VsDevCmd.bat" && ctest -S CI_Nightly_Debug.cmake"

Then add the scheduled task by doing:

schtasks /Create /SC DAILY /TN MaidSafeNightly /TR "D:\Tests\NightlyTesting.bat" /ST 01:05