Running Tests

Tests are a very important part for the build process for MaidSafe, just as they should be for any project. In an attempt to encourage co-development with our peers we have developed many tests. These tests allow developers to attempt fixes, optimisations and additions with some degree of comfort. If you work on code and the tests are still passing, it is usually a good sign. There is nothing that will grab a code reviewer's eye quicker than tests that have been altered. Such alterations should be well documented.

MaidSafe-supplied tests are prefixed with TEST. These are currently:

... TESTcommon
... TESTclient_manager
... TESTprivate
... TESTrudp
... TESTrouting
... TESTrouting_api
... TESTrouting_big
... TESTrouting_func
... TESTrouting_func_nat
... TESTencrypt
... TESTpassport
... TESTnfs
... TESTvault
... TESTdrive_listings_api
... TESTdrive_real_filesystem
... TESTdrive_virtual_filesystem
... TESTclient_user_input
... TESTclient_user_storage

Note: Some of these programs can run many hundreds of tests and some may run for several hours or even days, depending on values chosen.

Adding the appropriate compiler switch (-fsanitize=address|thread|undefined) to cmake allows gcc > 4.8 and clang to use the sanitizer options. This is done by running a test as ./TESTsomething 2>&1 | ../tools/clang/asan_symbolize.py . This asan_symbolize.py will ensure you get a clean output. NB: suggest at least gcc 4.8.1 is used as 4.8 has some issues with exceptions and sanitizer.

The tests are built using Google Test. To check options on any of these tests, use -h or --help as the command line switch (e.g. ./TESTcommon -h). The initial portion of the help output relates to logging arguments documented elsewhere in the wiki and is skipped here. The remainder of the help output is as follows:

This program contains tests written using Google Test. You can use the
following command line flags to control its behavior:

Test Selection:
  --gtest_list_tests
      List the names of all tests instead of running them. The name of
      TEST(Foo, Bar) is "Foo.Bar".
  --gtest_filter=POSTIVE_PATTERNS[-NEGATIVE_PATTERNS]
      Run only the tests whose name matches one of the positive patterns but
      none of the negative patterns. '?' matches any single character; '*'
      matches any substring; ':' separates two patterns.
  --gtest_also_run_disabled_tests
      Run all disabled tests too.

Test Execution:
  --gtest_repeat=[COUNT]
      Run the tests repeatedly; use a negative count to repeat forever.
  --gtest_shuffle
      Randomize tests' orders on every iteration.
  --gtest_random_seed=[NUMBER]
      Random number seed to use for shuffling test orders (between 1 and
      99999, or 0 to use a seed based on the current time).

Test Output:
  --gtest_color=(yes|no|auto)
      Enable/disable colored output. The default is auto.
  --gtest_print_time=0
      Don't print the elapsed time of each test.
  --gtest_output=xml[:DIRECTORY_PATH/|:FILE_PATH]
      Generate an XML report in the given directory or with the given file
      name. FILE_PATH defaults to test_details.xml.
  --gtest_stream_result_to=HOST:PORT
      Stream test results to the given server.

Assertion Behavior:
  --gtest_death_test_style=(fast|threadsafe)
      Set the default death test style.
  --gtest_break_on_failure
      Turn assertion failures into debugger break-points.
  --gtest_throw_on_failure
      Turn assertion failures into C++ exceptions.
  --gtest_catch_exceptions=0
      Do not report exceptions as test failures. Instead, allow them
      to crash the program or throw a pop-up (on Windows).

Except for --gtest_list_tests, you can alternatively set the corresponding
environment variable of a flag (all letters in upper-case). For example, to
disable colored text output, you can either specify --gtest_color=no or set
the GTEST_COLOR environment variable to no.

For more information, please read the Google Test documentation at
http://code.google.com/p/googletest/. If you find a bug in Google Test
(not one in your own code or tests), please report it to
<[email protected]>.

In addition there are some built third party tests that can be run manually. These tests are:

// crypto suite tests
... cryptest
// following are level-db tests
... arena_test
... cache_test
... coding_test
... corruption_test
... crc32c_test
... db_test
... dbformat_test
... env_test
... leveldb-testharness
... log_test
... skiplist_test
... table_test
... version_edit_test
... version_set_test
... write_batch_test

There are some special tests that run full test suites and publish to the MaidSafe dashboard. These are invoked simply by building the target and are powered by CTest:

... ExperCommon
... ExperPrivate
... ExperRudp
... ExperRouting
... ExperEncrypt
... ExperPassport
... ExperNfs
... ExperVault
... ExperDrive
... ExperClient