testing.md

Testing

Testing vcpkg is important whenever one makes changes to the tool itself, and writing new tests and keeping them up to date is also very important. If one's code is subtly broken, we'd rather find it out right away than a few weeks down the line when someone complains!

Running Tests

Before anything else, we should know whether you can actually run the tests! All you should need is a way to build vcpkg -- anything will do! All you have to do is follow the guide 😄

With $VCPKG_DIRECTORY being the directory where you have cloned vcpkg, create a build directory in $VCPKG_DIRECTORY/toolsrc (commonly named out), and cd into it. Make sure to clean it out if it already exists!

$ cmake .. -DCMAKE_BUILD_TYPE=Debug -G Ninja
$ cmake --build .
$ ./vcpkg-test # ./vcpkg-test [$SPECIFIC_TEST] for a specific set of tests
$ # i.e., ./vcpkg-test [arguments]

If you make any modifications to vcpkg, you'll have to do the cmake --build . step again.

Writing Tests

In your journey to write new tests, and to modify existing tests, reading the Catch2 documentation will be very helpful! Come back after reading those 😀

You'll want to place your tests in one of the existing files, or, if it doesn't belong in any of those, in a new file.

The layout of these tests is as follows:

// ... includes

TEST_CASE("Name of test", "[filename without the .cpp]") {
    // setup and the like
    REQUIRE(some boolean expression);
}

// etc.

You want to give these test cases good, descriptive, unique names, like SourceParagraph construct minimum -- it doesn't need to be extremely clear english, and shorthand is good, but make sure it's clear what the test is from the name. For the latter parameter, known as "tags", you should at least put the name of the file which the test case is in -- e.g., in arguments.cpp, you'd tag all of the test cases with [arguments].

If you wish to add helper functions, make sure to place them in an anonymous namespace -- this will ensure that they don't trample over anybody else's space. Additionally, there are a few helper functions that live in <vcpkg-test/util.h> and src/vcpkg-test/util.cpp -- make sure to look into them so that you're not rewriting functionality.

That should be all you need to know to start writing your own tests! Remember to check out the Catch2 documentation if you'd like to get more advanced with your tests, and good luck on your testing journey!

Adding New Test Files

Adding new test files should be easy and straightforward. All it requires is creating a new source file in toolsrc/src/vcpkg-test.

Example

Let's try writing a new test file called example (very creative, I know).

First, we should create a file, example.cpp, in toolsrc/src/vcpkg-test:

// vcpkg-test/example.cpp
#include <vcpkg-test/catch.h>

This is the minimum file needed for tests; let's rebuild!

$ cmake --build .
[80/80] Linking CXX executable vcpkg.exe

Okay, now let's make sure this worked; add a test case to example.cpp:

TEST_CASE("Example 1 - fail", "[example]") {
    REQUIRE(false);
}

Now build the tests again, and run them:

$ cmake --build .
[2/2] Linking CXX executable vcpkg-test.exe
$ ./vcpkg-test

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
vcpkg-test.exe is a Catch v2.9.1 host application.
Run with -? for options

-------------------------------------------------------------------------------
Example 1 - fail
-------------------------------------------------------------------------------
$VCPKG_DIRECTORY/toolsrc/src/vcpkg-test/example.cpp(3)
...............................................................................

$VCPKG_DIRECTORY/toolsrc/src/vcpkg-test/example.cpp(14): FAILED:
    REQUIRE( false )

===============================================================================
test cases:  102 |  101 passed | 1 failed
assertions: 3611 | 3610 passed | 1 failed

Hopefully, that worked! It should compile correctly, and have one failing test. Now let's try a more complex test, after deleting the old one;

// add #include <vcpkg/base/strings.h> to the top of the file
namespace Strings = vcpkg::Strings;

TEST_CASE("Example 2 - success", "[example]") {
    std::string hello = "Hello";
    REQUIRE(Strings::case_insensitive_ascii_equals(hello, "hELLo"));
    REQUIRE_FALSE(Strings::case_insensitive_ascii_starts_with(hello, "E"));
}

Now compile and build the tests, and this time let's only run our example tests:

$ cmake --build .
[2/2] Linking CXX executable vcpkg-test.exe
$ ./vcpkg-test [example]
Filters: [example]
===============================================================================
All tests passed (2 assertions in 1 test case)

Hopefully you have one test running and succeeding! If you have that, you have succeeded at adding a new file to vcpkg's tests. Congratulations! Have fun on the rest of your journey 🐱‍👤😁