CDRIVER-4497 CTest Test Fixtures for MongoDB Servers

[vector-of-bool]

Refer: CXX-2303 (this work can be applied to both the C and C++ driver projects, if desired).

This is a changeset that has been a long time in the works (first experimenting in November of 2021). It has been used and developed along with several other changes over the time frame, so the commit log will contain WIP-commits and merges of unrelated tasks from months prior. In short: The commit log is very not helpful here.

Instead, here's an ordered outline of the change and how one might go about understanding them:

1. Refer: CTest test fixtures. CTest gained the ability to run and order tests that represent the setup and cleanup of arbitrary test fixtures. A test may declare a FIXTURES_REQUIRED on one or more fixtures. A test may also declare FIXTURES_SETUP to inform CTest about the fixtures that it will "setup." Finally, a test may declare a FIXTURES_CLEANUP to tell CTest about which fixtures it will "clean up".

   For any fixture F, CTest will ensure that the SETUP "tests" of F are executed before every test that REQUIRES it. After all of its dependents are finished, CTest will execute the CLEANUP "tests" for F. Fixtures are scoped to the entire CTest run: The fixture will be set up once before its dependents run, and it will be torn down once after its dependents have finished. Fixture setup/cleanup "tests" can also depend on other fixtures. This creates a DAG of fixtures.

2. Refer: build/proc_ctl.py - This is a zero-deps Python script that implements very rudimentary long-running process control. This was added in IMDS Mock Server Testing #1108 and further updated in this PR. A child process is associated with a "control directory" where proc_ctl will write files to communicate with itself through time. When we "start" with proc_ctl, it will execute a child Python process that acts as a monitor. This monitor then spawns the actual child process. The monitor is left running while the top-level proc_ctl finishes. When the monitor process detects the child's exit, it will write information about that exit into the control directory. When we "stop" with proc_ctl, we send a SIGINT (or equivalent) to the child process. If it is well-behaved, it will shutdown. We spin until we see the child process exit.

3. Refer: mdb-ctl.py - This is a zero-deps Python script that wraps proc_ctl and provides utility commands to control a MongoDB server process. This is similar to mlaunch, but zero-deps and more tooling-friendly. This is the script that is invoked by CTest fixtures to start/stop/setup servers. This script expects to be given a filepath to a mongod executable: It does not run mongod from the PATH. In the future, if we can reliably obtain mlaunch, it may be better to prefer it as a more well-tested tool.

4. Refer: TestSuite and test .c files - A new parameter has been added to tests: "Meta". This is simply a string that will later be used by LoadTests.cmake to instrument tests with fixtures and labels. All tests that use a live MongoDB server (not a mock) have the USES_LIVE_SERVER meta-argument. The LABELS meta-argument is also supported, and generates labels on the associated tests. Tests can be selected/excluded based on their label in a CTest run with -L and -LE.

   For example, to run all tests that do not require spinning up a live server, pass -LE uses-live-server to the CTest command.

   Since "meta" gives us introspection into which tests use external resources, test parallelism is now possible. Passing -j <Number> to CTest will run <Number> tests in parallel, with a mutex on each of the live server fixtures (This "mutex" is created using the RESOURCE_LOCK CMake test property). Each server fixture has its own "mutex", so multiple server fixtures can be run simultaneously and their dependent tests run in parallel without contending.

5. Refer: build/cmake/QuickFixtures.cmake - This is a small CMake convenience module that allows concise definitions of CTest fixtures.

6. Refer: build/cmake/FindMongoDB.cmake - This CMake module scans for mongod executables on PATH and installed using m. For each found executable, it detects the version and sets CMake properties about them for later use.

   This module also defines three functions that are the main purpose of this PR:

   • mongodb_create_fixture() - Create a test fixture that runs a mongod server.
   • mongodb_create_replset_fixture() - Create a test fixture that spawns a replica set.
   • mongodb_create_sharded_fixture() - Create a test fixture that generates a very simple sharded cluster.

   This function records data about any created fixtures in a file MongoDB-CTestData.cmake, which is a script that will be loaded by CTest as part of test discovery. A particular fixture may be declared as "DEFAULT", which will be used later by LoadTests.cmake. TCP ports are allocated automatically for all servers that need to be spawned.

7. Refer: build/cmake/LoadTests.cmake - This file is included by CTest to discover test cases. It uses the "meta" data on each test to decide how to register each test.

   For every test T marked with USES_LIVE_SERVER, and for each MongoDB fixture F that was marked with DEFAULT, define a test T@F. This test will be marked as a dependent of F. The URL to the server associated with F will be injected into the test environment variables for T@F, allowing the test to communicate with the server fixture that it depends on.

8. Test execution: When CTest runs, if any test that requires a live server is selected (by default all tests are selected), the associated fixture will be marked for execution. Before running any test that requires the server, CTest will run the "setup" command for that server, which wipes the scratch data directory of the server and spawns a new server process. After all tests that require a particular fixture are finished, the server process will be stopped.

   For replicaset fixtures, multiple servers may be spawned. The first server in the replicaset will be forced to be primary. The CTest "setup" of a replicaset will not "finish" until the server reports that it has been elected as the primary. This can take a few seconds. This is required by tests that need to performs writes, as the port of the first server will be given as the port of the server fixture for the test. Currently, there is no functionality for tests to depend on secondaries, but this would not be difficult to add.

   For sharded cluster fixtures, config and data replica sets will be created, and an instance of mongos will be run against those two replicasets. The TCP port of mongos will be used as the TCP port given to the test.

9. Defining fixtures: By default, no server fixtures are defined. The file build/cmake/TweakMeServerFixtures.cmake is intended to define the fixtures for the current configuration. This file can be filled out manually or written as part of CI to define the test set. If no default server fixtures are defined, the tests that require them will be generated with placeholders marked as DISABLED. Note: LoadTests defines a lot of tests, and having many fixtures will generate a huge number of tests. While this is good in general, this will impact the startup time of CTest, which will need to generate thousands of test cases.