This is a simple testing framework for C99 on Linux, intended to be used in an academic course. It uses the Check framework for unit testing, some custom Bash scripts for integration testing, and the Memcheck Valgrind-based tool for finding memory leaks.
The authors of this framework use it for a series of projects in our CS 261 course (example assignments). The projects include an introduction to C programming followed by the multi-part development of a machine code interpreter for the fictional Y86 language used in the CS:APP textbook. We have attempted to clean up and generalize the code in this repository, but you may still find references to our course in these files. If you have any questions, please contact the authors or open an issue.
You will need to install the Check framework (instructions) in order to use this framework. If you are running this on a departmental server you will probably need to contact your sysadmin to install the software. Fortunately, it is a standard package on most major Linux distributions.
Clone this repository in your home directory and begin working. If you wish to use our grading scripts you will need to set up submission folders in a particular way; see the appropriate section below for more details.
The project sources are in the ref folder. There are two provided projects:
-
pT-blank- A mostly-empty project with the barebones necessary for a new project. Copy this folder to start a new project from scratch. -
p0-intro- A minimal project with a few required routines and I/O specifications. Several unit and integration tests are provided along with a project description.
To build and test both of the reference solutions, run make test in the
ref folder.
To create a new project, simply copy the pT-blank folder in its entirety to a
new folder and begin working. You will probably wish to change the filenames
first, and with that you will need to change the includes and Makefiles
appropriately. You should be able to search for "pT-blank" to find all of the
necessary locations.
The build system is a simple Makefile with support for multiple modules as well as pre-compiled objects. The latter is useful when projects build on each other; you can distribute stripped object files so that students who were unable to fully complete the first project can still complete the second. Note that optimization is disabled by default to make debugging easier; this is not a framework requirement.
All of the testing code for each project is in the tests subfolder. The
framework is intended to be self-contained (with only the Check dependency) and
so there is some duplication between projects. See sections below for further
details about how the tests work and for instructions on adding new tests.
Unit tests are intended to test individual routines (and their dependencies),
and they are written in C using the Check
framework. Often the code for these tests tends to resemble the solution so it
is sometimes desirable to distribute them without the source. Thus, there are
three main unit testing files: public.c, private.c, and hidden.c. The
first is distributed as-is so that the students can see how the tests are
written, the second is distributed as a stripped object file so that students
can run the tests and see the results but they do not have access to the source
code, and the third is not distributed at all, allowing for tests students
neither have access to nor know about.
Tests are delimited using the START_TEST and END_TEST macros provided by
Check. Testing expressions are written using Check assertion routines
(documentation).
Here is an example from the template project:
START_TEST (C_addabs_simple)
{
ck_assert_int_eq (add_abs(2,3), 5);
}
END_TESTTo add new tests, create a new routine in the appropriate file delimited by the
appropriate macros and give it a unique name (provided as a parameter to the
START_TEST macro). You will also need to add a tcase_add_test line at the
bottom of the file for your new test.
Aside: we use a test naming scheme where tests are labeled with a prefix
containing the grade corresponding to the test; i.e., students must pass all
D_* tests to receive a "D" grade, all of those plus all C_* tests to receive
a "C" grade, etc. This is not a requirement of the testing framework itself, but
the grading script does use it to automatically assign baseline grades based on
test results.
There is also a testsuite.c driver, but it is very generic and does not
generally need to be modified.
Integration tests are intended to test whole-program behavior and are written
using input/output file pairs. The inputs are stored in the inputs folder and
the expected outputs are stored in the expected folder. Tests are specified in
the itests.include file, which contains one line for each integration test
that specifies the name of the test (the output file must match this) as well as
the command line intended to evoke the corresponding output. The actual testing
is handled by the integration.sh Bash script, which should not need to be
modified when creating a new project except to change the executable name at the
top.
To add a new integration test: 1) put any necessary input (not all integration
tests require an input file; e.g., if all parameters are provided on the command
line) in the inputs folder, 2) put the required output in a text file in the
expected folder, and 3) add a corresponding line to itests.include.
Note that there is no "code" for integration tests and thus there are no
private integration tests; however, there is support for hidden integration
tests that are not distributed to students. Hidden integration tests can be
placed in the appropriate section in itests.include and the test name must
end with _H.
Because memory management is such an important part of coding in a low-level
language, the script also runs each test in Valgrind/Memcheck to check for
memory leaks. If you do not have Valgrind installed or wish to skip these
tests, you can comment out the offending calls in integration.sh.
Scripts are provided in the dist folder for building project distributions.
These scripts remove solution code and package up the project into a tarball.
For convenient testing, it also saves a snapshot of both the distribution files
and the full solution.
To mark solution code for removal, surround it with BEGIN_SOLUTION and
END_SOLUTION one-line comments. If the removal of the code would result in
invalid code (e.g., no return value), you can use a BOILERPLATE tag to provide
alternative code for the distribution. Here is an example from the template
project:
int add (int num1, int num2)
{
// BEGIN_SOLUTION
return num1 + num2;
// END_SOLUTION
// BOILERPLATE: return 0;
}In the distribution, this will be converted to:
int add (int num1, int num2)
{
return 0;
}To create distribution files for a new project, copy rebuild-pT.sh in the
dist folder and make any appropriate changes (file names, new files, etc.).
Make sure you use the provided "cleanup" function to copy any files that
contain source code. If you are providing prebuilt object files containing
solutions to a previous project, make sure you strip -S them after copying
them into the distribution folder.
We recommend double-checking the distribution files every time you rebuild them to ensure you are not accidentally distributing solution code because of some framework misconfiguration.
A sample grading script has been provided in /grading/grade.sh. To use this
folder, you will need to collect student submissions using a specific file
structure. There must be a root submissions folder with subfolders for each
student (we use the students' login IDs as subfolder names). Each student
subfolder should then have a subfolder for each project containing the actual
submission files. Each of these subsubfolders must have a unique project
identifier (e.g., 'p0', 'p1', etc.).
You must also create a project.include file (in the current folder wherever
you wish to run the tests, but we recommend it NOT be in the submissio folder)
containing filename and path information specific to your project. See the
template file or p0 file
for examples of what this file should look like. You will need to change all of
the files and paths to match your setup:
TAGis the unique project identifier.EXEis the name of the compiled binary.FILESis a list of submission files to be copied into the results folder.REFFILESis a list of reference testing files to be copied into the results folder (this is to prevent the student from altering the test files).REFis the path to the project reference files.SUBMITis the path to the root student submission folder, which should contain student subfolders and project subsubfolders as described above.RESULTSis the path to a folder where all of the results can be copied.
When the script is run, it copies all of the student submissions along with
testing files into subfolders in RESULTS. It then builds and runs all tests on
all submissions, printing a summary of the results to standard output.
The framework has been designed to be platform-independent and has been tested on a variety of Linux platforms as well as macOS High Sierra and Windows 10 (using Cygwin). In addition, we have provided Docker files so that you can run this framework on non-supported platforms. See the README.md for more info.
If you wish to contribute, please first discuss the change you wish to make via issue, email, or any other method with the owners of this repository. We may decide to incorporate your contributions in our project or we may suggest that you maintain them as a fork.
The primary developer (and current maintainer) of this project is Mike Lam, and the code was based on an older framework written by Michael Kirkpatrick. Both authors are faculty at James Madison University.
This project is licensed under the MIT License - see the LICENSE.txt file for details.