Learning goals

You will learn the following concepts/skills:

1. unit testing – systematically test your code
   1. framework – create a simple framework for unit-testing future C code
   2. organization – learn how to organize a test suite
   3. test coverage – measure how thoroughly your tests exercise your code
2. C preprocessor − practice using #define symbols and macros with #ifdef.

Overview of the Miniunit series (HW06, HW07, HW08)

This assignment is part 2 of a 3-part series collectively referred to as miniunit.

Real-world software projects typically comprise tens, hundreds, or many thousands of files. Even in your ECE courses, your programming assignments will be getting bigger and more complex. Testing code by hand (i.e., by playing with it) is useless since there are so many components, and ways things can go wrong. Hand-testing effectively would be complex—and prohibitively slow.

Most serious software projects use two tools: a unit testing framework (e.g., CUnit) to organize and coordinate tests, and a build system (e.g., make) to manage the process of compiling, testing, and deploying the project. In this homework, you will get a light introduction to both. In addition, you will learn to use console logging effectively (as a complement to gdb), without littering your code with printf(…) statements.

In the Miniunit Series, you will create the following:

• clog.h – your own library for smarter printf-style debugging and logging. From now on, instead of debugging with printf(…) directly, you will use function-like macros to print values in various formats and colors to make the output easier to view. You will use preprocessor directives to ensure that your debugging code does not interfere with your tests, or show up inadvertently when others are testing your code.
• Makefile – input file for the make build system. You will be able to build, test, submit, and/or pre-test your code with one command from bash or directly from Vim (and other editors).
• miniunit.h – your own simple unit test library. You can use this to test future assignments in this class, or for any other project you do in C (or C++) beyond ECE 264. This consists of four #define macros that you can use in your test code. The most important macro you will create is mu_check(…), which is somewhat similar to assert(…).

About miniunit.h

Until now, we have used a very low-tech method of structuring our tests with the expected.txt and the diff command. That method is based on the more general principle of unit testing.

In unit testing, programmers write collections of small functions that test some aspect of a program. For example, in mintf(…) in HW05, you might have a unit test for small positive integers in base 10, another unit for small negative numbers, one for extreme bases, extreme positive values, extreme negative values, and so on. Each test is its own function.

As the number and complexity of tests grows, it becomes necessary to have a foundation of code for running and organizing them. This is called a unit test framework. There are unit test frameworks for every major programming language. For C, one example is CUnit. These typically have special-purpose macros for checking assumptions, similarly to the assert(…) macro, which we have already covered this semester, but with a unit test framework, we may allow the code to continue, even if an assumption is not met, since the goal is simply to check. In addition to the checking macros, real-world testing frameworks have mechanisms for setting up the needed environment that the program-in-test depends on (e.g., files, network connections, etc.), and sophisticated interfaces for visualizing the results and scheduling test runs.

In this homework, you will create a very simple unit test framework… let's call it a unit test library. You may use this to test future assignments. We will illustrate by example.

Using miniunit.h to test — minimal example

Suppose you have the following (very trivial) module for doing arithmetic calculations involving the number 3.

three.c
 1 #include "three.h"
 2
 3 int times_3(int n) {
 4     return n * 3;
 5 }
 6
 7 int divided_by_3(int n) {
 8     return n % 3;  // BUG
 9 }
10
11 int plus_3(int n) {
12     return n * 3;  // BUG
13 }
14
15 int minus_3(int n) {
16     return n - 3;
17 }

three.h
 1 int times_3(int n);
 2 int divided_by_3(int n);
 3 int plus_3(int n);
 4 int minus_3(int n);

Here is the general format for every unit test function (e.g., in test_three.c). You will typically have several of these in your test_▒▒▒.c.

int test_▒▒▒() {
    mu_start();  // set up the test
    // …
    mu_check(condition);
    // …
    mu_check(condition);
    // …
    mu_end();    // finish the test
}

Then, to call all of your unit test functions, you will have a runner function (e.g., main(…)) like this:

int main(int argc, char* argv[]) {
    mu_run(test_▒▒▒);
    mu_run(test_▒▒▒);
    mu_run(test_▒▒▒);
    mu_run(test_▒▒▒);
    return EXIT_SUCCESS;
}

Using the above format, we create the following unit test suite (group of tests) for our three-arithmetic module.

// test_three.c

#include <stdio.h>
#include <stdlib.h>
#include "miniunit.h"
#include "three.h"

int test_plus_3() {
    mu_start();
    mu_check(plus_3(1) == 4);  // will FAIL
    mu_end();
}

int test_minus_3() {
    mu_start();
    mu_check(minus_3(1) == -2);
    mu_end();
}

int test_times_3() {
    mu_start();
    mu_check(times_3(1) == 3);
    mu_end();
}

int test_divided_by_3() {
    mu_start();
    mu_check(divided_by_3(1) == 0);  // will FAIL
    mu_end();
}

int main(int argc, char* argv[]) {
    mu_run(test_plus_3);        // will FAIL
    mu_run(test_minus_3);
    mu_run(test_times_3);
    mu_run(test_divided_by_3);  // will FAIL
    return EXIT_SUCCESS;
}

Your unit test library, using #define macros will convert that seemingly simple test code into a colored summary, including the name of the test function and the line where it failed. Here is the expected out put for the test above:

$ gcc -o test_three three.c test_three.c

$ ./test_three
Test failed: test_plus_3 at line 8
Test passed: test_minus_3
Test passed: test_times_3
Test failed: test_divided_by_3 at line 26

$

1. Set up your homework directory.

There is no starter code, though you will use your code from HW06.

Create a directory for HW07.

Copy your HW06 code into your HW07 directory.

The most important file you will create in HW07 is miniunit.h. You will create that from scratch (i.e., starting with a blank file).

2. Create your miniunit.h.

Create a file called miniunit.h from scratch.

Create the five #define macros as specified in the Requirements table.

⚠ Success and failure messages should be printed using logf_green(…) and logf_red(…). ⚠ Success and failure messages should be printed by mu_run(…) (not mu_check(…)), and should not rely on logf_green(…) and logf_red(…). Those will be disabled when DEBUG is not defined (i.e., when -DDEBUG was not passed to the compiler); miniunit.h is expected to work either way.

3. Use miniunit.h to improve test_count_words.c

This part will teach you how to use miniunit.h in practice for future assignments. We are giving you the code for free, so it shouldn't take more than about 15 minutes to type and test.

1. At the top of test_count_words.c, include miniunit.h. (You can also remove the #include "clog.h" since your miniunit.h will include that.)
2. Add three test functions above main(…): int _test_empty(), int _test_simple(), and int _test_hard().
3. Inside each test function, add mu_start() at the beginning, and mu_end(…) just before the end.
4. Add at least one call to mu_check(…) in between mu_start(…) and mu_end(…).
5. At the top of main(…), add mu_run(_test_empty), mu_run(_test_simple), and mu_run(_test_hard).
6. The result may look something like this:

   #include <stdio.h>
   #include <stdlib.h>
   #include "count_words.h"
   #include "miniunit.h"
   
   int _test_empty() {
     mu_start();
     //-------------------------------
     mu_check(count_words("") == 0);
     //-------------------------------
     mu_end();
   }
   
   int _test_simple() {
     mu_start();
     //-------------------------------
     mu_check(count_words("apple") == 1);
     mu_check(count_words("boring boxes") == 2);
     mu_check(count_words("apple banana") == 2);
     mu_check(count_words("apple banana cherry") == 3);
     //-------------------------------
     mu_end();
   }
   
   int _test_hard() {
     mu_start();
     //-------------------------------
     mu_check(count_words("famigerate fiddle-faddle") == 2);
     mu_check(count_words("Mary's mongoose") == 2);
     mu_check(count_words("plumbers' platitudes") == 2);
     //-------------------------------
     mu_end();
   }
   
   int main(int argc, char* argv[]) {
     mu_run(_test_empty);
     mu_run(_test_simple);
     mu_run(_test_hard);
   
     log_int(count_words("My apples are sweet."));
     log_int(count_words("My friend's apples are sweeter."));
     log_int(count_words("A pear is sweeter than an apple.."));
   
     return EXIT_SUCCESS;
   }
   

   Note: Due to the intentially planted bugs in count_words.c, _test_simple(…) will fail on "apple banana" (due to 'a'), and _test_hard(…) will fail on "familgerate fiddle-faddle" (due to '-').
Reminder: count_words(…) has intentionally planted bugs. It is okay that some of the results are obviously incorrect and the tests fail. The purpose of tests is to detect flaws. If the flaws are detected, then the test worked correctly.
We are giving you some code for free. You are welcome to make changes, or hand-copy as is, as long as you understand what it is doing. If not, please ask.
7. You must submit your test_count_words.c, even if it is identical to the code above.

How much work is this?

Requirements

Submit

To submit HW07 from within your hw07 directory, type 264submit HW07 miniunit.h clog.h test_count_words.c

Pre-tester ●

The pre-tester for HW07 has not yet been released. As soon as it is ready, this note will be changed.

Q&A

1. Should I have a semicolon at the end of the RHS of a #define macro

   No. The person using the macro will normally include the semicolon.

2. How can I continue the RHS of a #define macro definition onto the next line

   Put a backslash ("\") at the end of the line.

   #define profess_love_for_food(food) printf("I love %s", \
           food)

3. Can I have a macro with multiple C statements

   Yes. In theory, you could just have the two statements separated by a semicolon like this:

   // BAD
   #define profess_love_for_two_foods(food1, food2) \
         printf("I love %s", food1);                \
         printf("I love %s", food2)

   However, that would lead to surprising results if someone who doesn't follow the code quality standards calls that macro in an if statement like this:

   if(1 == 0)
       profess_love_for_two_foods("soap", "poison");

   Only the first statment would be connected to the if statement.

   if(1 == 0)
       printf("I love %s", "soap");
   
   printf("I love %s", "poison");

   It is tempting to just put curly braces around the two statements, but that also causes problems.

   // BAD
   #define profess_love_for_two_foods(food1, food2) { \
       printf("I love %s", food1); \
       printf("I love %s", food2); \
   }

   The problem comes back to uncivilized oafs who write if statements without curly braces, like this:

   if(age >= 30)
       profess_love_for_two_foods("chocolate", "pizza");
   else
       profess_love_for_two_foods("spinach", "broccoli");

   The above example would result in this:

   if(age >= 30) {
       printf("I love %s", "chocolate");
       printf("I love %s", "pizza");
   };  ← PROBLEM
   else {
       printf("I love %s", "spinach");
       printf("I love %s", "broccoli");
   };

   The standard solution is to wrap the statements in a do { … } while(false) block. Because the do…while requires a semicolon, this actually works out like we want.
   Yes, it is ugly. Hacks like this are not something the instructor would normally condone, but it is standard practice because there are very few truly versatile options for this.

   // USE THIS WAY
   #define profess_love_for_two_foods(food1, food2) do { \
       printf("I love %s", food1);                       \
       printf("I love %s", food2);                       \
   } while(false)

4. Why does miniunit.h need mu_start(…) and mu_end(…)

   These give a relatively clean and consistent form to your unit tests, so you can focus on the code that matters for each particular test.
   Without mu_start(…) and mu_end(…), one might resort to a naïve approach, like this:

   YUK!
   int test_count_words_▒▒▒▒▒() {
       int test__mu_failure_line_num = __MU_SUCCESS;
       mu_check(▒▒▒▒▒▒▒▒▒▒);
       mu_check(▒▒▒▒▒▒▒▒▒▒);
       mu_check(▒▒▒▒▒▒▒▒▒▒);
       return __mu_failure_line_num;
   }

   The programmer should not have to know the specific internal names used by miniunit.h! If they get it wrong, then it won't work properly with mu_check(…).
   Of course, they could avoid the problem of consistency if they do the whole thing without any macros—i.e., without miniunit.h—but then the above skeleton would be even messier!

   YUK! YUK! YUK! YUK! YUK!
   int test_count_words_▒▒▒▒▒() {
       const int __MU_SUCCESS = 0;
       int __mu_failure_line_num = __MU_SUCCESS;
       if(!(▒▒▒▒▒▒▒▒▒▒) && __mu_failure_line_num == __MU_SUCCESS) {
           __mu_failure_line_num = __LINE__;
       }
       if(!(▒▒▒▒▒▒▒▒▒▒) && __mu_failure_line_num == __MU_SUCCESS) {
           __mu_failure_line_num = __LINE__;
       }
       if(!(▒▒▒▒▒▒▒▒▒▒) && __mu_failure_line_num == __MU_SUCCESS) {
           __mu_failure_line_num = __LINE__;
       }
       return __mu_failure_line_num;
   }

   With mu_start(…) and mu_end(…)—and adding a divider comment (optional)—you get this:

   GOOD
   int test_count_words_▒▒▒▒▒() {
       mu_start();
       //────────────────────────────────────────
       mu_check(▒▒▒▒▒▒▒▒▒▒);
       mu_check(▒▒▒▒▒▒▒▒▒▒);
       mu_check(▒▒▒▒▒▒▒▒▒▒);
       //────────────────────────────────────────
       mu_end();
   }

   mu_start(…) and mu_end(…) also make your unit testing library more extensible. If you wanted to change how your tests are organized and/or reported, you could do so without changing all of your test code that uses it.

5. What does #x do in a #define macro

   It expands to the text of the expression, instead of its value. This is easiest to see if you test using the /usr/bin/cpp command.
   Here is an example, which uses the log_int(…) snippet given in the Requirements table.

   // demonstrate_hash.c
   #include <stdio.h>
   #include <stdlib.h>
   
   #define log_int(n) fprintf({{ stream }}, "%s == %d\n", (#n), (n))
   
   int main(int argc, char* argv[]) {
       log_int(3 + 3);
       return EXIT_SUCCESS;
   }
   

   If we process that with the preprocessor directly (instead of via gcc), we can see what it becomes.
   you@ecegrid-thin1 ~/HW07 $ /usr/bin/cpp demonstrate_hash.c -DDEBUG | indent -kr

   … int main(int argc, char* argv[]) { fprintf({{ stream }}, "%s == %d\n", ("3 + 3"), (3 + 3)); return 0; }
   Notice that the third argument to fprintf(…) is a string literal, "3 + 3"—the text of the argument that was passed to log_int(…). That is different from the fourth argument, which is the value of that parameter, 3 + 3 (= 6).

6. How do I make mu_run(…) print the success/failure messages in color even when the DEBUG symbols is not defined (i.e., -DDEBUG not passed to GCC), and without duplicating the ANSI codes in miniunit.h

   Define a helper macro for printing in color outside the #ifdef/#endif. You can use the one in the Q&A of HW06 or make your own. You will need to define the ANSI codes (#define ANSI_RED …, etc.) outside the #ifdef/#endif, as well.

7. GCC: “ISO C99 requires rest arguments to be used.” ⋯??

   In variadic macros, the ... can stand for 1 or more arguments. If your logf_red(…) looks like #define logf_red(format, ...) ▒▒▒▒▒▒▒▒▒▒, it will likely work as long as you pass ≥2 arguments (e.g., logf_red("I like %d", 5)), but if you pass only a string (e.g., logf_red("orange sky")), then there are 0 arguments for the .... It needs ≥1. The solution for our purposes is to remove the first argument (format). The correct form was given in the Q&A of HW06.

8. GCC: “error: ‘true’ undeclared” ⋯???
   GCC: “error: ‘false’ undeclared” ⋯??

   The true and false constants are defined in a standard header called stdbool.h; you need to include it (i.e., #include <stdbool.h>) in any file where you use them (e.g., clog.h). This was communicated in #8 of the Requirements table, as well as the Code Quality Standards, which have more about bool, true, and false.

9. GCC: “implicit declaration of function ‘isatty’” ⋯???
   GCC “‘STDOUT_FILENO’ undeclared” ⋯??

   The isatty(…) function and STDOUT_FILENO symbol are defined in a standard header called unistd.h you need to include it (i.e., #include <unistd.h>). This was communicated in #7 and #8 of the Requirements table.

10. GCC: “__mu_failure_line_num is not defined” ⋯ Why

    If you get that within main(…), make sure you are not trying to use the variable declared in mu_start(…) in your mu_run(…). Your mu_start(…) is expanded in the context of a _test_▒▒▒(…) function, while mu_run(…) is expanded in the context of main(…).

11. CPP: "Unexpected EOF (end-of-file)" (or similar). Why?

    You probably have unmatched braces somewhere in your clog.h or miniunit.h. First, fix your indents. It will make the errors more obvious. Then, be sure to use /usr/bin/cpp test_count_words.c | indent -kr or /usr/bin/cpp test_count_words.c -DDEBUG | indent -kr to make the output readable.

12. How do I start? Can I use TDD?

    Try this progression:
    1. Make it work for an empty test function (i.e., no calls to mu_check(…)).
    2. Make it work for tests that always pass. In other words, assume the condition is true.
    3. Make it work for test functions with one test that fails.
    4. Make it work for test functions with any number of tests that fail.

Updates