ECE 264 Exercise 7

Test Coverage

When you are writing a program, you will probably execute it many times to check whether the program behaves correctly.  This is a basic form of testing. Software testing is a complex topic. You have to develop a strategy to ensure that the essential features are correctly tested before releasing the program (or submitting it as an assignment).

Test Coverage

When you execute a program, do you know which line of the code is actually tested?  Obviously, you cannot find a problem if the code is not tested. Test coverage reports the percentage of code being tested. A test with a high coverage uses more code. Since the purpose of testing is finding problems, a good test finds problems in a program. A test with a high coverage is likely, but not necessarily, a good test. A low-coverage test is likely, also not necessarily, a bad test. This exercise asks you to use gcov to obtain the test coverage.

Download the source code. The main function calls f1 or f2 based on input argument. If there is no argument, neither f1 nor f2 is called. If the argument is 1, f1 is called. If the argument is 2, f2 is called. If the argument is 3, both f1 and f2 are called.

In Makefile, add the following flag in CFLAGS (after -Wall)

     -fprofile-arcs -ftest-coverage

The flags turn on the coverage function. Type

          make

and execute the program without any argument.  Some files with .gcno and .gcda extensions are created. Type

         gcov main.c

and a new file called main.c.gcov is created.

        -:    0:Source:main.c
        -:    0:Graph:main.gcno
        -:    0:Data:main.gcda
        -:    0:Runs:1
        -:    0:Programs:1
        -:    1:#include 
        -:    2:#include 
        -:    3:#include "f1.h"
        -:    4:#include "f2.h"
        -:    5:int main(int argc, char * argv[])
        1:    6:{
        1:    7:  if (argc > 1)
        -:    8:    {
    #####:    9:      switch(atoi(argv[1]))
        -:   10:        {
        -:   11:        case 1:
    #####:   12:          f1();
    #####:   13:          break;
        -:   14:        case 2:
    #####:   15:          f2(264);
    #####:   16:          break;
        -:   17:        case 3:
    #####:   18:          f1();
    #####:   19:          f2(264);
        -:   20:          break;
        -:   21:        default:
        -:   22:          break;
        -:   23:        }
        -:   24:    }
        1:   25:  return 0;
        -:   26:}

The file shows that lines 7 and 25 are tested once. Lines 8 to 24 have not been tested. Now, execute the program again with an input argument 1 and use gcov again.  The file main.c.gcov is different.

        -:    0:Source:main.c
        -:    0:Graph:main.gcno
        -:    0:Data:main.gcda
        -:    0:Runs:2
        -:    0:Programs:1
        -:    1:#include 
        -:    2:#include 
        -:    3:#include "f1.h"
        -:    4:#include "f2.h"
        -:    5:int main(int argc, char * argv[])
        2:    6:{
        2:    7:  if (argc > 1)
        -:    8:    {
        1:    9:      switch(atoi(argv[1]))
        -:   10:        {
        -:   11:        case 1:
        1:   12:          f1();
        1:   13:          break;
        -:   14:        case 2:
    #####:   15:          f2(264);
    #####:   16:          break;
        -:   17:        case 3:
    #####:   18:          f1();
    #####:   19:          f2(264);
        -:   20:          break;
        -:   21:        default:
        -:   22:          break;
        -:   23:        }
        -:   24:    }
        2:   25:  return 0;
        -:   26:}

It shows that lines 12 and 13 are tested once. Notice that the test counts are cumulative; lines 7 and 25 have been tested twice so far. You  can also use gcov on f1.c to obtain f1.c.gcov.

What to Submit

Execute the program 5 times: without any argument, with argument 1, 2, 3, and 4. Do not delete any file between tests. The file main.c.gcov should show "Runs:5". Submit f1.c.gcov, f2.c.gcov, and main.c.gcov.

FAQ

Q: Is a test good if it has a high coverage?

A: A good test discovers problems. If a test covers only 10% and finds problems, it is a good test.

Q: Then, why do we care about test coverage?

A: Coverage is still important because you still need a plan to cover the other 90%.

Q: In addition to coverage, what else can we say about testing?

A: There are many factors determining the quality of a test. Suppose a test can cover 95%, how fast does it reach that coverage? Does it test the same 90% place a million times before testing the remaing 5%? If a program has one more feature, how easy is it to modify the test to cover that new feature?

Q: Why don't you give us the test cases used in grading the programming assignments?

A: Desiging test cases is not trivial. You should learn how to design tests so that your program is fully tested.

Q: Is testing the best way to find errors (bugs)?

A: Many critical programs are formally verified. They are proved that they cannot make any mistake in any circumstance. Formal verification is one of the important areas in software development. In 2007, Clarke, Emerson, and Sifakis received the ACM Turing Award (the "Nobel Prize in Computing") for their contributions in formal verification.