Coverage Tests on Linux

Every time you want to add a new test case to your software implementation (i.e., a library, a framework, a network server), your client may want to know how these tests are being done, if the test specifications corresponds to the software requisites, and how much of these functionalities are being covered by the test cases. It may be difficult to say to anyone that is not a real hacker, or doesn't know so much about your implementation, that your tests are being effective. In order to allow that your test case script start reporting how much it has been covering from the target implentation, there is a tool that you can use: GCOV and LCOV, a great effort from the LTP group.

These coverage testing tools can be helpful to you that want a kind of automatic process to determine how much of the code is actually being covered by your tests, which is always an indirect measure, but it works. And its very important to know that coverage tests cannot find logical errors in a program! The code coverage analysis is very well discussed in the software testing and quality assurance fields of software development, and I will show now how you can start using them to help you on your tests.

At first, you should compile the programs you want to be covered with 2 special GCC parameters: -fprofile-arcs and -ftest-coverage. If you want to apply coverage testing to an API you are developing, certainly you should add these parameters to all source code being compiled (if you are using autoconf, add this to the CFLAGS variable, on configure.{ac,in}). This will cause the GCC compiler to add special symbols inside the generated object files.

Now let's compile some code, including in there the coverage information. Here is a code example that will certainly fall into some memory error and cause incomplete code execution:




#include <stdlib.h>
#include <stdio.h>

int
main( argc, argv )
     int argc;
     char **argv;
{
     int x;
     char **str = (char**)malloc(20 * sizeof(char*));
     for ( x = 0; x < 23; x++ )
     {
             str[x] = malloc( 3 * sizeof (char) );
             if ( NULL == str[x] )
                     exit(-1);
     }

     exit(0);
}

Now, compile the code above with the following line:

gcc -ftest-coverage -fprofile-arcs test_gcov.c -o test_gcov

This will generate not only the test_gcov executable object, but another important GCOV object structure, which you can see as test_gcov.gcno. This is only part of the things you need to do before running the tests using the types of coverage GCOV do to you. After this .gcno file is generated, you should execute the application:

./test_gcov

This will create another important file, test_gcov.gcda. With this test run description file, the "gcov" tool will inform you about how much percent you program could be covered by its automatic testing procedures. So you can run gcov passing the test_gcov source code to be analyzed:

gcov test_gcov.c

And you will see something like the following:

File 'test_gcov.c'
Lines executed:85.71% of 7
test_gcov.c:creating 'test_gcov.c.gcov'

Next, we will see how to make better reports over your coverage test scripts, presenting them in .HTML, and showing these percentages for each line of covered code.