Real Case Application of Coverage Tests on Linux - Part 3

These special topics on coverage tests I made are based on real case situations of tests implementation, when I used to think about a nice way to certify that the user requirements needs related to a MythTV protocol API implementation I´m developing are well concerned. This API is not the official one, based on the MythTV server and client implementations, but another totally different effort based on GObject/GLib frameworks and object model.

The main test architecture I had developed is to allow that the test script's writer could develops the testing scripts with the minimum possible effort. So, I choose to use Python as the language of choice on the test scripts, because it's easier to learn, and removes the extra step of compiling code, because Python is interpreted. Each one of these Python scripts do, at least, one basic task: start a process with the testing program, compiled with the gcov program line arguments. A very simple strategy had been used to guarantee that the process testing stoped at a given known time, and the Python script could be signalized about the program termination.

You can see all these concepts, and see how to code gcov-based tests in practice, seeing my contribution on GMyth's SourceForge: http://sourceforge.net/projects/gmyth.

Have a lot of fun! :)

More on Coverage Tests on Linux - Generating Better Reports

At the last post, I showed up some code that could be interesting for testing purposes, and to see how the gcov do these automatic testing procedures. In fact, gcov uses a number of techniques to cover a given test code. Internally, we can say that gcov will insert into your program object file, some kind of meta-code, which will try for a large number of meaningful execution paths. So, it uses some know strategies to select what are the most effective paths he must to cover, because is theoretically impossible just try to cover all the possible situations when a program is running.

There are 2 main types of coverage: branch coverage and loop coverage. Just to give you an idea, in the branch coverage, it is important to verify that every branch has been taken. When it uses the loop coverage, gcov try to take all the execution paths through a loop. To see more about gcov and some information about some internal working issues, refer to http://gcc.gnu.org/onlinedocs/gcc/Gcov.html.

Since our gcov execution at the last article produced some files that encapsulates all the information above, we'd got almost all we need. But could be better if we could put all the data collected from gcov in a more beautiful and useful way. That is what a tool called lcov do:

lcov -d . -c -o test_gcov.info

This simple command will search for the files generated by gcov (*.gcda, *.gcno) at the current (".") directory (-d option), and put this in the test_gcov.info. This file has the collected information about the program execution and automatic testing, but is not yet in a human-readable form. But we can put this file as an input to a very useful tool, genhtml:

genhtml test_gcov.info

After this program script finishes execution, type "ls" on your prompt and see what it had just created: a lot of images (.png), CSS (cascading style sheets) and HTML files. Open the "index.html" in your preferable browser, and it will show you how powerful this tool can really be, as a valuable way to certify that your test scripts are covering the software requirements initially specified, which test cases are important, and if you can find some dead code that you can throw off.

Coverage Tests on Linux

Every time you want to add a new test case to your software implementation (i.e., a library, a framework, a network server), your client may want to know how these tests are being done, if the test specifications corresponds to the software requisites, and how much of these functionalities are being covered by the test cases. It may be difficult to say to anyone that is not a real hacker, or doesn't know so much about your implementation, that your tests are being effective. In order to allow that your test case script start reporting how much it has been covering from the target implentation, there is a tool that you can use: GCOV and LCOV, a great effort from the LTP group.

These coverage testing tools can be helpful to you that want a kind of automatic process to determine how much of the code is actually being covered by your tests, which is always an indirect measure, but it works. And its very important to know that coverage tests cannot find logical errors in a program! The code coverage analysis is very well discussed in the software testing and quality assurance fields of software development, and I will show now how you can start using them to help you on your tests.

At first, you should compile the programs you want to be covered with 2 special GCC parameters: -fprofile-arcs and -ftest-coverage. If you want to apply coverage testing to an API you are developing, certainly you should add these parameters to all source code being compiled (if you are using autoconf, add this to the CFLAGS variable, on configure.{ac,in}). This will cause the GCC compiler to add special symbols inside the generated object files.

Now let's compile some code, including in there the coverage information. Here is a code example that will certainly fall into some memory error and cause incomplete code execution:




#include <stdlib.h>
#include <stdio.h>

int
main( argc, argv )
     int argc;
     char **argv;
{
     int x;
     char **str = (char**)malloc(20 * sizeof(char*));
     for ( x = 0; x < 23; x++ )
     {
             str[x] = malloc( 3 * sizeof (char) );
             if ( NULL == str[x] )
                     exit(-1);
     }

     exit(0);
}

Now, compile the code above with the following line:

gcc -ftest-coverage -fprofile-arcs test_gcov.c -o test_gcov

This will generate not only the test_gcov executable object, but another important GCOV object structure, which you can see as test_gcov.gcno. This is only part of the things you need to do before running the tests using the types of coverage GCOV do to you. After this .gcno file is generated, you should execute the application:

./test_gcov

This will create another important file, test_gcov.gcda. With this test run description file, the "gcov" tool will inform you about how much percent you program could be covered by its automatic testing procedures. So you can run gcov passing the test_gcov source code to be analyzed:

gcov test_gcov.c

And you will see something like the following:

File 'test_gcov.c'
Lines executed:85.71% of 7
test_gcov.c:creating 'test_gcov.c.gcov'

Next, we will see how to make better reports over your coverage test scripts, presenting them in .HTML, and showing these percentages for each line of covered code.