API Sanity Autotest

API Sanity AutoTest (ASAT) is an automatic generator of basic unit tests for shared C/C++ libraries. It is able to generate reasonable (in most, but unfortunately not all, cases) input data for parameters and compose simple ("sanity" or "shallow"-quality) test cases for every function in the API through the analysis of header files. The quality of generated tests allows to check absence of critical errors in simple use cases. The tool is able to build and execute generated tests and detect crashes (segfaults), aborts, all kinds of emitted signals, non-zero program return code and program hanging. It may be considered as a tool for out-of-box low-cost sanity checking (fuzzing) of the library API or as a test development framework for initial generation of templates for advanced tests. Also it supports universal T2C format of tests, random test generation mode, specialized data types and other useful features.

  See also: Upstream Tracker for OSS C/C++ libraries.

  We are looking for sponsorship to continue development of this tool. Please send your proposals to api-sanity-autotest@linuxtesting.org.

Contents 1 Downloads 2 License 3 Supported Platforms 4 System Requirements 5 Generation Algorithm 6 Examples of Generated Tests 7 Detectable Problems 8 Usage 8.1 Generating Tests 8.2 Viewing Tests 8.3 Building Tests 8.4 Running Tests 8.5 Improving Tests 8.6 Command-Line Options 8.7 Examples 9 Similar Tools 10 Bugs 11 Applications 12 Future Plans 13 Maintainers 14 Credits 15 Articles

Downloads

The latest release can be downloaded from this page.

License

This program is free software. You may use, redistribute and/or modify it under the terms of either the GNU GPL or LGPL.

Supported Platforms

GNU/Linux, FreeBSD, Haiku (BeOS), Mac OS X, MS Windows (Xp, Vista, 7).

System Requirements

Linux - GCC (3.0-4.5.2, recommended 4.0 or newer), binutils (c++filt, readelf, objdump) and Perl (base),
WARNING: if you are using ccache program (i.e. g++ points to /usr/lib/ccache/g++) then it should be newer than 3.1.2

MS Windows - MinGW (g++.exe, c++filt.exe), Active Perl and MS Visual C++ (dumpbin.exe, undname.exe, cl.exe),

In order to run the tool you should add g++.exe path (C:\MinGW\bin\) to your system PATH variable and run vsvars32.bat script (C:\Microsoft Visual Studio 9.0\Common7\Tools\).

Mac OS X - Xcode (gcc, c++filt, otool).

Generation Algorithm

The basic idea of the test data generation algorithm is to recursively initialize parameters of a function using the values returned (or returned through out-parameter) by other functions for structured data types (class, struct, union) or by some simple values for intrinsic data types (int, float, enum, ...). The recursion step includes the heuristic selection of the appropriate function, that should be called to initialize complex parameters of other functions. If some parameter of a function cannot be initialized, then the algorithm tries to select other function.

Let's see the example test for FT_Activate_Size ( FT_Size size ) function from the FreeType2 library:

   #include <freetype/freetype.h>
   int main(int argc, char *argv[])
   {
       FT_Library alibrary = 0;
       FT_Init_FreeType(&alibrary);//initialize "alibrary"
       
       FT_Face face = 0;
       FT_New_Face(
           alibrary,
           "sample.ttf",
           0,
           &face);//initialize "face"
       
       FT_Size size = 0;
       FT_New_Size(face, &size);//initialize "size"
       
       FT_Activate_Size(size);//target call
       return 0;
   }

In this test case the parameter "size" of target function FT_Activate_Size is initialized through the call of FT_New_Size function using its 2nd out-parameter. The first parameter "face" of FT_New_Size function is recursively initialized by the use of FT_New_Face function's 4th out-parameter. And finally the first parameter "alibrary" of FT_New_Face is initialized by the call of FT_Init_FreeType function on the 3rd recursion step. Other parameters of FT_New_Face function are initialized by intrinsic values.

Examples of Generated Tests

Library	Version	Number of Tests	Problems Found
freetype2	2.3.11	178	13
glibc	2.13	1996	340
ALSA	1.0.23	1292	48
libxml2	2.7.6	1633	56
libssh	0.4.0	224	83
libQtCore	4.6.0	2936	17
libX11	1.3.4	778	286

Detectable Problems

• Crash (segfault, signal SEGV)
• Abort (signal ABRT)
• All emitted signals: FPE, BUS, ILL and others
• Non-zero exit code
• Program hanging
• Requirement failure (if specified)

Usage

For generating, building and running tests you should provide the XML descriptor for your library version. It is a simple XML-file that specifies version number, paths to header files and shared libraries and optionally some other information. An example of the descriptor is the following (0.3.4.xml):

<version>
    0.3.4
</version>

<headers>
    /usr/local/libssh/0.3.4/include/
</headers>

<libs>
    /usr/local/libssh/0.3.4/lib/
</libs>

Generating Tests

Command for generating a test suite:
   perl api-sanity-autotest.pl -l <library_name> -d <version.xml> -gen 

Viewing Tests

You can view generated tests using the index file:
   tests/<library_name>/<version>/view_tests.html 
or manually in the directory:
   tests/<library_name>/<version>/groups/ 

Building Tests

Command for building tests:
   perl api-sanity-autotest.pl -l <library_name> -d <version.xml> -build 

Running Tests

Command for running tests:
   perl api-sanity-autotest.pl -l <library_name> -d <version.xml> -run 

After a time will be generated test report:
   test_results/<library_name>/<version>/test_results.html 

Improving Tests

To improve generated tests quality, you can provide the collection of specialized types for the library.

Command-Line Options

The tool has many useful options for manipulating with the test generation and execution processes. See the list of all options on this page.

Examples

Generate, build and execute the tests for the libssh library:
   perl api-sanity-autotest.pl -l libssh -d 0.3.4.xml -gen -build -run 

Tests will be generated to:
   tests/libssh/0.3.4/groups/ 
   tests/libssh/0.3.4/view_tests.html 

The report will be generated to:
   test_results/libssh/0.3.4/test_results.html 

Generate tests in the Template2Code (T2C) format:
   perl api-sanity-autotest.pl -l libssh -d 0.3.4.xml -gen -t2c 

The package with T2C tests will be generated to:
   tests_t2c/libssh/0.3.4/t2c-libssh-tests-0.3.4.tar.gz 

Execute the tests using the Xvfb server:
   perl api-sanity-autotest.pl -l allegro -d 4.9.21.xml -run -xvfb 

Generate tests using the Splint specifications (annotations) in the headers:
   perl api-sanity-autotest.pl -l popt -d 1.16.xml -gen -splint-specs 

Similar Tools

1. Trinity - The Linux system call fuzzer,
2. iknowthis - A system call fuzzer for UNIX-like systems,

Bugs

Please send your bug reports, feature requests and questions directly to api-sanity-autotest@linuxtesting.org or post to the issue tracker at forge.ispras.ru

Applications

LSB 4.0 certification test suites for Qt3 (9792 interfaces tested), Qt4 (10803 interfaces tested) and libxml2 (1284 interfaces tested) libraries were developed with the help of this tool, also known as "AZOV Framework" in the past (2007-2009).

Future Plans

• Add CUnit format support. See 2011 GSoC LSB projects.
• Extending support of Splint and ACSL annotations.
• Making better distinction between pointer arguments (*x) and arrays (x[]).
• More kinds of generated test data files (images, fonts, ...).
• Option for generating bound values for parameters.

Maintainers

The tool was developed by the Russian Linux Verification Center at ISPRAS. Andrey Ponomarenko is the leader of this project.

Credits

We would like to thank everyone who has contributed to the success of this project!

Articles

1. wikipedia.org, “Sanity testing in software development”
2. wikipedia.org, “Fuzz testing”
3. R. S. Zybin , V. V. Kuliamin , A. V. Ponomarenko , V. V. Rubanov and E. S. Chernov, “Automation of broad sanity test generation”