API Sanity Autotest
API Sanity AutoTest (ASAT) is an automatic generator of basic unit tests for a dynamic C/C++ library. It helps to quickly generate simple ("sanity" or "shallow"-quality) test cases for every function in an API using their signatures, data type definitions and relationships between functions straight from the header files ("Header-Driven Generation"). Each test case contains a function call with reasonable (in most, but unfortunately not all, cases) input parameters. The quality of generated tests allows to check absence of critical errors in simple use cases and can be improved by involving of highly reusable "specialized types" for the library. API Sanity AutoTest can execute generated tests and detect crashes (segfaults), aborts, all kinds of emitted signals, non-zero program return code, program hanging and requirement failures (if specified). API Sanity AutoTest may be considered as a tool for out-of-box low-cost sanity checking of library API or as a test development framework for initial generation of templates for advanced tests. Also it supports universal Template2Code format of tests, random test generation mode and other useful features. This tool is free software: you can redistribute it and/or modify it under the terms of the dual license: GNU GPL or LGPL.
See also: Upstream Tracker for OSS C/C++ libraries.
We are looking for sponsorship to continue development of this tool. Please send your proposals to api-sanity-autotest@linuxtesting.org.
Contents |
Downloads
The latest release can be downloaded from this page.
Supported Platforms
GNU/Linux, FreeBSD, Haiku (BeOS), Mac OS X, MS Windows (Xp, Vista, 7)
System Requirements
Linux - GCC (3.0-4.5.1, recommended 4.0 or newer), binutils (c++filt, readelf, objdump) and Perl (base),
WARNING: if you are using ccache program (i.e. g++ points to /usr/lib/ccache/g++) then it should be newer than 3.1.2
MS Windows - MinGW (g++.exe, c++filt.exe), Active Perl and MS Visual C++ (dumpbin.exe, undname.exe, cl.exe),
In order to run the tool you should add g++.exe path (C:\MinGW\bin\) to your system PATH variable and run vsvars32.bat script (C:\Microsoft Visual Studio 9.0\Common7\Tools\).
Mac OS X - Xcode (gcc, c++filt, otool).
Examples of Generated Tests
| Library | Version | Number of Tests | Problems Found |
|---|---|---|---|
| freetype2 | 2.3.11 | 178 | 13 |
| libQtCore | 4.6.0 | 2936 | 17 |
| ALSA | 1.0.22 | 717 | 27 |
| libxml2 | 2.7.6 | 1633 | 56 |
| libssh | 0.4.0 | 224 | 83 |
| libX11 | 1.3.4 | 778 | 286 |
Usage
For generating, building and running tests you should provide the descriptor for your library version. It is a simple file with an XML-like structure that specifies version number, paths to header files and shared objects and optionally some other information. An example of the descriptor is the following (0.3.4.xml):
<version>
0.3.4
</version>
<headers>
/usr/local/libssh/0.3.4/include/
</headers>
<libs>
/usr/local/libssh/0.3.4/lib/
</libs>
Generating, building and running stages are described below:
Generating Tests
Command for generating a test suite:
perl api-sanity-autotest.pl -l <library_name> -d <version.xml> -gen
Viewing Tests
You can view generated tests using the index file:
tests/<library_name>/<version>/view_tests.html
or manually in the directory:
tests/<library_name>/<version>/groups/
Building Tests
Command for building tests:
perl api-sanity-autotest.pl -l <library_name> -d <version.xml> -build
Running Tests
Command for running tests:
perl api-sanity-autotest.pl -l <library_name> -d <version.xml> -run
After a time will be generated test report:
test_results/<library_name>/<version>/test_results.html
Improving Tests
To improve generated tests quality, you can provide the collection of specialized types for the library.
Examples
Generate, build and execute the tests for the libssh library:
perl api-sanity-autotest.pl -l libssh -d 0.3.4.xml -gen -build -run
Generate, build and execute the tests for the directory with headers and shared objects:
perl api-sanity-autotest.pl -l libssh -d installed/libssh/0.3.4/ -gen -build -run
Generate, build and execute the tests for the header and shared object:
perl api-sanity-autotest.pl -l libssh -d libssh.so.0.3.4,libssh.h -gen -build -run
Tests will be generated to:
tests/libssh/0.3.4/groups/
tests/libssh/0.3.4/view_tests.html
The report will be placed to:
test_results/libssh/0.3.4/test_results.html
Generate tests in the Template2Code (T2C) format:
perl api-sanity-autotest.pl -l libssh -d 0.3.4.xml -gen -t2c
The package with T2C tests will be generated to:
tests_t2c/libssh/0.3.4/t2c-libssh-tests-0.3.4.tar.gz
Execute the tests using the Xvfb server:
perl api-sanity-autotest.pl -l allegro -d 4.9.21.xml -run -xvfb
Generate tests using the Splint specifications (annotations) in the headers:
perl api-sanity-autotest.pl -l popt -d 1.16.xml -gen -splint-specs
Options
The tool has many useful options for manipulating with the test generation and execution processes. See the list of all options on this page.
Bugs
Please send your bug reports, feature requests and questions directly to api-sanity-autotest@linuxtesting.org or post it to the issue tracker at cpp-autotest.sf.net/tickets (new).
Applications
LSB 4.0 certification test suites for Qt3 (9792 interfaces tested), Qt4 (10803 interfaces tested) and libxml2 (1284 interfaces tested) libraries were developed with the help of this tool, also known as "AZOV Framework" in the past (2007-2009).
Future Plans
- Extending support of Splint and ACSL annotations.
- More appropriate distinguishing of pointer arguments (*x) and arrays (x[]).
- New kinds of test data files (images, fonts, ...).
- Crash test generation on the bound values for parameters.
- Continuing testing on the different libraries.
Maintainers
The tool was developed by the Russian Linux Verification Center at ISPRAS. Andrey Ponomarenko is the leader of this project.