ISSRE 2012 Industry talk - Lakshmankumar Mukkavilli, Cisco Systems Inc.

• Report rights infringement
• published: 10 Jun 2014
• views: 105

Unit testing (UT) is an important step in ensuring the quality of software. Considerable effort is spent in unit testing. There are several frameworks to help with UT. Some common frameworks are Cunit, Junit, Nunit etc. All of these have very similar capabilities. They provide capabilities for organizing and registering tests, registration of test initializers and finalizers, test execution and asserting the test outcome and reporting the summary. The actual task of bug finding is left to the user. Users write the code to perform the tests and check the outcome. A common characteristic of the unit tests is that the fraction of the code that actually performs the tests is small. The bulk of the test code was for bookkeeping. My observation is that most tests are very basic. Mostly they are API/functions calls. We were very concerned about the ROI (Return on Investment). A significant percentage of the CFD's (Customer Found Defects) were identified as UT escapes. I started looking at enhancing the value of unit testing. I designed and built a new unit testing framework. My goals were to maximize bug finding capabilities while reducing the effort in creating the tests. The new framework has direct support for Fault Injection, automatic generation of API robustness tests, code coverage, MBT (Model Based Testing), performance and scalability testing. The model used for defining a test is a directed graph. Every test is structured as a directed graph. Evaluation of the new framework was performed by using two features for which unit tests were written in one of the standard UT frameworks. These UT efforts were considered to be very good. The same features were tested using the new framework. No new test code was added. Here are the results: Feature 1 -- The original tests found 8 bugs. The new framework found 24 bugs. Included in these 24 are the 8 bugs that were originally found. Of the remaining 16 bugs, 8 were API robustness failures. 5 bugs were bugs in the error handling code (two of these 5 pointed at missing error handling code). 3 bugs were uncovered due to the sequencing of the original tests. Since the tests are embedded in a directed graph, the testing involved traversal of various paths. Code coverage (statement coverage) increased by about 8%. The number of lines in the test code decreased by about 50%. Feature 2 -- The original tests found 10 bugs. The new framework found 30 bugs. Included in these 30 are the 10 bugs found originally. Of the remaining 20 bugs, 12 were API robustness failures. Four were bugs in the error handling code (two of the 4 pointed at missing error handling code). Four bugs were uncovered due to the sequencing of the original tests. Since the tests are embedded in a directed graph, testing involved traversal of various paths. Code coverage (statement coverage) increased by about 10%. The number of lines in the test code decreased by about 50%. The new UT or whitebox testing framework is being used in several projects. Testers use the framework for whitebox testing. Unique features of the new framework like API robustness test generation, Fault Injection, MBT, performance/scalability analysis have accounted for about 70% of the bugs found during unit testing.