Unit Testing

Each line of code you design is testable, and the most efficient way to test it is through unit-tests. In fact, we should test code before the inception of the code itself. Sounds strange? Not really, this is called Test-Driven-Development, or TDD. Without unit-testing, we may be really shit developing rather than software developing. Unit-Tests provide an important barrier very, very close to the developer to ensure quality at inception of the code.

Without unit-tests:

• You cannot assess how well the code will work (if at all)

• Someone could alter the code and easily break it (software regression)

• Refactoring and improving the code is much harder, and time consuming

Definition

• Typically automated; GDB is not a replacement

• Testing one code module at a time, such as file.c

• Write code to test code (same programming language)

  • Write a test case for each line of production code

  • Write a code module and test it, such as memory_buffer.c

• Unit Testing (UT) is not necessarily testing it on your board

  • Typically, UT means you test it on the machine that is compiling the code Hence, no need to load the code to the target, or the embedded processor

  • Using a printf to manually inspect whether your code works is not unit-testing

• In some industries such as Aerospace, it is required to run UTs on your target

  • This validates that the compiler for the target is free of bugs

  • For example, if you run UT on x86 machine, and your code is targeted on Cortex-M4, then that proves that the code will work on x86, but not necessarily on the M4

Benefits

• Develop without hardware

• Refactor code with confidence

• Reduce the costly debugging sessions


• Significantly Accelerate development

• Establish a strong barrier against bug leaks

• Serves as a double check during development

• Reduce the discovery, investigation, and patches for software bugs

• Get immediate feedback if the code is working as the developer intended

Cost

The cost of writing unit-tests is negligible compared to the cost of finding and patching a bug at a later time. This cost is not just the developer time alone, but also the time of many other parties involved, including the customer itself which suffers from lost productivity. It is easy to test weird scenarios your code could get into during unit-tests, and it is difficult to force your code to go into complex scenarios during product's integration tests.

Clean C++

Open

The cost of writing a UI test is often high, and hard to automate. In terms of raw code, it is easy to write an automated test and focus towards a very precise functionality and it is usually difficult to inject and create certain test scenarios towards the higher end of the pyramid. The cost of creating the anti-pattern below is prohibitively high.

Clean C++

Open

Pitfalls

Unit-Tests take too much time and slows me down

Open

This is the most popular fictional statement. It is like saying that you do not have time to fix your bicycle when you have miles ahead to travel.

Unit-Tests are lot of maintenance

This should signal code smell, possibly in the unit-test code, production code, or both. Fundamentally, code should be modularized, and designed for simplicity to reduce maintenance. If unit-tests are a lot of maintenance, then they can probably be improved, and this may indicate potential problems with the production code.

We will hire an intern to do unit-testing later

This is an lol statement. Usually, later will never happen anyway. Further, if your code was not designed with testing in mind, then it is probably not testable. Without tests in the first place, the code likely suffers design and quality issues, and trying to perform unit-testing after the fact is not that useful. If the thought of UTs comes after the code is written, some of the benefits are already stolen from you.

It is the developers responsibility to deliver code that is believed to be bug-free, unit-tests provide a good way to ensure high quality before another un-biased party tests your code as a product black-box.

Test Frameworks

Unfortunately C and C++ do not have unit-testing built into the language, and therefore there are many test frameworks that came to existence.

• None

  • Simply use <assert.h>

• C & C++ Frameworks

  • Check (C)

  • CGreen (C and C++)

  • Boost (C++)

  • Google Test & Google Mocks (C++)

• Unity & CMock – C Test framework

  • This README is focused on this

  • Small foot-print to run on the microcontroller itself

  • Combined with Cmock, provides huge value to UT and “mock” APIs

Comparisons

• CMock vs. FFF

• Another Comparison

Unity & CMock

The Unity & CMock framework is one of the best frameworks we have found for C language. There has not been anything fundamental this framework lacks, and it reduces a lot of coding effort because it generates the Mock functions for you. You can read about these two frameworks here but it is advisable to read this article first.

• Unity README

• CMock Summary

Unity

There are two basics things to understand about a unit-test framework. The first is the unit-test framework itself; this only provides the ability to perform assertions and write tests in a way that the framework understands how to run. Some frameworks allow you to “register” for the tests, and then they will invoke all of the registered tests. Other frameworks may either use macros that register themselves, or use scripts at compile time to register and run your test methods. Unity provides the ability to run tests in a structured way.

CMock

CMock creates mocks and stubs for C functions. It's useful for interaction-based unit testing, where you want to test how one module interacts with other modules. Instead of trying to compile all those real units together, CMock helps you by creating fake versions of all the "other" modules. You can then use those fake versions to verify that your module is working properly!

http://www.throwtheswitch.org/cmock

A secondary artifact for a test framework is the "mock" functionality. The mocks provide the ability to mock-out an API and hijack the functions calls outside of your code module. For instance, you can mock an API that is going to delete a database and perform assertions. The objective would be that you are testing a code module that is interfacing to a database, but you don’t want to use the database code module itself and instead want to use a dummy mock. This will make a whole lot of sense during our unit-test examples.

CMock is useful when you wish to test a piece of code without inheriting another code module. In the following example, our focus is to test my_app() but we want to Mock database_connect() function and make it return either NULL or a pointer to the database to test further code. CMock in this scenario would allow you to "stub" the database.h API, inject, verify parameters, and make functions return whatever values you wish.

In your unit-test, you can then do this:

CMock Plugins

CMock has many plugins that are built when you compile the library. You can simply enable all of them because it is up to your test code to use it or not. Including more plugins than necessary should not cause any side effects.

• ignore

• ignore_arg

• expect_any_args

• array

• cexception

• callback

• return_thru_ptr

Test Driven Development

This design technique should be used when the requirements of the code are clearly stated. It is meant to simplify code, and produce minimal code necessary to solve a problem.

One of the goals to keep in mind is that It is about how little code solves a problem, not how much and TDD helps you with this.

1. Create empty tests

2. Write failing test

3. Write just enough code to pass

4. Refactor the code, cleanup and optimize

5. Repeat until all tests are passing

Benefits

• No dead code

• Sign off with stakeholder

• Unit Tests shape your production code

In TDD, after you write a test, and just enough code for the test to pass, it reaches a critical point of success:

Now we have reached a remarkable point in the process. If the tests pass now, we always have 100% unit test coverage at this step. Always! Not only 100% in the sense of a technical test coverage metric, such as function coverage, branch coverage, or statement coverage. No, much more important is, that we have 100% unit test coverage regarding the requirements that were already implemented at this point!

Stephan Roth. “Clean C++.”

Example

Let's design a buffer module with TDD.

TODO Screencast with example

Philosophy

The most popular argument against writing unit-tests is that they slow down development. Absolutely nothing could be farther away from reality for this statement. If I could let me nerves respond to this comment, I would respond back by saying that whoever has made such claims is either an incompetent developer, or is simply not experienced enough. This sentiment may exist because the developer has simply not practiced unit-testing to reveal the benefits to debunk this assumption.

Clean C++ shares more or less the same sentiment on the benefits of unit-tests:

• Fixing bugs after software has shipped is more expensive than having unit tests in place

• Unit-tests give an immediate feedback about your entire code base. Provided that test coverage is sufficiently high (approx. 100%), developers know in just a few seconds if the code works correctly.

• Unit tests give developers the confidence to refactor their code without fear of doing something wrong that breaks the code. In fact, a structural change in a code base without a safety net of unit tests is dangerous and should not be called Refactoring.

• A high coverage with unit tests can prevent time-consuming and frustrating debugging sessions.

• Unit tests are a kind of executable documentation because they show exactly how the code is designed to be used. They are, so to speak, something of a usage example.

• Unit tests can easily detect regressions, that is, they can immediately show things that used to work, but have unexpectedly stopped working after a change in the code was made.

• Unit testing fosters the creation of clean and well-formed interfaces. It can help to avoid unwanted dependencies between units. A Design for Testability is also a good Design for Usability...


 Refactoring without tests isn’t refactoring, it is just moving shit around

— Corey Haines

Aim for 100% Test Coverage

Setting a high bar yields high quality software. Anything less than 100% coverage is an arbitrary number, and therefore, the only acceptable measure should be exactly 100% code coverage. As developers write a line of code, it should be immediately tested. This discipline pays off well because the code is testable to begin with, and often times, the developers would be motivated to write less code to solve a problem. Remember that It is not how much code you write, it is how little said one of my past co-workers, and I still remember that statement today.

Emphasize the tests that matter

There may be code that is way too trivial to test. For example, if an RTOS task code, or the code of a main() function is kept simple (and branchless), then that may be once place you can skip the unit-test effort. Generally, creating rules, and then creating exceptions is not the way to go. However, there are certain situations where this particular logic makes sense.

The case we are setting forth is that top level “glue code” may be exempt from 100% code coverage. Experience suggested that when the code is modularized, it was always the modules that were at fault, but not the code that glued different pieces together. This glue code should have the following properties:

• No branches

• Uses dependency injection to connect objects

• Runs a periodic loop or spawns a task

We encourage this rule because even if we were to test this branchless code, the only thing we would test is that certain code is called in the right order with the appropriate parameters. Let’s demonstrate this by example using a top level RTOS task.

In the code above, we have these things going on:

• Setup a buffer of 512 bytes

• TCP connection being setup on port 1200

• Loop to service the TCP connection

The unit-test for this code would look like this:

You should be able to now reason with this approach, but we have to proceed with a discipline that rightfully justifies this exception. In the test code above, all we are really doing is testing if certain functions are getting called, but there is no branch logic to test. The 100% code coverage of buffer, tcp_connection is actually responsible to make sure that the code works well, and there is little that could go wrong in this top-level RTOS task. Sure, you could go ahead and test that the tcp connection is not getting passed a NULL pointer for the buffer, but it is trivial enough to ensure this by running perhaps the simplest test on your target platform.

The task level code should just be the glue code that connects the TCP connection to our buffer, and then the connection utilizes the buffer to perform I/O. Statistically speaking, there is little that can go wrong in this code as it is just few jigsaw pieces that we need to connect. The actual bugs are likely to occur inside of the buffer or the TCP connection code modules, and the suggestion is to maximize the testing to 100% in these modules, rather than focusing on top level glue code that simply pieces things together.

Positive and Negative Testing

Positive and negative testing is a fundamental mindset when testing any unit. It is the idea of testing against both valid inputs and invalid inputs. Testing against invalid inputs ensure the unit under test is sufficiently robust enough to handle irrational inputs.

In this scenario, the valid input domain for resistance R is (0.0, FLOAT_MAX]. Voltage can be anything in this case.

Positive test cases (test rational resistances)

1. Expect calculate_current(0.0, 1.0) == 0.0

2. Expect calculate_current(0.0, FLOAT_MAX) == 0.0

Negative test cases (test irrational resistances)

1. Expect calculate_current(0.0, 0.0) == 0.0 (No runtime divide by 0 exception should occur)

2. Expect calculate_current(0.0, -1.0) == 0.0 (No runtime divide by 0 exception should occur)

Basics of Unity & CMock

The Unity and CMock unit-test infrastructure relies on developer discipline to create files with consistent names:

• your_module.h Example: gps_string_parser.h

• your_module.c Example: gps_string_parser.c

• test_your_module.c Example: test_gps_string_parser.c

The Test framework automatically picks up source code that starts with test_ and then begins to create an executable specifically to perform the unit-test of one file at a time.

How it works

Unity and CMock framework uses Ruby and Rake to turn your test_your_module.c into a standalone executable What this means is that each test_* is actually a separate executable and is compiled by resolving the header files you included in this test_your_module.c source file.

Run Tests

Running the Unity test framework with rake is very simple.

• Go to the folder that contains rakefile

• Type rake on the command-prompt

  • To run a single test, type rake unit single_file=code_test\test_simple.c

• The rake build system will run all unit-tests as separate executables

• Ensure that gcc.yml contains the paths to your source code

Files of the Test Infrastructure

There are a few ruby files that glue things together.

• rakefile

  • This is the entry point when you type rake to run the tests

  • Additional logic can be added here to customize the unit-test framework

• rakefile_helper.rb

  • rakefile uses this code to compile and run tests

  • We built this from the Cmock example, and customized it

• gcc.yml

  • rakefile.rb uses this configuration to compile your code

How it Builds

• The ruby script forms a list of all tests that begin with test_ in your code_test folder

• Script compiles a separate executable for each test

  • This means each unit-test file, such as test_buffer.c is a standalone program

  • All dependencies need to be #included in your test_ file

• As part of the compilation of the unit-test executable, files are mocked

  • Each #include that begin with #include "Mock" will not build the real code, but instead it will build Mocked code

  • For foo.h, it will be Mockfoo.h and Mockfoo.c at ut_build/mocks

• Be careful of nested dependencies, such as your code_under_test.c depending on buffer.c

  • If there is a dependency you #include which has another dependency, you will also need to #include those dependencies in your test_code_under_test.c as well. So if you do not mock buffer.c, then you will also need to build the real sources that the buffer.c depends on.

  • The other option is to mock the header file to avoid picking up nested dependencies. So if you #include "Mockbuffer.h" then you do not need to worry about the nested dependencies of buffer.c

Behind the scenes, the script will include all files you included in your test_buffer.c and use that to build the executable. The trick is that the files that are #included as Mockfoo.h are mocked, meaning that the header file is used to initially compile, but the code is linked to the CMock, instead of the real foo.c file.

Available APIs

In a lot of the sample code below, we will see some "magical" APIs, so let us first unravel the mysteries and point out what kind of function you get with Unity and CMock.

Unity

Unity provides you with:

• Assertion APIs

• setUp(), tearDown() invocations before and after each test method

CMock

The idea behind the Mocks is that sometimes you want to test a module and you do not want to inherit the functionality of another object that you have little or no control over. Consider a naive piece of code that you wish to test.

In the naive example above, you wish to perform two tests:

• When database_connect() returns NULL

• When database_connect() returns a valid database pointer

CMock provides the Mock functionality. It extends the Unity assertions, but fundamentally provides you with a make system to stub out an external module's functionality with a fake one.

The APIs are dynamic and auto-generated based on the file/module being mocked. What really happens is that when your test_app.c performs a #include "Mockdatabase.h", then the build system purposely omits the real database.c and replaces the implementation with Mock replacements.

The test code for app_test() would look like this:

To Mock or not to Mock?

This is an important decision. A general guideline is that:

• Do not Mock APIs that are very trivial

  • Example: bit_count.h

  • Let the bit counting happen the way its meant to be

• Mock code modules when

  • Code modules are more complex; example: database_connect()

  • Code modules may create a distraction from the object under test

You have two options; the first option is to #include "bit_count.h" at your unit-test file. This way, you do not mock this file but inherit the real functionality of this file.

Sometimes, it is easier to build the real code, such as a simple utility called bit_count.h and it is better to not mock this file. At other times, it is better to be able to hijack an API and make it do what you want for the sake of ease of unit-testing. The idea is that you are unit-testing your module, and you don't want to depend on a behavior of another module you inherited. The testing of the other module is the other module's responsibility, and not yours.

The second option is to mock this header file such that you can hijack its function calls, inject and return data for the sake of unit-testing. Which option you choose depends on your test.

When you include Mockbit_count.h, the UT framework will not build and include the real file bit_count.h. Instead, it will redirect the API as given in your header file with the mock framework's implementation which will impose Expect requirements in your unit-tests. The Expect API has a few flavors as listed in the next section.

Mock Variations

Mocks always provide StubWithCallback which means you can install your own Mock function. Typically, other variations provided should be able to do the job, but the callback can be used to do perhaps a fancy operation inside of the Mock.

The Ignore() should rarely be used. It means that always ignore whenever a function invocation occurs. The ExpectAnyArgs() is better because at least you are stating that a function is expected. Once you do Ignore() in a test function, then whether a function is called zero times, or a million times, it really is ignored.

• Expect for functions that do not return a value

• ExpectAndReturn for functions that return a value

• ExpectAnyArgsAndReturn to ignore input arguments but return a value

• Ignore dangerous method of just ignoring the expectation

• IgnoreAndReturn Ignore but always return something for functions that return a value

• StubWithCallback Use a custom callback that can have a small test driver of your own

What this means is that:

• For a function with no return values: void foo(void), you will have:

  • foo_Expect()- Expect a function call to occur

  • foo_Ignore()- Ignore the function call (if any)

  • foo_StubWithCallback(your_func)- Go to your your_func() when foo() is called

• For a function with a return value: int foo(void), you will have the word AndReturn:

  • foo_ExpectAndReturn(#)

  • foo_IgnoreAndReturn(#)

  • foo_StubWithCallback(func_ptr)

Mocks for functions that have arguments are covered in the next section, but here is a brief summary to get the idea of the generated API.

• For a function with arguments: void foo(int arg_name), you will have:

  • foo_Expect() - Expect a function call with specific argument values

  • foo_ExpectAnyArgs() - Expect the function with no checks on any of the arguments

  • foo_IgnoreArg_arg_name() - After expecting a function call, ignore a particular argument

  • foo_StubWithCallback(func_ptr)

Parameters

For each function parameter, you would have another API available per parameter. The Ignore_arg_name() API should be called after setting up the Expect().

For example, if a function to mock is void foo(int a, int b), then the following APIs are generated:

• foo_Expect(#, #) - Expect a function call with specific args

• foo_ExpectAnyArgs() - Expect the function call with any args

• foo_Ignore()- Ignore the function call (if any)

• foo_IgnoreArg_a() - Invoke after foo_Expect(a, #) to ignore the first parameter value

• foo_IgnoreArg_b() - Invoke after foo_Expect(#, b) to ignore the second parameter value

Pointer Parameters

Functions that use pointers as parameters have more variations. For each pointer parameter, you have the following options available:

• ReturnMemThruPtr Modify something inside of a parameter that is a pointer

  • For example, if the function parameter was named ptr then:

  • ReturnThruPtr_ptr() will be available

  • Good candidate if your pointers are void* and the length of the data is not known

• ReturnThruPtr

  • Use when your type is known, such as void foo(int *)

  • You can then ask the Mock to return a type: int value = 2; foo_ReturnThruPtr(&value);

• ReturnArrayThruPtr

  • Similar to ReturnThruPtr but you can return an array of integers

  • int values[] = {11, 22}; ReturnArrayThruPtr(&value, 2);

Add a Unit-Test file

If you don't want to get your hands dirty, and you want to simply leverage from the test infrastructure that is setup for you, then there is little to do:

• Create a new C module

  • You may have to edit gcc.yml to include the new folder path of your new module

• Create a new file that begins with test_ and put it in your test folder

  • Example: test_buffer.c

• Go to your test folder in a command-shell

  • Type rake to run all unit-tests

  • All test files at <your_test_dir>/tests will execute their unit-tests

Unit Test Best Practices

• Exclude third party code

• Unit-test should run extremely quickly (in second)

• Keep tests focused; one test per functional

• Code quality for tests should be equal to the production code

• Avoid hacks and blocks of #ifdef UNIT_TESTING in your production code