It’s a fact: When you create code, you create bugs. Listen in as we present the finer points of Test-Driven Development (TDD) - a game-changing approach to embedded software development that can greatly reduce bugs early in the development process. TDD is not a debugging technique but rather an approach to proactively preventing defects.
Slide 1: Preventing Embedded Software Bugs with TDD
Moderator: Welcome and thank you for attending Barr Group’s webinar Preventing Embedded Software Bugs with Test-Driven Development. My name is Sandy, and I will be your moderator for the next hour. Today’s webinar presenters are Michael Barr (Chief Technical Officer) and James Grenning (Principal Engineer).
Slide 2: About Barr Group
Thank you. My name is Michael Barr; I am the Chief Technical Officer and the Co-Founder at Barr Group. Here at Barr Group, our goal is to help as many engineers as possible build safer, more reliable, and more secure embedded systems. That is, we are in the quality business.
Barr Group is an independent consulting firm, specializing in embedded systems and software. We regularly consult with numerous companies in many industries on software process improvement and embedded systems (re-)architecture for systems and software. As well we take on design and development work, where we participate in the software and sometimes also electrical and mechanical design of systems. As well, we train engineers in best practices through webinars like this, through public training events and also at private companies. And finally, we sometimes, our expert sometimes testify before judges and juries about issues relating to embedded systems and embedded software.
Slide 3: Upcoming Public Training
As is typical at Barr Group we have some upcoming public training courses. The full list of upcoming topics, dates, and locations can be found on the Training Calendar page of our website.
Slide 4: Webinar Presenter
In a moment I am going to turn this webinar over to James Grenning, James is a well-known speaker and presenter and coach on the subject of Test-Driven Development, specifically targeted at embedded systems which, as you understand surely even before this webinar, presents some unique challenges for doing Test-Driven Development or TDD. And there is no-one like James who understands this better and that’s why we have him on our team!
Slide 5: Test-Driven Development for Embedded C
Hello, this is James Grenning. Thank you for joining us today to hear about Test-Driven Development. There is a lot to Test-Driven Development and in this webinar we will only scratch the surface. We are going to start with one of the more obvious benefits to Test-Driven Development, and that benefit is Defect Prevention. To give yourself a fighting chance to prevent defects, first you have to make an admission.
Slide 6: The First Step
Repeat after me: “I am a programmer and I write bugs!”
Slide 7: The Waterfall
Does your company follow a software development life cycle modeled after the waterfall? The waterfall is a nice model, but it’s far from reality. Let me ask you, when are requirements done? Given the timeline on the slide, you might say, May 1st. Now that’s when the requirements were signed off, but were they really done? Rarely, requirements are done when you discontinue support for the product. Consequently, the other phases, Design, Code and Test are also never done. There are feedback loops needed, like it or not. Winston Royce, the first to define waterfall warned that it does not work except for the simpler systems. How many of you are building the simpler systems?
Slide 8: Iterative and Incremental
Instead of waterfall, maybe you follow an Iterative and Incremental approach like Agile or Scrum. The Iterative model is more honest as it attempts to manage the evolving requirements design and code. Too many adoptions of Agile that I have seen in the industry ignore the hard part adopting the engineering practices of extreme programming to support iteration.
Slide 9: Saving Test for the End
No matter what approach you take, and how careful you are, saving testing till the end risks it all. When testing is not continual and automated, problems are sprinkled around the code, only to be discovered later.
Slide 10: Doesn’t End Well
When we get close to release, the problems become all too evident and our efforts often degrade into a chaotic and reactive firefight.
Slide 11: Wisdom From John Gall
Why does this happen? John Gall, in The Systems Bible offered some wisdom to us all. Your program will have bugs, and they will surprise you when you find them.
Slide 12: Celebration Spoiled
CommitStrip.com caught the all-to-frequent release party spoiler in this cartoon. The developers are celebrating the release and the bug comes and interrupts a party. Quickly, one of the programmers Samurais the bug and everyone feels relief.
Slide 13: The Last Bug, Really?
Now they can get back to their celebration after killing the last bug. But look what’s hiding nearby in virtually every corner. Some of the undiscovered bugs looked pretty worried, why does this happen? Is there some law of Software Physics that says, we must write bugs and then kill them later? Do the bug just magically appear from nowhere?
Slide 14: Our Workflow Produces Defects
Of course not, we put the bugs there. It’s because of how we work. When development is followed by test what will we find? Defects, bugs, but unfortunately this approach leaves many bugs still in hiding.
Slide 15: The Physics of Debug Later Programming
The most popular way to program on planet Earth is what I call Debug Later Programming. Let’s look at the physics of this popular technique and its risk.
Slide 16: The Physics of Debug Later Programming
People make mistakes; we are quite good at it. Unfortunately, many of our programming mistakes go unnoticed and are left in the code.
Slide 17: The Physics of Debug Later Programming
Sometime later, the mistake is discovered, maybe you discovered the mistake yourself a few hours after writing the code. Maybe your colleague in system test discovers a bug a few months after you wrote the code. Maybe the customer discovers the defect months or years after the mistake was originally made and the writer of the bug is long forgotten.
Slide 18: The Physics of Debug Later Programming
Now that we know that bug exists, and it’s been deemed important enough to fix. How long does it take to find the cause of the bug? That question is not answerable. This is a huge risk to our product, customer, schedule and reputation.
Slide 19: The Physics of Debug Later Programming
But once you find the cause, it may be very easy and fast to fix. But what if the broken code has been part of the system for a while? What if he had a Boolean function that is exactly wrong? Other working features would depend on it being wrong. Your system is working by accident due to offsetting errors. Fix this bug and you create a batch of new problems to be discovered and debugged later.
Slide 20: Edgar Dijkstra’s Vision
Let’s hear what Dr. Edgar Dijkstra pioneering Computer Scientist, has to say about debugging. “If you want more effective programmers, you will discover that they should not waste their time debugging, they should not introduce the bugs to start with.” Unfortunately, Dr. Dijkstra did not tell us how to not write bugs! He was researching former proofs of code correctness; he discovered no silver bullet. What if there was a way to find an approximation of Dijkstra’s dream of not writing bugs in the first place. Would you be interested in giving up Debug Later Programming? Let’s look at the case study.
Slide 21: The Famous Zune Bug
Some of you may recognize this device. It’s a Microsoft Zune, a music player that competed with the iPod. On December 31st, 2008, the Zune had a problem, it did not play music, it became a pocket warmer as its battery quietly drained. For a music player, not working on New Year’s Eve is certainly a category one defect. What else is special about December 31st 2008, besides being the worst day for a music player in the western world to break? It was a last day of leap year.
Slide 22: The API Responsible for the Zune Bug
Many bloggers dug into this bug. The broken function was in the clock driver, converts the number of days, since January 1st, 1980 to a data structure containing the year, the month, the day of the month and the day of the week, except for the last day of the leap year where it never returns.
Slide 23: The Function Responsible for the Zune Bug
I downloaded the driver and inspected it. I concluded along with many bloggers who said that the cause the defect was this line of code.
Slide 24: It Only Takes One Line of Code
Was the defect cause by a missed requirement? No, the programmer knew leap years have 366 days. How do I know the programmer’s mind? The evidence, Your Honor, is in the code before you. About 9 out of 10 articles offered that if the conditional that specified greater than or equal rather than just greater than, there would have been no Zune bug. I came to the same conclusion, but I have come to not completely trust myself in these matters, so I wrote a test.
Slide 25: The Test That Could Have Prevented the Zune Bug
First, I had to write a function to pretend to be the clock chip and produce the right number of days for the 366th day of 2008, the last day of leap year. I ran the test. My test run never returned, just as expected. I applied the fix, but much to my surprise the test failed. The code under test reported that it was Tuesday, 2009, December 0. Now, good news is, we just have a category two defect for a music player that also doubles as alarm clock.
Slide 26: Defect Not Prevented
One test could have prevented the Zune bug!
Slide 27: Bugs Can Hide Anywhere, What Has To Be Tested?
The programmer knew about the leap year requirement, they just got the code wrong. They did not choose to make a mistake there. Bugs can hide anywhere, so what do we need to test?
Slide 28: We Need to Test
We need to test everything. That sure sounds like a lot to test.
Slide 29: How Many Tests Are Needed To Test These Modules Together?
Don’t worry too much ... yet. Let’s look at a couple simple models. If you had three modules that each have 10 paths through them and you chose only to test these modules together, how many test cases would you need? The math is not difficult but the answer is discouraging, you need a thousand test for the simple system, forget it. We are not writing a thousand tests for this unless there are lives on the line.
Slide 30: How Many Tests Are Needed To Fully Unit Test These Three Modules
Is there something else we can do? Yes, unit testing. If you unit test each of these modules, you will need 10 test for each module and a handful of integration test. For the skilled engineer, this is probably less than a day’s work. A modular system can be thoroughly unit tested.
Slide 31: Integration Tests Are Needed
Great news, the code compiled and linked, the APIs are compatible, the unit test pass, ship it. Wait, not so fast. Just because you cannot practically do thorough testing, of the system as a whole, does not mean you do not need integration of load test. You need a subset of these tests to make sure that the parts of the system are interacting properly. So how do we get started?
Slide 32: The First Step To Recovery
Repeat after me, I am a programmer, and I write bugs. We need to accept that we are responsible for the bugs we produce and look for way to improve. Maybe we have to stop doing Debug Later Programming, what should we do instead? Test-Driven Development, also known as TDD.
Slide 33: Test-Driven Development Micro-Cycle
In Test-Driven Development, programming is done in a tight feedback loop. Start by writing a test for a small behavior you want to add to the code. Just one test not all the tests, just one.
Slide 34: Test-Driven Development Micro-Cycle
In the early stages, the test calls a function that does not exists, the test should fail the build. If it does built it means that the function name is already declared and implemented somewhere. But in the early cycles, you would first expect an undeclared function error. Look at the compiler error; make sure it’s the one you expected.
Slide 35:Test-Driven Development Micro-Cycle
Add just enough code to satisfy the compiler and then the linker, rigging the function to fail the test. By writing code that will fail the test first, we assure that the test can tell when the production code misbehaves.
Slide 36: Test-Driven Development Micro-Cycle
Once you have seen the test fail, make it pass, but don’t write the expected full implementation of the function, write just enough code to assure that the current implementation will pass the new test and pass all the prior tests. You are moving the code, one small step closer to being done. One defined and test behavior at a time. It’s normal at this point to have hard-coded return values and partial implementations. Writing just enough code to pass the test, means the implementation will be incomplete until all the tests have been written. By the way, people new to this are going to find this disturbing.
Slide 37: Test-Driven Development Micro-Cycle
After several TDD cycles are complete, you may think of a better way to solve the problem, you might want to change your name of some variables or functions or remove some duplication. You may discover a flaw in your approach, the tests are there to help. You can safely refactor your code because you have test to finding each behavior. Refactoring is how Test-Driven Developers keep code clean for long and useful life.
Slide 38: Test-Driven Development Micro-Cycle
You continue test driving, one behavior at a time. Cleaning the code as you go and once you can’t think of any more tests, you are done. Not only does TDD help you get the code to working in the first place, it produces a regression test suite to support you or someone else when you find that code needed to be change in the future. It provides a document defining each behavior. Making sure that code is correct is so important; we test every line, every branch, every conditional. Making sure, code is designed well and understood and changed is so important. We make design part of everyday. We make it safe to refactor and we do refactor.
Slide 39: Development and Test Are Done In Parallel
You can see that the Test-Driven Developer does not do all the test and then the code, as the name might imply. We do test and code in parallel.
Slide 40: Development and Test Are Done In Parallel
Each TDD cycle we add one small behavior. When all the tests are written, the code is done, at least for now.
Slide 41: The Physics of Test-Driven Development
Let’s look at the physics of Test-Driven Development and how it is radically different from Debug Later Programming. And my apologies to any physicists out there for taking liberty with your science.
Slide 42: The Physics of Test-Driven Development
As humans we make mistakes, we are really good at this.
Slide 43: The Physics of Test-Driven Development
But with the small steps of TDD and the immediate feedback of the Unit Test, the mistake is right in your face, not hidden like the bugs in the comic we saw earlier.
Slide 44: The Physics of Test-Driven Development
Usually the mistake is obvious, we can find the mistake quickly. Why? It’s in the code you just changed in the last few minutes. It’s fresh in your mind.
Slide 45: The Physics of Test-Driven Development
So fix it, mistake corrected, defect prevented.
But sometimes your new test passes and several previous tests fail, warning you about a previously defined behavior that’s now broken. You likely overlooked some detail that’s not currently foremost in your mind, but it really matters. Back your change out preserving existing behavior and rethink the problem. When this happens to you, that’s when you really get hooked on test-driven.
Before I started doing TDD, I thought I was good at programming. Now I realize that mistakes are the norm and not the exception. I am a programmer and I write bugs, though not so many anymore.
Slide 46: Test-Driving a Circular Buffer
Let’s look at an example. You are all likely familiar with the Circular Buffer or a FIFO: an array of integers with a couple of indexes to manage where to put new integers and where to get all values from, so we can get values in our First-In First-Out order.
Slide 47: Envision A Solution
You could envision an implementation with the data structure holding couple of indexes, maybe a length, maybe some other things and a pointer to an array, to hold the values in.
Slide 48: Make a Test List
Before writing the first test case, we could make a list of what to test. We can anticipate the need for tests around various special cases, like tests around the empty case, tests around the full case, making sure that we get items in First-In and First-Out order.
What do we do about some of the error cases like putting to a full buffer and getting from an empty? Did wrap-around really work?
We don’t have to worry about completely being exhaustive when we are looking at this list of tests because when we get into the details, we will find the tests we didn’t think of while we were brainstorming here.
Slide 49: TDD State Machine
There is a very regular pattern that we are going to follow while we are test driving. And this pattern helps us to free our brain to problem solve, and to better manage some of the details we are working with. So when you start, you choose a test, you write the test. The test probably won’t compile, make it compile. Then the test probably won’t link, make it link. But make it link such that the implementation is wrong and will fail. Watch it fail, then make it pass. And once you make it pass, after you have done a few iterations through here maybe you are going to want to clean up. But right now it’s our first iteration. It won’t matter. Pick another test. Keep going around this loop.
When you can’t think of any more tests, you are done.
Slide 50: CPPUTEST Test_Group
To test drive, you will need a test harness. You can build your own or you can choose one of many open source test harnesses available. You don’t need to buy expensive test tools to do TDD. Here is CPPUTest. I am one of the authors of CPPUTest. Now, we built it to do TDD. It uses C++ but the C++ is hidden in macros and works well for unit testing both C and C++.
Tests are organized into groups in this page, you can see there is a test group and in parenthesis its called (CircularBuffer), each test is part of a group, we will get a fresh copy of the CircularBuffer because setup is called before every test case and teardown is called after every test case. In addition, the variable buffer will be available to every test case.
Slide 51: The First Test CPPUTEST Test Case
Let’s look at a test case. See the macro test and it has two parameters: CircularBuffer which is the name of the test group and empty_after_creation, that’s the name of the test case. In the body of the test we are checking a condition; we are checking that the return result for CircularBuffer_ IsEmpty comes back true, this is what we would expect after creating a CircularBuffer, that it would be empty.
From this point we are going to have to go through a few steps where we will get a compiler error so we go add the signature up to the header file, then we will get a linker error in which case we will go ahead in implementation but we will rig the implementation to return false, so that this test will fail. And once we have seen that failed test, then we will go and make this change.
Slide 52: The Simplest Implementation to Pass All the Written Tests
To make the test pass, we don’t go, put the final implementation that we can imagine in the code now. What we do is we hardcode it to return the value true for the test cases we have, that’s the appropriate implementation. The fact that we can get away with hardcoding a return result needs we don’t have enough test cases so what’s the next test case we will write?
Slide 53: Write the Next Text to Drive Out the Hardcoded Return Result
Let’s put something in the buffer and that it won’t be empty anymore. This gives us two things to do, we are going to have to define an API for put and we are going to have to evolve the implementation a bit to, IsEmpty to make it work for both test cases. Hardcoded return result won’t work anymore, so we are creating a new API, we are going to go through the compile link fail stage for that and then we are going to advance the implementation a little bit towards the end that we envision for this function.
Slide 54: The Simplest Implementation to Pass All the Tests
We have in mind that we can use the input index and the output index as a way to distinguish empty from not empty. So if the input index and the output index, both point to the same location, the CircularBuffer would be empty. To do a put, we would advance the index. We will need to do some other things, we will need to store a value etc., but we are not even getting a value at, so we don’t need to store anything. And likewise, well, input index equals 0 that will be true for the test cases we have right now, later we will need to make the conditional more comprehensive, but right now, I think this will work for the test cases that we have.
Slide 55: Grow Your Code One Behavior at a Time
We know IsEmpty is incomplete, but close to being done. If we test the transition back to empty, we are covering another boundary case and defining a new API, CircularBuffer_Get. Let’s drive our CircularBuffer another step closer to being complete by implementing the empty after removing last item test.
Slide 56: Grow Your Code One Behavior at a Time
After going through the header file and link stage and watching the test fail, our implementation that passes the test looks like this, CircularBuffer_Get increments in outdex, number of the output index that we have just added to the data structure, returns a (-1) because we have to return something to keep their compiler happy. We didn’t have to do any change to put and IsEmpty now we are doing the comparison to make sure that we are defining empty to be the condition on both the input index and the output index to pointed to the same cell, that function might be done now, even though putting together are not done, IsEmpty might be done depending on the approach that we are taking for CircularBuffer.
Slide 57: Repeat Until Done
We continue in small verifiable steps, one behavior at a time, creating an executable specification of all the behaviors of the CircularBuffer. And as we advance our design, if we discover something is not quite with our implementation, if we have a wrong idea of what might work, we have the existing behaviors, well defined, executable; the cost of retesting is zero, that I can run as I restructure the inner workings of my CircularBuffer.
Slide 58: One Test at a Time is a Fundamentally Different Way
I expect that looks really strange to most of you watching and listening to this. What you are used to, Debug Later Programming is having some code without a new feature, envisioning a design that would handle that feature, making all those changes all at once and then spending time, trying to figure out what went wrong and actually ending up somewhere different than you envisioned.
With Test-Driven Development, what we do is we try to keep the code working all the time. We are looking at programming differently. We are adding one behavior at a time rather than writing one file or writing one function or adding one data structure. We are thinking about one behavior at a time; we are establishing cause and effect. I believe it’s a very engineering focused way of working. What problem am I going to solve? Well, my problem is I don’t have a CircularBuffer, they can tell if it’s empty, well I can do that. Now I can’t tell if it knows when it’s not empty, well if I put something and it wouldn’t be empty, one behavior at a time, I grow the behavior of the code undertest, of the code under development.
Slide 59: You Can’t Do It All At Once
John Gall, in his wisdom told us that “A complex system that works is invariably found to have evolved from a simple system that worked.” I read that quote first in the early 90s in the context of Object-Oriented Design. I consider TDD is built on that observation. In TDD we evolve code, one behavior at a time.
Slide 60: A Skill To Hone
People new to TDD think it is odd and dangerous, they see all these procrastination going on. Procrastination is a skill to be honed in practice by Test-Driven Developers. Procrastination is a survival technique; it’s gotten a bad name from lazy people using it. I have come to see that procrastination helps Test-Driven Developers manage complexity.
Slide 61: Managing Complexity
It seems that there is psychological research to back that up. I have been reading a book called Thinking, Fast and Slow. The author explains that humans can only manage so much complexity. We manage complexity by holding most variables constant in changing one at a time. We focus best on one problem at a time. How many details is a human mind good at keeping in the foreground? Well not so many. How many details must be correct for CircularBuffer to be correct? By my account at least a dozen.
Debug Later Programming requires you to keep a lot of balls in the air. TDD limits number of balls in the air to just a couple. Tests become part of your long-term memory, defining what the code is supposed to or what the code must do. These requirements are checked every time you make any change to the code automatically.
Slide 62: How I Discovered TDD
I would like to tell you about how I discovered Test-Driven Development? In 1999, I attended the first Extreme Programming Immersion, a five day training class where I met Kent Beck, the author of Extreme Programming Explained. Kent was performing a demo of test-first programming as TDD used to be called. The demo was in Java using the JUnit test harness. At first it looked kind of crazy, but as he proceeded, it struck me, this is really important.
Slide 63: Target Hardware Bottleneck
I had 20 years of experience when I saw Kent’s demo. Every product development effort met waiting for hardware to run my code. Once we had hardware, we chased down the bugs. We tried everything to get that code in shape before the hardware arrived. And virtually every case, extreme heroics were needed to make the date. And in too many cases, extreme heroics were not enough.
As I watched Kent, it struck me, code can be run in a test harness. In an embedded systems development, it’s a norm to only run code in the embedded system. It is also not uncommon to be debugging the hardware platform along with the code right up to the ship date. Integration never goes smoothly. Why can’t embedded programmers test code in a test harness independent of the target platform? Well, of course they can, maybe not all the code but a lot of it. And then we would not have to wait for the execution environment. We could get code working in advanced of hardware delivery. We can avoid the target hardware bottleneck.
Slide 64: TDD for Embedded Uses Dual Targeting!
You probably never knew what Homer’s Doh was all about, frustration from debugging on the hardware. Let’s look at how we can separate production code from its platform dependencies and tested effectively in the unit test harness.
Slide 65: Work In The TDD Fast Feedback Environment
To avoid debugging on hardware, move some of your code off-target, and then you can do TDD in a fast feedback unit test environment. You should be able to do, an incremental unit test build in only a few seconds. This really helps you keep your mind on the programming problem.
Slide 66: Periodically (or on check-in) Run a Target Build Too
When you are satisfied with your code working off target, run the target build and make sure that you have written code that’s compatible with the target compiler.
Slide 67: Run Test in Eval Hardware
You can’t setup a test build for Eval hardware that uses the same processor as the target hardware. With an Eval system you can relieve memory constraints and possibly have a more reliable execution environment that is close to your target environment. And this lets you find portability problems with your code.
Slide 68: Run Test in the Target
Depending on your target platform, it may be possible for you to run the tests in the target hardware. And if this is the case you can also start to write some test that exercise your actual hardware.
Slide 69: See If the System Works
Finally, you have to see if your product works, you are likely to find integration problems and timing problems, but you probably won’t find many annoying defects making it into the target. You have prevented those defects from seeing the light of day.
Slide 70: Each Stage Detects Different Kinds Of Problems
Each stage finds different kinds of problems, in Stage 1, when you are doing TDD, you are finding logic, interface, boundary conditions, design, and modularity problems. Stage 2, you are finding compiler compatibility problems. Stage 3, you are finding portability problems. Stage 4, same as Stage 3 as well as you can start to use test cases to explore the functioning of your hardware. And finally, Stage 5, you are finding out, did I get the features right? Do all the pieces fit together?
Slide 71: Automate Builds With CI Server
You may think that this is a lot of extra work, it does take some time to setup, and it does help you go faster, it relives tedium and boredom and makes sure you detect problems quickly.
Stages 2 through 4 can be automated through applying continuous integration servers, many of you are probably using CI servers like Jenkins and CruiseControl. These handy tools are direct descendants of Extreme Programming, and the practice continuous integration.
Slide 72: Continuous Integration For Embedded
Let’s look at what CI means to embedded programmers.
Slide 73: Developer Works in the TDD Fast Feedback Environment
The developer programs in the fast feedback environment of TDD and gets all there unit test to pass.
Slide 74: Check In Their Work When Tests Are Passing
Developers check in their working code.
Slide 75: Continuous Integration Server
The CI system monitors your source code repository and what it notices it changes, your checks out the work, does its own local builds to make sure that software is rebuildable and runs a unit tests that the developers have created.
Slide 76: Test Management System
After the off target tests, have been successfully run, the continuous integration server, builds for the target or builds for Eval board and maybe builds for simulator and runs a code there. If any problems are detected, the developers that contributed to that build are notified. Problems don’t get to stay in the system for very long, they are detected right away which is the easiest time for a developer to find what they did wrong.
Slide 77: Test-Driven Developer Automate
TDD developers, automate tedious and error prone steps. You can see TDD is following in that same principle. Retesting code is boring although writing tests is interesting and being able to rerun all the test you wrote before is very rewarding when they help you find errors and help you keep your code working. So, enough about that continuous integration, let’s get back to another example of TDD in an embedded system.
Slide 78: Testing Code with Dependencies
The CircularBuffer is a nice example to get started with because people understand it, and it’s small, but what about real code? Real code has dependencies; well CircularBuffer is real code that most of the code that really matters to us has dependencies. Look at this code, what if we had a home automation system with a Light Scheduler and every minute the Light Scheduler is going to get a callback from your real-time operating system to wake up and looking at schedule. And if there is anything on schedule that needs to be done, the Light Scheduler would either turn on or turn off a light. And you can see here this network of objects will need a real-time operating system, will need some hardware to run this.
Slide 79: Managing Dependencies with Interfaces
The test-driven approach makes us look at our work little bit differently. Instead of thinking the Light Scheduler, is using a real-time operating system and using a Light Controller, let’s think of it as using interfaces, an interface to a time service, an interface to a Light Controller. Let’s break the dependencies on the hardware so that we can actually test this code somewhere else.
Slide 80: Testing With Test-Doubles (Spies and Fakes)
If we live the knowledge of the Light Scheduler to just the interfaces for the time service in the Light Controller, we can do something interesting. We can swap the hardware and operating system dependencies for a Light Controller Spy in a Fake Time Service. For a Fake Time Service, it can be whatever time we want. And for Light Controller Spy we don’t need any hardware for that, the spy is just going to remember what the Light Scheduler told us. We can setup various scenarios of things scheduled and times that occur and see what the scheduler does. And get this all ready without hardware. Well why not deploy it and run it in the hardware as well. But we would like to work in this fast feedback environment of Test-Driven Development, so we can make sure that this important code works.
Slide 81: Test Each Behavior Independent of HW and OS
We could write a test like this, a test for the LighScheduler that assures that the light does not get turn down at the wrong time. You have got the API, LightScheduler_AddTurnOn, so we are going to add a TurnOn event for Light number 3 everyday at 1200. We are going to fake out the day in the minute to not be the right time, not be 1200. And then we are going to simulate the callback to the scheduler and we are going to make sure that the spy reports it nothing has happened.
Slide 82: Test Each Behavior Independent of HW and OS
The next test, LightSchedulaer, light_turns_on_at_the_right_time is almost the same as the previous test except you will notice that the minute setting in the FakeTimeService says to 1200th minute. So this is the scheduled time and when we simulate the callback to the LightScheduler, you will see that when we ask the spy, it tells us that light 3 is on.
Slide 83: I Don’t Like Testing
An alternative to proactively test-driving features is to manually test some after the code is written. This may be interesting at first but quickly becomes boring and repetitive and error prone. Maybe we can find somebody else to do it to make sure that our code works. I don’t like testing, that seems like something odd for the author of Test-Driven Development for Embedded you to say, I don’t like testing. Testing is a pain, Test-Driven Development isn’t.
Slide 84: TDD is Fun!
TDD is Fun!, it’s a challenge, it’s like navigating a maze, if you like programming you will like TDD, you are writing code. It helps me see my mistakes, it helps me learn it structured problem solving, there is a lot to like there.
Slide 85: Adding tests after is not so fun
Adding tests after that’s not so much fun, especially after manually testing code first, either way it feels like extra work. It’s harder to be thorough, you get this ID, your code is well tested before it is.
Slide 86: TDD Cannot Be Dictated
TDD cannot be dictated to a development team. It is voluntary, it is a skill that must be learned, practice and mastered. But the business can and should dictate test automation.
Slide 87: Companies Are Organized As If This Was True
Let me go through a simple model. Your company is probably organized as if this were true. The effort to test is proportional to the effort to develop. How can I say that? Well, do you have a fixed number of developers and a fixed number of testers? Or do you allocate fixed proportions of development and test effort that are fairly consistent year after year. This is true in most companies I visit.
Slide 88: But Systems Don’t Like Being Changed
But systems don’t like to be changed, when you add a new set of features, you must retest all prior working features. But who does that, when they rely on manual test.
Slide 89: With Each Feature All Prior Features Need To be Retested
When you add the third set of features you need to retest the first two.
Slide 90: With Each Feature All Prior Features Need To be Retested
And so on.
Slide 91: With Each Feature All Prior Features Need To be Retested
Don’t worry about the proportions, I think I am under-representing the effort growth by assuming the retest time is less than the original test time. The model also shows growth is linear, let’s say that’s optimistic. This is a conservative model.
Slide 92: Unsustainable Growth of Manual Test Effort
Manual test is unsustainable. The available test effort is fixed while the needed effort continues to grow. What happens next, we cut corners and test what needs to be tested.
Slide 93: The Untested Code Gap
But more test effort was needed and this produces what I call the untested code gap.
Slide 94: Lightning Strikes
In the untested code gap, risk accumulates and at the worst possible time probably right before release, lightening strikes and it all goes up in flames. A firefighting ensues a death march. How do I conquer the untested code gap? Test automation. Automation keeps the cost of retest low. We can’t make it zero but we can get a lot closer than we are. TDD is an effective way to produce unit test, in the skill to automate the other test you are going to need.
Slide 95: Defect Prevention
Repeat after me, "I am a programmer and I write bugs!"
If we want to prevent defects, we need to stop practicing debug later programming and rely on manual test. I hope you can see the possibilities of TDD and how it could help you detect your programming mistakes before they become bugs. There is more to do besides TDD, TDD won’t prevent all defects but any serious defect prevention effort must include Test Drive Development.
Slide 96: There is a lot more to Test Driven Development
There is a lot more to Test Driven Development than we can cover in one short webinar. We encourage you to learn more. You can read my book, you can attend one of our training courses.
Question & Answer
Thank you for listening, and now we have time for a few questions and maybe some answers.
Moderator: Thank you, James and Mike, for your informative presentation. Now we will start the Question & Answer session. Our presenters will review the questions as they come in and answer the questions that come up most often. Keep in mind the we are hosting hundreds of participants today so not every questions can be answered. Feel free to start submitting your questions now.
Q: How do you know you are not going to forget an important test?
Well the best you can do in writing code is, write the code you think you need. This programmer thought that the last day of Leap year was special. They wrote code for the last day of Leap year being special. They just didn’t bother to test it for some reason or they didn’t test it in this case.
Now before you would ever write that conditional in an if statement you would have said, well you know what the 366 day of a Leap year is important. So I better write a test that’s about the 366 day of Leap year. Okay, so I hope that helps, basically what, it’s going to be difficult to think of all the tests, but I might add to what I said there you know, if you know what your code is supposed to do, you should be able to define the test, it’s a matter of growing the skills to be able to think of those tests as you are writing the code. You might think, actually I have come to think that writing the code before the test might actually be kind of silly because, how do you know what code to write if you don’t know what your code is supposed to do? So and then also how do we make sure we don’t forget test, you might have reviews, you might choose another one of your extreme programming practices for your programming to a company test-driven development to help you learn it faster and help you grow the quality of your code and help make sure that you have test everything.
Q: How do you test tasks? For example, we use free RTOS and our tasks that run in an infinite loop with queue and semaphores to synchronize the time tasks.
Well you have got a couple of things that make your code kind of untestable there. One is an infinite loop, so if you really have to have an infinite loop—and I understand in embedded systems they show up a lot—I wouldn’t do anything in that loop except call the function you are going to delegate to.
And in that function or maybe in the infinite loop you might wait for something in a queue and take that thing and then go process it somewhere else or we will just kind of explore that, I will just take one of these, I am going to skip the semaphore thing now I do have a series, a three-part article series on my blog about dealing with semaphores and such, but first I might want to separate my loop from the body of the loop and then the next thing I am going to do is probably provide a test stub or the Queue Read function and the Queue Write function.
So my test case, could tell the fake version of the Queue, some people this might be a Mock Queue there is a bit of vocabulary which you learn when I teach you TTD or a Fake Queue where I can just say if someone says, “get me something from a queue,” give them this. So we can give you the message, you could give your code the message it needs, then it can go off and process it and say the result of that message being processed is it sends some other message. You could have a fake as send function that your test case could then interrogate the message that was sent and you could make sure that everything in the test case, you could check to make sure everything is correct. Okay, so that’s one approach to that.
Q: Resource recommendations for a test batch?
Okay. So I use CPPUtest. Now I am completely comfortable with C and C++ so it’s not a problem for me. But CPPUtest, we designed with C programmers in mind and for most C programmers it’s a fine test and as to use, you don’t really have to learn it in the C++ but you know after a while you might actually learn some by osmosis, so I like CPPUtest and not CPPUnits, please avoid CPPUnits because it requires you to install test cases individually and CPPUtest automatically installs all the test cases.
Another test harness which I use in the book is Unity. And Unity is a good test harness, it’s a C only test harness, but it also relies on Ruby scripts to create the test runners and such. Google Test is another option, although the people like Google are really building test harness for embedded systems developers, embedded C developers. So in CPPUtest we tried to make it easy to port as easy as C++ could be this to port, by using very few of the C++ features in our implementation.
Q: I am developing an 8-bit PIC. Is a TTD method relevant there?
It requires some abstraction, probably HAL, so there is not much resources for that in an 8-bit-er so I would suspect that you wouldn’t be able to run your test very well in the 8-bit-er, but are you writing your code in C? And I am just going to rhetorically answer that for you; yeah I am writing my code in C. So your challenge will be to not rely on all these special things that you pick, compiler does for you.
Try to isolate where you take advantage of the special features of the PIC and try to isolate the code that could be plain old C code. So yeah you do have a little bit of a HAL, little bit of a hardware abstraction layer, but I do have a GitHub example, I forget exactly what the GitHub account is called? Well it’s jwgrenning that is my GitHub account, you can look at some of the examples there, I do have an example of some very primitive and compiler specific code of it, you could look at, how I dealt with that there.
Q: How do you test specific embedded things like DMAs and drivers?
Well in the training class we are going to go into test driving a driver. You can take Test Driven Development within one instruction of the hardware, basically if we intercept calls to I/O read and I/O write we can verify drivers, do what we think they are supposed to do. Embedded DMA, yeah so that’s going to be, are you programming the DMA properly? This might be something that’s not really a direct hit for TTD because you probably get the working once right and then leave it alone forever. If you have drivers that you do regular maintenance on or evolve, or have conditional logic in them, then test driving is helpful approach.
Now let’s see, yeah I am just giving you a few words here, if you are really interested in getting into some of these details, come into my class in a couple of weeks it would be great, love to see you there. Also you can find some help in my book.
Let’s see I have got a couple of other, I know I had a question earlier and I am going to go in a slide that happens to be helpful with that. So I am going to show this slide, let me just set up my sharing again. Someone asked me and let me find the wording again for that.
Q: What types of documentation artifacts are generated from Test-Driven Development and how might these be used to satisfy premarket submission requirements for FDA?
Alright, so I got to say, I don’t know about premarket submission requirements for FDA, but I suspect that you do. I also know from doing ISO9001 certifications and such in the past that essentially we define our own process and made it as hard on ourselves as we wanted to. I believe in the FDA there are requirements, but they are rather general and it’s up to you to say, how you meet them. So you could certainly write your requirements for the FDA around Test-Driven Development. But now let’s look at the first part of that question is what documentation and artifacts are generated and I can bring you this picture, hopefully you can all see that list of test cases.
Now that the test runner that I use anyway CPPUtest, I can have it report every test case as it runs and if you are careful about how you name the test cases, for instance TEST, (CircularBuffer, put_to_full_does_not_damage_contents). Well here’s a scenario where we are going to have a full CircularBuffer and we are going to put something into it and make sure that nothing was hurt. So we tested that, every time we run the test runner, we test that. Every time we make any change to the CircularBuffer, we test that.
CircularBuffer is_not_full_after_creation, down here in the middle, every time we make another change we assure that each one of these behavioral conditions of this module are satisfied. So that’s one thing, this could be a document automatically generated; you know software developers tell me that they don’t love writing documents. So what if you got something that you did love writing code and you could generate a document from the code?
This might be a double win. I could have fun writing this in code, I could get my maybe a triple win. I can get my code to work and keep it working and I could not have to write that document that says what I tested, because I can just generate the report of what I tested, sounds kind of cool.
Also I can write my tests so that they tell me a story. They tell me what I am testing. So for instance, for my Light Scheduler, (LightScheduler, no_lights_controlled_when_it’s_not_the _scheduled_time). So if I had a LightSchduler and I had to schedule for light 3 everyday at the 1200th minute of the day and the clock transition to Sunday at 11:99, the lights would not be changed because the time is not right.
Second test, (LightScheduler, light_turns_on_at_the_scheduled_time_for_everyday) and if I had a scheduler turned on, light 3 everyday at 1200 and it happened to transition to Sunday at 1200 we would expect light 3 to be on. So I can search for right test, it could double, if I am careful about how I refactor my test and how I have them describe what’s happening, I can have those tests be part of my documentation.
Q: It seems like there will be a lot of code, a lot of test code to maintain. Does this end up being a problem?
Well, actually the test code it’s very interesting, you do have to be very careful with your test code. You must structure it so that it reads well, so it’s low in duplication, so it’s structured well, just like you would have to with production code, equally or maybe more important. So the test code must be well structured. If you allow duplication in your test code and you come across a change that might require you to go visit 25 test cases, you are going to think seriously about throwing those test cases out. But if you carefully structure your test cases and this can be done as you are learning how to do this, and as they change you see how they change and you structure them for the kinds of changes you see. You can, the test cases become a really important resource in keeping your code going.
The reason that Ward Cunningham and Kent Beck started doing this test first thing is because they needed it to survive. They were finding that when they produced code that someone asked for, when they showed it to them, one of the first reaction, was often gees, thanks for giving me what I asked for, but I want something a little bit different. And so they are having to make changes. And as you make changes the code, because it’s so fragile, you are likely to break something. And then you are stuck with a lot of manual retest. So Kent and Ward two extremely brilliant men to come up with this, I just don’t know how they thought outside the normal Debug Later Programming box, to come up with this, they decided that automating of test was their only thing that they could do to survive. So, yes, there will be a code to maintain, but it’s going to help you evolve your product.
Q: From your book, is something like C Unity CMock automatically generating mocks from headers for CPPUtest?
CPPUtest has a slightly different approach to mocking than Unity. Unity is very powerful, it’s a great thing. I prefer the CPPUtest approach because it kind of makes you a little bit more aware of the dependency structure rather than the automatic mock generation that Unity does. You could have some problems in your dependencies and because it makes it easy to deal with problem dependencies, you know you could end up doing the wrong thing, I have known some people do the TDD get stuck in there. I know experienced TDD people won’t have that problem to CMock that new people might to mock.
Q: Please comment on testing Legacy Code and also third party code, such as an RTOS.
Well let’s see, in general, I will just make a comment about the RTOS, oh and while I am still on Unity, someone is asking, I thought there was, Unity was a game development tool, yeah, I think the guys that invented the Unity test harness are kind of, wish that they didn’t call it Unity because of Unity game development environment, but they are not related, even though you could use them together. Okay, so there is no relationship there and if you find the Unity test harness on GitHub I believe that’s handle is embunity on GitHub.
Let’s see, how about legacy code, well, legacy code is a thing that’s going to cause a problem, because it doesn’t have test and if you don’t have tests, it’s very unlikely that your code is structured to be easily tested. So, one of the things that we will talk about in the training class is to approach to dealing with legacy code and the approach that a test-driven developer takes with legacy code is, and I would refer you to Michael Feathers book, Working Effectively with Legacy Code as kind of the one of the good sources to learn about it. My book has a chapter on this as well that I get a lot of the ideas from there.
With legacy code, we have to assume that every line of code is there for a good reason. And so we have to not change the existing code, as much as possible until we can get some tests around it, so we can be sure what that code is doing. So, if I have to change a certain function and what I am going to want to do is call it from a test harness, that’s going to be a challenge, because pulling code that’s never been in a test harness before will be hard. I have a procedure that I follow called Crash to Pass. As you are working through the recipe to get legacy code under test, with a very careful approach, respecting the code that’s there, trying to add test to it, trying to tees apart some of the dependencies so that we can put tests around it make sure that we know what the code is doing before we start changing it.
To me I have come to appreciate that I am not qualified to change existing code unless I can write test for it, because I just find, I used to think I was good at programming, here’s true confessions time. Until I started doing Test-Driven Development, 20 years into my career I was probably not too bad it by then. I thought I was pretty darn good, started doing Test-Driven Development and I discovered wow, you sure mistakes a lot. Test-Driven puts them in your face, when you start to experience that you are going to be really careful as you modify code that doesn’t have test.
So, what about third party code like an RTOS? Well if I am testing my code that uses the RTOS, I am going to create a subversion of the RTOS. By the way there are some tools that are really handy to creating a sub like that. And certainly CPPUtest, mocking framework is helpful and the fake function framework is helpful, so there are some things you can do. If I want to make sure I know how the RTOS works, well I could write some test to interact with the RTOS, but that’s more of, you could have some of those but probably you are going to generally trust the third party code is doing what it’s supposed to do.
I am going to add a caveat to that, when I am learning something new, learning to use something new, I find the easiest place for me to learn what it does is by to start writing test for the thing that I want to use. So, if it was a real-time operating system I might actually start to create a thread from the test case and look for, well how do I wait for the thread to terminate? I might make a test case that created a thread that immediately terminates. And then I might create another test which created a thread and then had it, wait on the semaphore. And then that second test I would probably create a semaphore and trigger it, so the code would be released and come back and I’ll wait for that code. So I could write some tests like this, one thing that you would want to do is not ever let a test case end without the thread that it might have created terminate, okay, bad things will happen and I just guarantee it.
Looks like we are down to like one minute, there is couple of other good questions here, but I think my voice is pretty washed out at this point. And I want to thank you again all for coming, so I hope to be in touch with you in the future. If I see some questions on Twitter, I will probably, you know, have some time this afternoon to answer some of those. So, feel free to go ping me on Twitter, and I will try to get to those this afternoon while this webinar is all fresh in my mind. And again, thank you very much for attending.
