Software has become ubiquitous. From microwave ovens to electronic throttle controls, it is now embedded into the fabric of our lives through billions of new products each year. Whether for infringement of intellectual property rights or product liability, when products controlled by software are the subject of litigation, it is imperative to analyze the embedded software, also known as firmware, properly and thoroughly.
This article enumerates six best practices for embedded software source code discovery and the rationale for each. The relative value and importance of the individual practices will vary by type of litigation, so the recommendations are presented in the order that is most readable.
1. Get the Source Code Early
Any serious litigation involving embedded software will require an expert review of the firmware’s human-readable source code.
TIP: The source code should be requested early in the discovery process to streamline the forensic review process, which could result in significant savings in time and money for both experts and litigators.
In our experience, the owners of source code often resist requests for its production. However, procedures limiting access to the source code by only a list of named and pre-approved experts with access under some sort of physical security--such as on a non-networked computer with no removable storage --tend to be agreed upon or ordered by a judge.
2. Ask for the Bug List
At least in cases of alleged product liability, the software development team’s defect database—or “bug list”--should be routinely requested and supplied in discovery.
Very nearly every piece of software ever written has both known and unknown defects. During product development, thorough software teams will maintain a defect database--or “bug list”—to record problems observed in the software as well as track the status of bug resolution. This list of fixed and known bugs—or the company’s lack of such a list—is germane to issues of software quality and provides helpful guidance to experts during the source code review process.
TIP: It is also recommended that a request be made for copies of software design documents, coding standards, build logs and associated tool outputs, testing logs, and other artifacts of the embedded software design and development process.
3. Insist on an Executable
In software parlance, the “executable” program is the binary version of the program that’s actually executed in the product. The machine-readable executable is constructed from a set of human-readable source code files using software build tools such as compilers and linkers.
TIP: It is important to recognize that one set of source code files may be capable of producing multiple executables based on tool configuration and options.
Sometimes it is easy to extract the executable directly from the product for expert examination—in which case the expert should engage in this step. For instance, software running on Microsoft Windows consists of an executable file with the extension .EXE, which is easily extracted. However, the executable programs in most embedded systems are difficult, at best, to extract.
TIP: If the expert is able to extract an executable from one or more exemplars of the product, an automated comparison should always be made between the installed and produced binary files.
Executables can make a large impact in litigation. Though not human-readable, an executable program may provide valuable information to an expert reviewer. For example, one common technique is to extract the human-readable “strings” within the executable. The strings in an executable program include information such as on-screen messages to the user (e.g., “Press the ‘?’ button for help.”). In a copyright infringement case once consulted by Barr Group, several strings in the defendant’s executable helpfully contained a phrase similar to “Copyright Plaintiff”—a simple yet important string that had a major impact on the case.
It may also be possible to reverse engineer or disassemble an executable file into a more human-readable form. Disassembly could be important in cases of alleged patent infringement. For example, in situations that appear to be an infringement of a method claim in the source code, the code in question may not actually be part of the executable in the product as used by customers, but rather, unused code.
4. Reproduce the Development Environment
The dichotomy between source code and executable code and the inability of even most software experts to make sense of binary code can create problems in the factual landscape of litigation. For example, suppose that the source code provided by a defendant was inadvertently incomplete in that it was missing two or three source code files. In complex systems, even an expert reviewer looking at the source code might not know about the absent files. And no expert can find the bugs or infringing code in a missing file.
Fortunately, there is a reliable way for an expert to confirm that all of the source code has been provided—reproduce the software build tools setup and compile the produced source code.
To do this it is necessary to have a copy of the development team’s detailed build settings, such as make files, preprocessor defines, and linker control files. If the build process completes and produces an executable, it is certain the other party has provided a complete copy of the source code.
TIP: Experts may also need to be provided certain third-party header files or libraries utilized by the developers.
If the executable as built matches the executable as produced—or ideally, extracted from the product—bit by binary bit, it is certain that the other party has provided a true and correct version of the source code. Unfortunately, trying to prove this part may take longer than just completing a build. The build could fail to produce the desired proof for a variety of reasons. To get exactly the same output executable, it is necessary to use all of the following:
- Precisely the same version of the compiler, linker, and each other build tool as the original developers
- Precisely the same configuration of each of those tools
- Precisely the same set of build instructions
Even a slight variation in just one of these details will generally produce an executable that doesn’t match the other binary image at all—just as the wrong version of the source code would.
5. Try for the Version Control Repository
Embedded software source code is never created in an instant. All software is developed one layer at a time over a period of months or years in the same way that a bridge and the attached roadways exist in numerous interim configurations during construction. The version control repository for a software program is like a series of time-lapse photos tracking the day-by-day changes in the construction of the bridge. But there is one considerable difference: it is possible to go back to one of those source code snapshots and rebuild the executable of that particular version. This becomes critically important when multiple software versions will be deployed over a number of years.
It may not be possible to gain access to the entire version control repository that holds all of the past versions of a company’s firmware source code through discovery. For example, a judge may only allow the plaintiff to choose one calendar date and to then receive a snapshot of the defendant’s source code from that specific date. If the plaintiff is lucky, evidence of their intellectual property in that specific snapshot will be found. Unfortunately, the observed absence of their intellectual property from that one specific snapshot does not prove that the alleged theft did not happen earlier or later in time.
There are some problems with the examination of an entire version control repository. For complex systems, it may be difficult to make sense of the repository’s structure. Or, if the structure can be understood, performing a thorough review of the major and minor versions of the various source code files may take multiple times longer than reviewing just one moment in time. At first glance, many of those files may appear the same or similar in every version—but subtle differences may be important to making a case. To streamline the review process for high volumes of code with multiple versions, it may be necessary to obtain a chronological report provided by a bug list and/or other production documents which describe the source code at various points in time.
6. Remember the Hardware
Embedded software is always written with the hardware platform in mind. In embedded systems, hardware and software are complexly intertwined and can be one of the most important considerations of all when analyzing firmware. Therefore, the expert must keep the hardware-software interaction in perspective throughout the analysis process for the following reasons:
Reason 1: It is only possible to properly reverse engineer or disassemble an executable program once the specific microprocessor (e.g., Pentium, PowerPC, or ARM) is known.
Reason 2: In embedded systems, only one or more features of the hardware are enabled or active when the hardware is set to a particular configuration.
For instance, consider an embedded system with a network interface, such as an Ethernet jack that is only powered when a cable is mechanically inserted. In this scenario, some or all of the software required to send and receive messages over this network may not be executed until a cable is inserted.
Tip: To perform a thorough analysis of a system, firmware testing should be done on the hardware as configured in exemplars of the units at issue. Therefore, it is useful to ask for hardware during discovery if you are not able to acquire exemplars in other ways.
Reason 3: There are times when a bug is not visible in the software itself. Such a bug may result from a combination of hardware and software behaviors or multi-processor interactions.
For example, one motor control system in a case previously supported by experts from Barr Group had a dangerous race condition. The bug was the result of an unforeseen mismatch between the hardware reaction time and the software reaction time around a sequence of commands to the motor.
Bonus Tip: Hire the Right Experts
The best practices presented here are meant to establish the critical importance of making certain specific requests early in the legal discovery process. They are by no means the only types of analysis that should be performed on the source code. This and other types of technical analysis should be well understood by any expert witness or litigation consultant with the proper background. For cases involving embedded software, an software expert should typically have a degree in electrical engineering, computer engineering, or computer science plus years of relevant experience designing embedded systems and programming in the relevant language(s).
Barr Group's team of electronics and software expert witnesses provide experienced and unbiased source code reviews, expert reports and testimony for product liability, patent infringement, software copyright, and trade secrets litigation involving computer-based technology and software. HIRE AN EXPERT
