Software Reliability and the Internet of Things

Posted January 07, 2016

As Internet connectivity advances, the transportation, automotive, medical device, smart grid and other industry sectors have become more dependent on embedded software. But is embedded software up to the required level of reliability?

Automotive, train and aircraft transportation are becoming more dependent on embedded software and thus on embedded software safety and security. So too are the smart grid, industrial control and automation, and access control. As these industries become more dependent on embedded software, they must rely more on embedded software safety and security. A typical automobile will have 50 microprocessors in it. A high-end automobile, perhaps 100. Instead of the concern being bumpers and fuel tanks, now the concern is, “Can the vehicle next to me tamper with my engine control unit while I’m driving?”

Why Embedded Software is Complex

Embedded software tends to be even more complex than typical “desktop” software because it deals with resource-constrained hardware and real-time deadlines that must be met to avoid malfunction in a large variety of environments and by a variety of users. Also, many kinds of people — electrical engineers, computer scientists and sometimes even mechanical engineers or technicians—write embedded software. Some of these people have learned “on the job” and therefore might not apply the appropriate level of engineering rigor that is required for safety-critical software.

Lastly, as teams become larger and more distributed, and schedules and budgets become squeezed, quality becomes more difficult to control. And so we continue to hear news reports about big product recalls. Recent cases were an airbag subcontractor of Japanese car manufacturers and the crash of an Airbus military transporter, most likely caused by a software bug in the engine control.

There are best practices, tons of software tools, proven processes and even certifications that promise to make sure software does what it should do and nothing else. Yet still many opportunities exist for bugs to be introduced. For instance, if the code is maintained by a new developer who isn’t familiar with the code or who hasn’t been trained on the company’s internal development processes and tools, bugs can slip in. Some bugs are introduced when hardware changes, even though the software is unchanged. And some kinds of complex problems, such as race conditions and hardware glitches, are exceptionally difficult to discover without very comprehensive software and systems testing.

Security and Safety: Correlated but Not the Same

There is a correlation between safe software and secure software, but they are not the same concerns. For example, developers of a safety-critical system will apply techniques such as Failure Mode and Effects Analysis (FMEA) and Fault Tree Analysis (FTA) to identify areas of safety concerns, in order to ensure that the product not only operates safely, but fails safely as well. A security engineer has different concerns and will apply techniques such as Threat Modeling to identify security concerns and determine what countermeasures are appropriate.

A system can be relatively safe but insecure, or unsafe but relatively secure, but the best systems are designed to be both safe and secure. For example, an embedded software defect as buffer overflow or undefined behavior can result in both safety hazards and security vulnerabilities. It’s important that developers try to prevent or eliminate these types of problems before addressing more specific security or safety concerns.

Connection and Exposure

When we consider the safety and security traits of embedded software, safety seems to be more mature and well understood compared to security concerns. Thankfully, we don’t frequently hear of planes falling out of the sky, missiles launching themselves, or medical devices killing patients. But every week, it seems, there is a router running embedded Linux being hacked, a smartphone establishing insecure Internet connections, or patient/customer data being exposed by some small Internet-connected (”IoT”) device.

Many areas and industries are exposed, particularly from a security perspective. Devices in many industries are being connected to the Internet, often with no consideration of security. Medical devices, many of which are life-critical, contain private patient data, can receive software upgrades, contain sensitive and very highly guarded intellectual property (IP), and may even contain many “locked” upgrade features that are a “simple hack away” from being unlocked for free. Just think of what an attacker could do to an insecure device.

One of our biggest concerns at Barr Group is products, often low-cost ones, that are being “Internet enabled” without any concern for security. Does that refrigerator, smoke detector, or thermostat really need to be connected to the Internet? Sure, the convenience and “coolness” are nice, but they come at what cost? These are things that need to be considered in our quest for connectivity.

Obstacles to Software Quality

The three biggest obstacles to software quality are cost, schedule, and expertise. Cost and schedule pressures aren’t going away, so it’s our goal—really our mission—to improve software quality by working with companies to improve their skills, their processes, and their tools. We do this through a mixture of consulting and training. Our training classes for embedded systems designers teach very specific skills, best practices, and processes that are focused on preventing defects and improving quality.

The later a defect is discovered, the more expensive it is to fix (the worst case being a product recall), so we focus on techniques and processes that keep bugs out of the embedded software from the outset. As an example, we strongly recommend the use of an effective coding standard, code reviews, and static analysis tools. The net effect is embedded software that ships on schedule and on budget.