Bug fixes, patches, and third-party application programs are all possible in today’s more powerful embedded systems, but you need to plan for them at design time. This article shows you how.
Software upgradability and extensibility are now considered important features for a wide range of embedded systems, particularly those on a network. This may mean anything from a complete replacement of the memory image to an add-on software module developed by a third party. This article will provide an overview of the key hardware and software design issues that you must consider to make upgradability and extensibility possible. In addition, I’ll present a set of guidelines for attracting third-party software developers to your platform.
It is increasingly the case that embedded systems are designed with software upgradability in mind. Version control is often used to track bug-fixes and other changes to the firmware; in-system programmable memory technologies like flash are rapidly supplanting ROM; and a variety of mechanisms are employed for quickly and easily modifying a system’s firmware. These relatively recent changes in embedded system design may be intended merely as conveniences for the developers—it’s a lot easier to reflash a system than it is to remove an EPROM, erase it under a UV lamp, burn the new firmware, and reinsert the chip—but there are long-term benefits as well.
Deployed systems that include flash memory devices and software upgrade mechanisms have lower maintenance costs. If a bug is found in the firmware, sending the customer a binary file and instructions for reflashing the box is far cheaper than having the box shipped back to the factory, its ROM extracted and replaced, and the box returned to the customer’s site (not to mention less downtime and frustration for the customer). And if the deployed system is on any sort of a network, even if it’s only connected a fraction of the time, the customer may not need to be involved in the upgrade process at all.
I will describe three different types of software upgrades that may be desirable in an embedded system—replacements, patches, and extensions—and the architectural considerations you should make when planning for each. In the process of describing the different types of software upgrades, I will develop a complete list of hardware and software requirements for any system that is to be field upgradable. By paying attention to these requirements during your next design phase, you can easily build an embedded system that is both upgradable and extensible.
Software replacements
A software replacement is a complete substitute for the firmware originally shipped within a particular embedded system. Typically, a region of flash memory containing the original firmware is erased and the new program is written into that same region of memory. Of course, the new program may be either smaller or larger than the original, depending on the types of changes made.
Three scenarios illustrate the complete range of uses for a software replacement. In the first scenario, the software replacement mechanism provides a fast and easy way for members of the software development team to modify the system during product development and testing. In the second scenario, a software replacement is used to upgrade a system already in the field to the latest release of the firmware. This may be required if a significant number of bugs (or one really significant bug) have been found since the product was originally deployed. Finally, a third scenario involves the use of the software replacement mechanism to provide customer-specific flavors of the product. In such a use, the hardware inside the box would remain the same, but different versions of the firmware would be used to give the product a unique “personality” for a special customer (or set of customers).
A software replacement is the simplest type of software upgrade to plan for, and all of its architectural requirements are necessary to support patches and extensions as well. The basic requirements for any software upgrade are shown in Figure 1. First, the replaceable part of the firmware must be stored in an in-system programmable memory device. Second, there must be a delivery channel through which the new code can travel to the system. Finally, a special piece of embedded software called a loader must be present to install the new code. Each of these requirements is described in more detail below.
Requirement #1: Flash memory
First, some sort of in-system reprogrammable, nonvolatile memory device must be present into which software upgrades can be placed. This is true for all three types of software add-ons. Flash memory is probably the best technology for these types of applications. The low cost-per-byte and fast read times of flash make it a reasonable alternative to just about any other nonvolatile memory type—including masked ROM, OTP, EPROM, and EEPROM. Of these, only EEPROM is also in-system reprogrammable. So a flash memory, an EEPROM, or a nonvolatile RAM is the first requirement for a system to support software upgrades and extensions of any sort. For the rest of this article, I’ll assume you’ve chosen a flash memory device. It matters little which type of memory you actually select.
Requirement #2: Delivery channel
Equally important is the existence of some channel through which software upgrades and enhancements can reach the deployed system. This channel may be as simple and automatic as a direct network connection back to the manufacturer; or it may be as complex and awkward as the delivery of bits on a floppy disk (via the postal service) and a subsequent download over a PC’s serial port to the target. Whatever the mechanism, some way to get binary data from the manufacturer to the deployed system must exist. Ideally, this delivery channel would not involve a return of the device to the factory. However, if the only type of software changes you will support are replacements and you don’t expect to make any of these unless some major flaw is found, a return of the device to the factory may be acceptable.
In the case of software delivery over a network, the network may be fixed or wireless, bi-directional or broadcast, or always on or only occasionally connected. So it is not unreasonable to consider the delivery of software upgrades from a base station to a cell phone, or from a satellite or cable provider’s head-end equipment to a set-top box. Of course, the features of your particular network (available bandwidth for upgrades, broadcast or bi-directional, and so on) will affect the design of your upgrade mechanisms. If you have the luxury of designing both the network and the embedded device that requires upgrading, you should consider the needs of your software delivery channel carefully before committing to a particular network architecture.
Requirement #3: Loader
A special piece of embedded software called a loader solves two of the problems that arise from the previous pair of requirements. The first problem is that flash memory can only be reprogrammed with the help of the embedded processor. The processor’s address and data buses are used to begin the erase and write cycles of a flash memory device. So there must always be software in the embedded system that is capable of performing these steps. Otherwise, there would be no way of getting a downloaded software replacement, patch, or extension into memory to be executed.
The second problem that the loader solves is that of communication over the delivery channel. How does the embedded system know that a new piece of software should be downloaded into memory? The loader is responsible for handling the target portion of whatever notification mechanism you select. (An authorized host computer handles the other end and provides the upgrade software.) It is also the loader’s job to decide when to look for software upgrades. Immediately after each system reset? Every morning at 3 a.m.? Every 15 minutes? Or only upon receipt of a “software update” message from an authorized host? (Of course, the loader is also responsible for answering the question: which systems are authorized hosts?)
The loader is typically a task that runs at a lower priority than the system software, waking occasionally to check for new software upgrades. This loader wake-up may be periodic (timed) or only in response to a “software update” message from a host computer or human operator. When the loader finds that an upgrade is available and should be loaded into the system, it should halt the rest of the software, overwrite the appropriate portion of the flash, and, if necessary, restart the system.
The loader must, therefore, be separate from the program in the upgradable region of memory. It should never be overwritten—even by a “complete” software replacement. Since it is logically separate, the loader may be located either in a separate ROM or within some other region of the flash memory. However, if the loader is stored in the flash memory, the part of it that performs the actual flash erase and write operations (the flash driver, for example) will have to be copied into RAM for execution. That’s because all of the instructions and data stored in a flash memory device are unreadable during both erase and write operations. The processor can only fetch and execute the instructions that make up the flash driver if they are located in some other memory device at the time.
Because the loader is so important and can never be overwritten, it is often combined with the “boot code” and placed in a special write-protected region of memory. When these two pieces of software are combined, you will sometimes hear the result referred to as a bootloader. Combining the boot code and the loader has a practical benefit as well. Namely, the validity of the rest of the firmware can be confirmed—perhaps by computing a checksum and comparing it with a stored result—before it is actually executed. If the bootloader detects a possible error, it can use the delivery channel to request a software replacement, download the received code into flash, and then restart the system. This part of the bootloader’s functionality can even be used in the factory, to simultaneously automate the remainder of the firmware installation process and test the on-board parts of the delivery channel for defects.
Software patches
In some cases you may prefer to make only a small change to your firmware instead of replacing the entire memory image with a new one. For example, suppose that after your product has begun to ship, a customer discovers a bug during some Y2K-compliance testing at its site. As the lead software developer, you quickly realize that the source of the error is a date-dependent logging function borrowed from an older project. If that one routine could somehow be “patched over,” the product would work flawlessly; no other parts of the code use the date.
You’ll find several advantages to using a patch instead of a software replacement when making such a small change. For one thing, a patch requires that far fewer bits be downloaded over the delivery channel. Since the bandwidth for software upgrades may be limited and/or upgrades may have to live in the delivery channel for a long time, minimizing the total size of all software upgrades is often advantageous. You’ll save bandwidth and reduce the length of time required to perform an upgrade. To understand the limitations on bandwidth, consider this example.
A system I helped design a few years ago was on a broadcast network that had no uplink capability. The set-top box we were developing was intended to receive television transmissions from a satellite, decode them, and display the resulting images. In addition to the various video and audio channels beamed down to the set-top box by the satellite, there was also a data channel dedicated to the delivery of software upgrades. But the network had no way to know how long it had been since any particular set-top box had been turned on—it was a broadcast network, after all. All of the software upgrades we made after shipping our first unit would have to be transmitted in perpetuity, in a continuous loop, on the delivery channel! Whenever one of these set-top boxes was reawakened in the field, it would listen to the delivery channel first to learn about any new software upgrades. Then, beginning with the first upgrade it hadn’t previously installed, it would store them into memory, in the order they were made. The number and size of the software upgrades we made might, over time, use up all of the bandwidth available on our delivery channel and significantly affect the wakeup time for a long-sleeping unit. So it was very important that we reduced the size of each individual upgrade.
Smaller software upgrades can significantly reduce the bandwidth requirements of the delivery channel and also shrink the length of the upgrade period. The purpose of a software patch is, therefore, to minimize the size of any particular fix. But this is not as easy as it sounds. There is no guarantee that a patch will be exactly the same size as the code it replaces. And anyway, flash memories require an erase cycle before any write and only support block erase operations. These factors make it difficult to “patch” a piece of firmware in the same way that you would a moth-eaten sweater.
Requirement #4: Patch table
Unfortunately, you need to plan for software patches by anticipating (at design time) what parts of your program might require later changes. If you anticipate correctly, you can leave in all of the hooks you will need to easily replace these routines with patches. Such hooks typically center on a central “patch table,” through which key functions are actually called. These hooks create a small inefficiency at run time, but if you don’t have them in place, the patch process will be a lot more difficult.
Before executing one of the patchable functions, a lookup operation is performed to determine the address at which the function begins. The patch table is a data structure that contains the mapping necessary for this lookup operation to be performed. When a function is replaced by a patch, its entry in the patch table is modified to point to the newer code. All subsequent calls to that function will invoke the patch instead of the original code. Figure 2 shows how this might work in practice. When the original getDate function is patched over, only the appropriate entry in the patch table is actually modified. The original function remains in memory and the new function is located in some new area of the flash memory. A calling function, like createLogEntry, will automatically invoke the patch from now on.
For efficiency reasons, the patch table is usually just an array of addresses, organized by function number. But the actual data structure is up to you. If your target processor supports them, you might consider using software interrupts instead. In that way, the patch table will just be part of the interrupt vector table. Or, if you have a dynamic linking capability built into your software, you could use the symbol table to resolve all patchable function calls.
Since we are testing only the data bus at this point, all of the data values can be written to the same address. Any address within the memory device will do. However, if the data bus splits as it makes its way to more than one memory chip, you will need to perform the data bus test at multiple addresses, one within each chip.
It’s up to the author of each patch to decide whether that particular patch will completely replace the original function or will also invoke the old code as part of the new functionality. For example, you can imagine a patch that simply returns an error code if a particular argument’s value is invalid. It’s clear that the original function was flawed by not checking this, but the rest of the code in the original function is still good. The patch might simply perform this check and, if the argument is valid, call the original function. Thus a clever programmer can further reduce the size of a software patch.
Software extensions
In addition to fixing problems with the original software, you may also want to allow extensions to it, to make it do something it couldn’t when it was shipped from the factory. A base-3 log function could be added to an extensible calculator; a memo pad capability to an electronic book; or an address book to a cellular phone. This type of flexibility-of-use—traditionally associated only with general-purpose computers—has become more popular in embedded systems in recent years.
I believe that extensible embedded systems are the wave of the future, an essential component of the so-called post-PC era. 3Com’s Palm Computing Platforms, a range of PDAs running the PalmOS, were the first big commercial success in this regard. But just think of all the other extensible devices—electronic books and notepads; cellular phones, pagers, and GPS receivers; DVD players and set-top boxes—that will soon surround us. Each of these embedded devices will have a display, some sort of user input mechanism (a touchscreen, stylus, keypad, or full-blown keyboard), and lots of spare processor cycles. Software extensions from the manufacturer or third-parties can be used to personalize these devices to the needs of a particular user.
But extensions are not possible unless the designer of the embedded device plans ahead. In addition to providing a way to get such extensions to the device (delivery channel), a place to store them (flash memory), and a mechanism for storing them (loader), the designer must also create a software environment within which they can be run. In other words, you’ve got to pay attention to the needs of applications and application developers. You can no longer hobble together a piece of firmware that just happens to make the device behave properly, burn it into a master flash, and start shipping your product.
Requirement #5: Application launcher
First of all, the owners of your product are going to need some way to launch individual applications. If that’s not possible, then it doesn’t make sense to support extensions at all. What I have in mind here is something like the application launcher in the Palm devices: a folder (or folders) containing named icons that can be launched with the touch of a stylus or finger. But there’s no reason your application launcher has to be GUI-based. You might use soft buttons or a DOS-like command line interface instead. What the application manager actually looks like is up to you and should be based on the considered needs of your users.
An application launcher should also include a mechanism for removing applications from the system. Otherwise, the memory set aside for software extensions may eventually fill up with applications that are no longer useful or desired.
Requirement #6: Application identifiers
It is frequently desirable for the system or application launcher to distinguish between the various applications that are currently installed. In a PDA, for example, having a unique icon, application name, and application identifier is often helpful. The latter is used to uniquely identify each application and as an index into a variety of lookup tables. (If you will have third-party developers, you’ll need to plan a way to control the assignment and use of application identifiers.) The icon and application name are primarily provided for the convenience of the PDA user, who needs some way to select the appropriate application to run at a particular time.
Requirement #7: Application programming interfaces
Historically speaking, embedded developers haven’t really had to worry about defining the interface between the main application and the supporting software. Sometimes an effort is made to define the interface between the operating system and everything else, but that’s about it. Application code, device drivers, protocol stacks, and operating system calls have all mingled together to produce a working product. This is completely acceptable when software extensions are not supported.
To support the addition of new applications to the system after it has shipped, you’ll want to have a well-defined set of APIs that the application developer can use to create them. These APIs will reduce the likelihood of resource conflicts between two or more applications. For example, if an LCD display is only “owned” by one application at a time, there is no way for another application to interfere with its output. The second application will instead be stalled, inside an API call, waiting for its turn to “own” the display.
The typical types of APIs you’ll want to have are related to:
- Database functions—most user applications will need to store and query data
- Display functions—all of the applications should use the same set of widgets, for a common look and feel
- Character input functions—they’ll also all need a way to read data from a stylus or keyboard
- Interface-specific functions— provide a way to control the system’s interfaces with the outside world
In addition, a set of service APIs provides the ability for add-on software to make calls into the operating system, communicate with each other, and share data. All of these APIs should be stable and language independent.
Requirement #8: Expansion options
When you’re designing the hardware for an extensible system, leave application developers some extra input and output options. For example, even if your firmware doesn’t need to use a keyboard or a printer, put an infrared receiver and a parallel port on the system. The relatively small manufacturing cost of these features may have a big payoff in increased sales. For example, if you build a set-top box with these extra interfaces, someone else might come along and write a software extension for your system that allows it to be used as a word processor as well. Your set-top box could immediately be repositioned and marketed as an alternative to a personal computer. As a result, you might sell millions of extra units to families who need word processing but not other features traditionally found in a PC.
The list of expansion options to consider should include at least:
- Serial ports—for a modem, connection to another computer, and so on
- Parallel ports—for printers, external drives, data acquisition, and so on
- Infrared transmitters/receivers—for data exchanges with PDAs, printers, and digital cameras, and also for use with remote controls and infrared keyboards
- PC card (PCMCIA) slots—for flash memory cards, network connections, modems, and so on
So, if you’re designing an extensible embedded system, be sure to include a few of the extra interfaces that fit in best with the size and price of your box and the types of applications that might make sense for it. The beauty of designing a system around one set of functionality but allowing third-party developers to expand upon this is that you don’t have to think of every possible application at the time you’re designing the box. Leaving a few extra hooks for extra input and output mechanisms will increase the value of your system over the long term and may also increase its lifetime considerably. (Oh yeah, and don’t forget to provide APIs to access them.)
Requirement #9: Extra processing power
This requirement isn’t too tough to accomplish, nor should it be unexpected. Most embedded systems have leftover processing power. In fact, that’s a big reason that the prospect of software extensions is so exciting. When I’m not using my set-top box to watch TV or surf the Internet, why not use it as a word processor?
Requirement #10: Reset
Let’s face it, though. If a lot of your users are out there downloading new applications onto your product, they might occasionally mess something up. A poorly written application might crash the system. Or the user might simply get tired of having all those extra applications in memory and just want to go back to using the box for its original purpose. To support this type of recovery, it is a good idea to include some hardware-initiated mechanism for restoring the system to its factory default state. For example, the user might have to hold down three of the buttons on the front panel simultaneously and then answer a yes/no question. A positive response to this question should result in the bootloader removing any software add-ons installed after the user purchased the product, then restarting the system.
In addition to all of these “requirements,” there are also a couple of options you might want to consider.
Option #1: Backup mechanism
Ideally, a backup mechanism should be in place to allow users of your device to save—and, if necessary, restore—their personal data. For these purposes, it is reasonable to assume that the various software extensions could themselves be easily reinstalled after a hardware reset or a crash. So you don’t need a way to save and restore those. But the user may have stored names and addresses or his schedule for the next three months in your system and he probably doesn’t want to lose them.
You may, therefore, want to provide a way for the user to save his data. This could either be stored on a removable medium (a flash memory card or floppy disk, for example) or backed up to a personal computer (perhaps via one of those serial ports). While not strictly a requirement for supporting software extensions, this is certainly an important consideration for keeping your users happy.
Option #2: Centralized distribution
It may also be desirable for you to offer a centralized distribution mechanism for all of the software replacements, patches, and extensions. By this I mean something like a central phone number that every device dials into or a special URL on the Internet that the device is programmed to visit as part of the add-on process. In addition to making software add-ons easier to find, this has the added benefit of allowing you complete control over which third-party applications are available to the users of your product. You might even want to test third-party applications internally before allowing them to be distributed and authenticate developers or applications or both with encryption keys. This is often the most desirable way to allow users to download extensions to a device while still keeping the risk of a system crash or loss of user data to a minimum.
Attracting third-party developers
If you want your product to be both extensible and wildly successful in the consumer market, you’ll probably want to attract a large number of third-party developers to your platform. This can take considerable effort on your part. Many computing platforms are competing for the time and talent of independent software vendors (ISVs). Therefore, you must ensure that your platform is more attractive than the others it most directly competes with.
Always remember that one of the reasons that Windows has been more successful than MacOS and the Palm more successful than the Psion has been the number of third-party applications available. Microsoft and Palm have both gone out of their way to attract and support the ISVs who develop add-on software for their platforms. You need to do the same, while simultaneously building market share for your product. You must create a critical mass of users and applications in order for your platform to succeed against its rivals.
Fortunately, application developers are easier to attract than users; it is easier to define what they need to be kept happy. The first issue is, of course, tools. How do developers write and test software for your device? You should either provide or point developers to a tool suite consisting of at least: assembler, compiler(s), debugger, and simulator. The entire tool suite should be inexpensive (relative to the kind of income they can expect to earn writing software for your platform), feature-filled, and available on the most popular developer platforms (Windows 95/98/NT, Linux, and MacOS). In many cases, these will be the same tools you used internally when developing your embedded software.
Third-party developers will expect to find stable, well-documented APIs. Think about how your APIs might need to change in the future and plan for backwards-compatible changes wherever possible. There’s no reason that all of the application developers should have to rewrite their programs whenever you release a new version of the firmware (a software replacement or patch, for example).
In addition to documenting your APIs, you should provide a couple of sample applications with full source code. If these small applications are easy to mimic, that’s a plus. Use these examples to clearly explain what is expected of any application: where to put the icon, name, and application identifier; what the program’s main processing loop should consist of; and how to run and exit gracefully. Clearly define the minimal requirements for an application, as well as the broader options. Are there particular functions that must be implemented?
Developers feel good when they see their names on screen, so make it easy for them to create an “About…” box for their application. In addition, encourage the creation of help screens—to make applications easier to use. Also, make it easy to obtain application identifiers and, if you require them, digital signatures. Make any application testing and certification process efficient and inexpensive and any other barriers to developer entry as low as possible. You might even want to start a developer support program, organize periodic developer conferences or meetings, provide training for developers, and/or staff a developer support hotline.
Paying attention to the little details of the software development experience will help attract ISVs to your platform and keep them writing programs for it. If your product is successful, they will be successful. But your own product’s success also depends upon their hard work. So let them know that you care.
Plan ahead
With more and more embedded systems in the hands of consumers, there is a greater need for software upgrades to fix bugs and software extensions that increase the functionality of devices. Developing a system that allows these sorts of software changes and additions depends upon extra care and planning in the design stage. The list of requirements and options provided here should start you on your way.
This article was published in the September 1999 issue of Embedded Systems Programming. If you wish to cite the article in your own work, you may find the following MLA-style information helpful:
Barr, Michael. "Architecting Embedded Systems for Add-on Software," Embedded Systems Programming, September 1999, pp. 49-60.
Related Barr Group Courses:
For a full list of Barr Group courses, go to our Course Catalog.
