Systems developers add real-time operating systems to meet timing limits

Andrew Wilson

Editor at Large

Smart cameras, unmanned vehicles, robotic-vision systems, and automatic landing systems all require video image processing in predicted time frames. These so-called real-time imaging systems generally use reduced-instruction-set computers, digital-signal processors (DSPs), or general-purpose central processing units to perform computational imaging analyses.

To build such systems, designers are turning to several off-the-shelf processors, frame grabbers, and image-processing boards, as well as numerous real-time operating systems (RTOSs) developed especially for imaging-system applications for which demanding timing constraints must be met. During the design of a real-time image-processing system, developers usually begin with the selection of the host processor.

In many imaging applications, the host processors are either Pentium, x86, 68k, PowerPC, or DSP-based. Because of this, RTOS vendors offer specialized products that are front-runners in the embedded-systems market. To date, several vendors, including JMI Software Systems (Spring, House, PA), Kadak Products (Vancouver, BC, Canada), Lynx Real-Time Systems (San Jose, CA), and VenturCom (Cambridge, MA), offer Pentium-based RTOSs. Other suppliers, such as Cygnus Solutions (Sunnyvale, CA), Enea OSE Systems (Dallas, TX), and Mentor Graphics (Wilsonville, OR), also offer distinctive products such as PowerPC, MIPS, and ARM processors.*

Multitasking operation

Once the choice of host processor has been determined, system developers must then decide on the type of RTOS needed. An RTOS increases a system`s structure by taking the operations most common within the real-time applications and making them standard services. Some basic operations include dispatchers to provide multitasking, interprocess communications, and memory management to provide dynamic memory allocation.

"Three basic methods exist to perform multitasking," says Ralph Moore, president of Micro Digital (Garden Grove, CA). "The simplest is the round-robin scheme, a variation of a loop where each task has its turn and runs until finished. Slightly more sophisticated is the time-slice approach, where each task is allowed to run for a fixed time and is then suspended until the next cycle. And, there is the pre-emptive model, where the most important ready task runs until it is finished or pre-empted by a more important task. Whereas the first two methods repetitively cycle through all of the tasks, the third method does not and is probably the best choice for a real-time system," adds Moore.

A developer of intelligent digital imaging products for OEMs, Sound Vision (Framingham, MA) needed such a real-time multitasking operating system to develop its Clarity platform. This platform consists of hardware, software, and services designed to provide digital-camera manufacturers with an integrated solution to shorten time-to-market and decrease development costs. It is based on a custom ASIC that includes the ARM720TDMI processor from Advanced RISC Machines (Los Gatos, CA), sensor interface, USB, CompactFlash, SmartMedia, and memory interfaces (see Fig. 1). Using the Nucleus operating real-time multitasking operating system from Accelerated Technology Inc. (Sunnyvale, CA), the Clarity evaluation platform is capable of rapid image acquisition via image buffering and background processing.

"We chose the Nucleus operating system for several reasons," says Bob Caspe, founder and chairman of Sound Vision. "First, it offers standard task control, intertask communication, intertask synchronization, memory management, timer management, and interrupt management. In addition, it is compliant with C++ standards that allow you to interface to all kernel facilities as objects and use object-oriented design and programming techniques."

Consequently, when the user attempts to take another picture after compression has started, the compression task is suspended while more images are being acquired without delay because the Nucleus operating system within the camera supports multiple independent tasks with different priorities.

Context switch time

Two important factors--context switch time and interrupt latency--are used to compare RTOS performance. Linda Thompson, at the Mentor Graphics embedded software division (Wilsonville, OR), says, "While context or task switch time purports to be an indicator of RTOS efficiency, it measures the time taken for a RTOS to save the context of one task, puts its registers and stack pointer into a control block, and loads the context of another task from a second control block. Context switch times provided by RTOS vendors frequently do not even include the time it takes the RTOS to decide which task to schedule."

She adds, "Indeed, context switch time never shows up in isolation within an application but as an added overhead on an RTOS call. Because context switch time alone provides little information about the overhead of the RTOS, and it is not reflective of any outwardly visible operation within a real-time system, its worth as an indicator of efficiency is questionable. Developers should never rely on context switch time to compare RTOSs." Interrupt latency is defined as the worst-case time that can pass between the assertion of an interrupt and the execution of the first instruction of the corresponding interrupt service routine (ISR).

System interrupt latency depends on the interrupt latency of the processor hardware. It consists of the time it takes the RTOS to handle the interrupt and pass control to the appropriate ISR and the RTOS`s interrupt disable time, which is the worst-case length of time that the RTOS can disable interrupts to protect its own critical code sections. "Every RTOS has some interrupt latency associated with it," Thompson says. "Indeed, interrupt latency is certainly one indication of the responsiveness of an RTOS."

Many real-time image-processing and machine-vision systems are required to image data and process the data to determine whether a specific event, such as a faulty part, has been detected. To do so with an RTOS, either polling or interrupt mechanisms can be used.

Whereas polling requires periodic examination of the environment by the processor, interrupt-driven methods allow the imaging hardware to inform the processor of an event. "The main disadvantage to polling," says Ed Klein, northeast regional field application engineer at Wind River Systems (Alameda, CA), "is the amount of processing time required to perform the poll. But if the processor is in an idle state, polling is an acceptable alternative to pre-emption," he adds. Other drawbacks to polling are the potential for missed events and increased latency.

Depending on the persistence of the environmental conditions and the period and duration of the poll, polling can miss an event. "However, polling is a simple, deterministic method for implementing synchronous systems," Klein says. Unlike polling, interrupts pre-empt the processor and cause the current task to suspend execution. The main advantage to this method is that events are processed as they occur. However, interrupts usually require additional hardware support and tend to be more costly.

Although interrupts provide asynchronous processing, they also introduce more complexity. Alternatively, a signal event can be used to interrupt process flow. Signals are used to tell the current process to terminate, clean up, and start another process. "Graphics controllers are good examples of how signals can be used," says Klein. "The controller may be in the middle of building an image when a new image requires servicing. The process is signaled, the old image is dropped, and the process is reset to build the new image," he adds.

Pre-empt or not

"In choosing an RTOS," says Thompson of Mentor Graphics, "a distinction must be made between pre-emptable and non-pre-emptable kernels. Whereas most RTOS kernels support pre-emptive task scheduling, in which application tasks can pre-empt each other at will, they are also interruptible. This means that when an interrupt occurs while the kernel is executing, an ISR will be invoked almost immediately. Many commercial RTOS kernels are not pre-emptable, however. So when an ISR exits through the kernel, any kernel service executing when the interrupt occurred must be completed before the kernel`s task dispatcher can run," she says.

"If the kernel is executing a non-time-critical operation on behalf of a low-priority task, this operation must complete before a high-priority task the ISR made ready can run," Thompson continues. This adds to the rescheduling latency between the time an ISR exits and the time a task can be dispatched in response. In a non-pre-emptable kernel, this rescheduling latency could equal the length of the slowest system call. "But a pre-emptable kernel can put the low-priority operation "on hold" and dispatch the high-priority task at once, minimizing the rescheduling latency and improving the responsiveness of the system," she states.

The OSE Realtime Kernel from Enea OSE Systems (Dallas, TX), a fully pre-emptive kernel optimized to provide low-latency and high rates of data throughput, is used in distributed and multiprocessor systems. Agema Infrared Systems--now FLIR Systems (Portland, OR)--chose the OSE RTOS to develop its THV 400 infrared (IR) camera. At the heart of the camera system, three 8-bit 8051 processors from Intel Corp. (Santa Clara, CA) are used for IR scanner control, analog-to-digital conversion, operator interface, temperature calculation, color palette display functions, and floppy-disk control.

The development environment was based on networked PCs running an Intel ASM51 assembler and a PLM51 compiler. To test the code, Agema engineers used both the EMUL51 hardware/software debugger from Nohau (Campbell, CA) and the OSE signal tracking debugger from Enea OSE Systems. Says Hans Hallin of FLIR Systems, "Buying the OSE kernel saved at least four months in the development of the THV system." A system operator is presented with a real-time color (or shaded monochrome) image directly on the camera monitor where temperature and emission data are displayed (see Fig. 2). Once digitized, images can be stored either on disk or continuously on video.

While many RTOSs are available today, designers are striving to shorten the development cycle. The key to achieving this time-to-market goal resides in the selection of appropriate tools for improved productivity. Ideally, this choice involves the use of turnkey development systems with a variety of coordinated tools.

Because of this, products such as Photon from QNX Software Systems Ltd. (Kanata, Ontario, Canada) are offered as a microkernel surrounded by several core managers for input, graphics, and font management, and a team of optional, cooperating processes. These products allow developers to scale their systems from small embedded systems to workstation-class development environments.

At the Neptec Design Group (Kanata, Ontario, Canada), QNX`s Photon has been adopted as the operating system of choice for an Advanced Space Vision System (ASVS). Used with the Canadarm (a space shuttle remote manipulator system), this guidance system allows astronauts to position and orient a payload by providing the exact location, orientation, and motion information of objects outside the space shuttle. To operate correctly, the ASVS identifies target dots that have been placed on the object being monitored.

Once in orbit, the ASVS uses one or more of the shuttle`s external cameras to monitor the target dots. As the object moves, the ASVS measures the changing position of the dots. An autoranging feature permits the ASVS to switch from one set of targets to another when a set is moved out of its field of vision. At this point, photogrammetric analysis is used to calculate the three-dimensional position and orientation of the object relative to the camera, based on the two-dimensional image. By measuring the precise positions of each dot on the image plane and knowing the distance from the image plane to the projection center and the geometry of the real target dots, the ASVS calculates the distance from the camera to the payload. The calculated position, orientation, and rate data are then displayed as either x, y, z, roll, pitch, or yaw Cartesian data, along with the rate information of each value.

To perform this task, video signals from the shuttle`s exterior cameras are fed to the ASVS, which consists of two networked IBM Corp. (Armonk, NY) ThinkPads mounted on custom expansion pads (see Fig. 3 on p. 60). The ASVS determines the cameras` positions and orientation information and then passes it to the shuttle`s video system for display on flight-deck monitors. It can also control many camera functions, such as pan, tilt, focus, and zoom, to best track the target array and keep as many target dots in the field of view as possible.

As newer processors increase the speed and performance of embedded imaging systems, RTOS vendors will most likely supply the tools required to support them. In choosing vendors, however, systems integrators should investigate the vendors` product support history--whether it is in the DSP, RISC, or CPU arena--as well as their ties to semiconductor manufacturers. In addition, they should also pay careful consideration to the types of support tools needed to help them bring their products to market more quickly.

Click here to enlarge image

FIGURE 1. Designed to provide digital-camera manufacturers with an integrated solution-shortening time to market while decreasing development costs

Click here to enlarge image

the Clarity development system is based on a custom ASIC that includes an ARM720TDMI processor, a universal asynchronous receiver-transmitter, a first-in first-out buffer, and memory interfaces.

Click here to enlarge image

FIGURE 2. FLIR Systems uses the OSE RTOS in its THV 400 infrared camera to run three 8-bit 8051 processors that control scanner control, analog-to-digital conversion, operator interface, and temperature calculation. During camera operation, the user is presented with a real-time picture directly on the camera`s monitor, where temperature and emission data are also readable. The images can be stored on disk, and the whole measuring session can be recorded on video.

Click here to enlarge image

FIGURE 3. Neptec Design Group`s Advanced Space Vision System (ASVS) can calculate the distance from a space shuttle`s exterior camera to an orbiting payload. Calculated position, orientation, and rate data are then displayed as either x, y, z, roll, pitch, or yaw Cartesian data along with the rate information of each value. To perform this task, video signals from the shuttle`s exterior cameras are fed to the ASVS, which consists of two networked IBM ThinkPads that use QNX`s Photon windowing system running under QNX`s real-time operating system.

Systems developers add real-time operating systems to meet timing limits

Voice Your Opinion

To join the conversation, and become an exclusive member of Vision Systems Design, create an account today!

Latest in Imaging Boards and Software

Neurocle Releases Updated Inspection Software

High-Performance Edge Computing with PICMG COM-HPC: A Virtual Event Series

Sam’s Club Launches Machine Vision Technology to Expedite Shopping

Avnet Launches Development Kit for Embedded Vision Products

Micropsi Industries Launches Updated AI-vision Software for Robotics