Developments in FPGA software, custom-built processors, single-board computers and compact vision systems offer designers a variety of choices when building embedded vision systems.
Andrew Wilson, European Editor
With the advent of faster processors, FPGA-based intellectual property, custom image processors, open-source software and modular compact vision systems, designers are faced with numerous choices when developing embedded vision systems. However, although the definition of what comprises an embedded vision system can be loosely described as one that combines an embedded computer and a vision system, this can vary depending on the final product manufactured.
Designers of smart cellular mobile devices or vision systems for automotive applications, for example, require highly integrated, rugged and compact vision processors and image sensors. Alternatively, those involved in the development of medical imaging systems such as borescopes, may require a less-compact, board-level based system that can be integrated into end-user product housings. For those integrating systems for the machine vision market, a smart camera may be viewed as such an embedded machine vision system. Many developers of machine vision systems, however, think of such systems as rugged, expandable products that allow a variety of camera interfaces to be deployed using off-the-shelf software.
Cross-pollination
In developing products for any type of embedded vision system, developers must be aware of the cross-pollination that exists between various hardware and software products that are available. Developers of mobile devices, for example, can take advantage of embedded vision processors that incorporate multiple processing elements to perform imaging tasks. Similarly, camera and frame grabber designers can leverage the power of an FPGA vendor's intellectual property (IP) to perform functions such as camera standards interface conversion, Bayer interpolation and lens distortion correction.
While embedded vision processors may incorporate multiple very long instruction word (VLIW), reduced instruction set (RISC) CPUs and custom hardware to perform image processing functions, their designers have realized that programming such devices must be easy to use. The EV61, EV62 and EV64 embedded vision processors from Synopsys (Mountain View, CA, USA;www.synopsys.com), for example, combine a 32-bit scalar core with a DSP and an optimized convolutional neural network (CNN) engine. To speed software development, the devices are supported by a software programming environment based on OpenCV (http://opencv.org) and OpenVX (www.khronos.org/openvx), a royalty-free standard for cross-platform acceleration of computer vision applications designed by the Khronos Group (Beaverton, OR, USA; www.khronos.org).
Similarly, the Myriad 2 processor from Movidius (San Mateo, CA, USA;www.movidius.com) also contains hybrid processing elements including twelve 128-bit VLIW processors and two 32-bit RISC processors. As well as a software development kit (SDK) that enables developers to incorporate proprietary functions, the company also supplies a reference design that includes a camera and micro-electro-mechanical (MEM) sensors (Figure 1).
While individual processors can be incorporated into embedded systems, there are other options to accelerate these tasks. These include embedding either embedded "soft-core" processors or IP into the FPGAs used in such systems. Such products can prove useful for those developing such products as smart cameras, drones and FPGA-based frame grabbers.
One example of such soft-core IP processors is the VectorBlox MXP Matrix Processor from CEVA (Mountain View, CA; USA;www.ceva-dsp.com). As a scalable soft-core processor designed for FPGAs, the IP can be implemented in an FPGA as a plug-in IP block that implements parallel vector processor algorithms on the 2D and 3D matrices commonly found in image processing. According to CEVA, the soft-core processor core can enhance the performance of standard MicroBlaze processors from Xilinx (San Jose, CA, USA; www.xilinx.com) by a minimum of 1-2 orders of magnitude. Taking advantage of this, Altek (Hsinchu, Taiwan; www.altek.com.tw) recently licensed CEVA's IP to perform object detection and tracking and 3D depth sensing in the company's drones and smart cameras.
FPGA libraries
Just as developers of cameras and frame grabbers can leverage the power of soft-core processors, they can also take advantage of FPGA IP libraries to perform dedicated image processing tasks. These include camera interfacing, image pre-processing functions such as Bayer interpolation, image compression, stereo vision, face detection and motion detection.
For the developer of machine vision systems that incorporates standard camera interfaces such as GigE Vision and CameraLink and CoaXPress, FPGA vendors and third-party suppliers offer IP to perform these tasks. Xilinx, for example, has developed its GigEVCore1.2, an FPGA core that maps the GigE Vision control and message channels to a soft-core embedded processor such as the MicroBlaze. Supporting both Xilinx and FPGAs from Altera (San Jose, CA, USA;www.altera.com), Sensor to Image (Schongau, Germany; www.s2i.org) has developed FPGA IP to allow designers to implement Camera Link, GigE Vision and CoaXPress (CXP) interfaces on FPGAs.
While making camera interfacing easier, the advent of high-density FPGAs also allows many image pre-processing functions to be incorporated. While pre-processing algorithms such as Bayer interpolation, bad pixel correction and color balancing can be performed on a CPU, they are more effectively performed in an FPGA (see "CCD cameras embed imaging algorithms,"Vision Systems Design, May 2007; http://bit.ly/VSD-0507). In the past, developers of cameras and frame grabbers needed to develop these algorithms. Today, however, vendors offer IP libraries to perform these tasks.
CoSynth (Oldenburg, Germany;www.cosynth.com) offers several IP cores optimized for use in FPGAs that support image processing functions such as Bayer pattern demosaicing, color-space conversion, image segmentation and morphological operations.
For use with camera based systems, IP cores for the integration of cameras via LVDS and Ethernet are available. Similarly, Logic Bricks from Xylon (Zagreb, Croatia;www.logicbricks.com) provide image processing functions ranging from Bayer decoding (demosiacing), perspective transformation and lens correction and MJPEG decoding.
Application specific
Just as many IP vendors implement basic image processing functions such as these, other companies (including Xylon) have realized the need to develop more application specific FPGA IP to speed the time to market of their customers' products. As well as providing such basic functions, Xylon has developed IP that performs tasks such as face detection and tracking, vehicle detection and vehicle driver drowsiness detection, all targeted at niche markets.
Xylon is not the only company to have developed such IP. For motion detection of ground, surface and aerial objects, Riftek (Minsk, Republic of Belarus, https://riftek.com) offers FPGA IP that allows both tracking and motion detection of objects. The company's T-COR-30 tracking core, for example, can be used in both Xilinx and Altera FPGAs.
For stereo vision applications, both Nerian Vision Technologies (Leinfelden-Echterdingen, Germany; https://nerian.com) and Fujisoft (Kanagawa, Japan;www.fsi.co.jp) both offer FPGA IP to perform stereo object matching, albeit using different algorithms.
After images are first rectified to compensate for lens distortions and camera alignment errors, Nerian Vision Technologies stereo vision core performs stereo matching on two grayscale input images by applying a variation of the Semi Global Matching (SGM) algorithm, a method originally developed by Heiko Hirschmueller of the German Aerospace Center (DLR; Wessling, Germany;www.dlr.de). His original 2005 paper entitled "Accurate and Efficient Stereo Processing by Semi-Global Matching and Mutual Information," can be found at http://bit.ly/VSD-SGM.
Other stereo methods can be used to perform this task, most notably the sum of absolute difference (SAD) algorithm that has been used by Dr. Keiji Saneyoshi, Associate Professor of Tokyo Institute of Technology (Meguro, Japan;www.titech.ac.jp). In conjunction with Altera, Fujisoft deploys this algorithm in its Stereo Vision IP Suite allowing stereo matching to be implemented in a combination of FPGA hardware and software on a pair of ARM Cortex-A9 CPUs both contained in a single Altera Cyclone V. (see "Expanding the Applications of Stereo Machine Vision," http://bit.ly/VSD-SMV).
Rather than use individual IP cores, developers can instead use graphical programming interfaces for programing FPGAs. These allow a number of image processing functions to be implemented in a data-flow manner. Two examples of such graphical programming interfaces are Visual Applets software from Silicon Software (Mannheim, Germany; https://silicon.software) and National Instruments' (Austin, TX, USA;www.ni.com) Vision Development Module that can be used with the company's LabVIEW FPGA Module to process images on an FPGA. Using these graphical interfaces functions such as pixel manipulation, image transforms, color space conversion and morphological operations can be graphically pipelined and embedded into FPGAs.
Board-level designs
Just as frame grabbers and smart cameras may incorporate FPGAs, designers of embedded systems using board-level products such as off-the-shelf CPUs, frame grabbers and I/O peripherals are faced with an even wider range of products from which to choose. Here, numerous board-level products based on standards ranging from OpenVPX, VME, CompactPCI, cPCI Express, PC 104, PC/104 Plus, EPIC, EBX and COM Express boards can all be used to build vision systems with different camera interfaces and I/O options.
For developers of embedded systems that choose to use board-level products, many standards organizations offer valuable information and manufacturers' product listings. These include the VITA Standards Organization (VITA; Oklahoma City, OK, USA,www.vita.com) that has defined and developed the VMEbus (and its various incarnations), VPX, and mezzanine card standards such as the PCI Mezzanine Card (PMC), switched mezzanine card (XMC) and FPGA Mezzanine Card (FMC), all primarily for military and aerospace applications. As the latest extension of the VME standard, the VPX standard retains both 6U and 3U Eurocard form factors and supports PMC and XMC mezzanine cards. A listing of companies that support this standard can be found at http://www.vita.com/vpx.
For over a decade, numerous frame grabber and camera interface boards have been developed to support PMC mezzanine cards (see "PMC frame grabbers attack military market,"Vision Systems Design, January 2005; http://bit.ly/VSD-PMC). With the emergence of so called "switched fabric" interfaces, many of these camera interface boards have migrated to XMC designs. Like older PMC based designs, these support numerous types of camera interfaces (see "Frame grabbers target embedded imaging," Vision Systems Design, April 2014, http://bit.ly/VSD-FG).
Many of these support well-established camera-to-computer interface standards such as Camera Link as well as more recent standards such as CoaXPress. The Condor 4100 XMC Series from EIZO Rugged Solutions (Altamonte Springs, FL, USA;www.eizorugged.com), for example, are XMC designs based on AMD's Radeon E8860 GPU that support camera input standards such as 3G-SDI, HDMI, NTSC and CoaXPress (Figure 2).
Similarly, the Peripheral Component Interconnect Special Interest Group (PCI-SIG; Beaverton, OR, USA; https://pcisig.com) is a consortium responsible for specifying the Peripheral Component Interconnect (PCI), PCI-X, and PCI Express (PCIe) and PCI Express Mini Card standards. Add-in boards that implement the PCIe standard include those based on CompactPCI and computer on Module Express (COM Express) that are defined by the PCI Industrial Computer Manufacturers Group (PICMG; Wakefield, MA, USA;www.picmg.org).
While numerous companies support the PCIe standard to allow cameras with disparate outputs to be interfaced to host computers, embedded systems designers looking to reduce the cost and size of their systems can now leverage the power of a number of small form factor PCI Express Mini Cards with which to perform this task (see "Frame grabbers add features and formats",Vision Systems Design, June 2015; http://bit.ly/VSD-FGF).
Companies that produce such PCI Express Mini Card boards include the AVC8000nano, an eight-channel NTSC/PAL frame grabber from Advanced Micro Peripherals (Witchford, UK;www.ampltd.com), the CM313BW from AVerMedia (New Taipei City, Taiwan; www.avermedia.com ) that supports a single 3G-SDI input, the HD-SDI VCE-HDmPCIe01 from Imperx (Boca Raton, FL, USA; www.imperx.com) and the PIXCI EB1mini Base Camera Link card from EPIX (Buffalo Grove, IL, USA; www.epixinc.com).
COM Express
Other options exist for designers of embedded systems. Depending on the application to be formed, the use of Computer-On-Module (COM) boards offer developers a means to both implement embedded single board computers or use such processor modules with a carrier board that contains other application specific I/O. Typically, such COM Express boards consist of a processor, memory, graphics display capability and numerous types of I/O including Ethernet, USB 3 and VGA capability.
Developers using USB3 or Ethernet-based cameras can use such boards to build relatively inexpensive imaging systems using these interface standards. One example of such a COM Express board is the SOM-6898 from Advantech (Milpitas, CA, USA;www.advantech.com), a Type 6 COM Express Compact Module that features an Intel i7 processor, up to 32GBytes of memory using a small outline dual in-line memory module), Ethernet, USB and graphics interfaces (Figure 3). Where other camera interface standards are required, COM Express carrier boards can increase the functionality of the embedded vision system.
For example, the Express-BASE6, an ATX size COM Express Type 6 carrier board from ADLINK Technology (New Taipei City, Taiwan;www.adlinktech.com), can be used with COM Express Type 6 modules and allows PCI Express x 1, x 4 or x 16 add-in cards to be added as required (http://bit.ly/VSD-ADLT). More information on the COM Express standard can be found at http://bit.ly/VSD-COMX and http://bit.ly/VSD-COMX2.
Small form factors
Other popular add-in boards that incorporate the PCIe bus for use in embedded systems include those based on the PCI/104-Express form factor. Specifications for these boards are controlled by The PC/104 Consortium (Los Gatos, CA, USA; http://pc104.org). Here again, numerous frame grabbers are available to capture images from a variety of camera interfaces (see "Small-Form-Factor Frame Grabbers Enable Embedded Applications,"Vision Systems Design February 2013;http://bit.ly/VSD-0213).
Like XMC mezzanine cards, PCI/104-Express support numerous types of analog and digital inputs including Camera Link and the CoaXPress (CXP) standards. Perhaps the first to support the CXP standard in this form factor, the Coaxlink Duo PCIe/104-EMB board from Euresys (Angleur,Belgium;www.euresys.com) features two CoaXPress CXP-6 channels each with 1250 MBytes/s bandwidth (Figure 4).
Just as designers using COM Express processor boards can expand their systems using carrier boards, developers employing frame grabbers can add processing and I/O capability to PCI/104-Express based systems using multiple PCI/104 Express modules and carrier boards. Here, two single-board form factors known as EPIC (Embedded Platform for Industrial Computing) and EBX (Embedded Board, eXpandable) allow different types of PC/104 modules to be added to a host single-board computer.
As an example, the Atlas PCI/104-Express SBC board from Diamond Systems (Sunnyvale, CA, USA;www.diamondsystems.com) is a PCI/104-Express single board computer with an Intel Atom N2800 CPU that allows PCI-104 and PCIe/104 cards to be added. In this way, vision systems using PCIe/104 frame grabbers can incorporate USB 2.0, Gigabit Ethernet, SATA and digital I/O functionality.
Numerous choices
While the choices of which type of processor, IP, add-in board or vision system may at first seem overwhelming, several organizations aim to make the process easier. While the Embedded Vision Alliance (Walnut Creek, CA, USA;www.embedded-vision.com) provides information that focuses on processors, cameras, vision algorithms and development tools used to build embedded vision systems, the Imaginghub (https://imaginghub.com) recently launched by Basler (Ahrensburg, Germany; www.baslerweb.com) offers developers the chance to participate and create embedded projects.
Examples of such projects include how to create a face detection script with OpenCV and Python 3 (www.python.org) on a AIIS-1200P vision system from Advantech using a Basler dart camera. As part of the Basler Imaginghub, Dream Chip Technologies (Garbsen, Germany; www.dreamchip.de) has developed a ready to use Cyclone V-based BCON camera interface IP for easy integration into customers FPGA designs.
By using custom processors, FPGA-based software, off-the-shelf board-level products and open-source and commercially-available software, developers can build products faster and more cost-effectively. Because of such developments, embedded systems will be deployed more rapidly in such disparate applications as factory automation, medical imaging, military and consumer applications.
Companies and organizations mentioned
ADLINK Technology
New Taipei City, Taiwan
www.adlinktech.com
Advanced Micro Peripherals
Witchford, UK
www.ampltd.com
Advantech
Milpitas, CA, USA
www.advantech.com
Altera
San Jose, CA, USA
www.altera.com
Altek
Hsinchu, Taiwan
www.altek.com.tw
AVerMedia
New Taipei City, Taiwan
www.avermedia.com
Basler
Ahrensburg, Germany
www.baslerweb.com
CEVA
Mountain View, CA, USA
www.ceva-dsp.com
CoSynth
Oldenburg, Germany
www.cosynth.com
Diamond Systems
Sunnyvale, CA, USA
www.diamondsystems.com
Dream Chip Technologies
Garbsen, Germany
www.dreamchip.de
EIZO Rugged Solutions
Altamonte Springs, FL, USA
www.eizorugged.com
Embedded Vision Alliance
Walnut Creek, CA, USA
www.embedded-vision.com
EPIX
Buffalo Grove, IL, USA
www.epixinc.com
Euresys
Angleur, Belgium
www.euresys.com
Fujisoft
Kanagawa, Japan
www.fsi.co.jp
Imaginghub
https://imaginghub.com
Imperx
Boca Raton, FL, USA
www.imperx.com
Khronos Group
Beaverton, OR, USA
www.khronos.org
Movidius
San Mateo, CA, USA
www.movidius.com
National Instruments
Austin, TX, USA
www.ni.com
Neria Vision Technologies
Leinfelden-Echterdingen, Germany
https://nerian.com
OpenCV
http://opencv.org
OpenVX
www.khronos.org/openvx
PC/104 Consortium
Los Gatos, CA, USA
http://pc104.org
PCI Industrial Computer Manufacturers Group (PICMG)
Wakefield, MA, USA
www.picmg.org
Riftek
Minsk, Republic of Belarus
https://riftek.com
Sensor to Image
Schongau, Germany
www.s2i.org
Silicon Software
Mannheim, Germany
https://silicon.software
Synopsys
Mountain View, CA, USA
www.synopsys.com
The German Aerospace Center (DLR)
Wessling, Germany
www.dlr.de
Tokyo Institute of Technology
Meguro, Japan
www.titech.ac.jp
VITA Standards Organization (VITA)
Oklahoma City, OK, USA
www.vita.com
Xilinx
San Jose, CA, USA
www.xilinx.com
Xylon
Zagreb, Croatia
www.logicbricks.com
For more information about embedded vision companies and products, visitVision Systems Design's Buyer's Guide http://buyersguide.vision-systems.com/index.html
