The 3D graphics and embedded vision industries have both seen tremendous growth in recent years, fueled by advancements in GPUs (graphics processing units), sensors, cameras and visual AI. However, while the graphics industry has benefited from open standard APIs, such as OpenGL and Vulkan, that enable software portability across many generations of GPU hardware from diverse vendors, the vision market has lacked interoperability standards for portable access to embedded camera systems, often raising costs and slowing innovation when utilizing vision processing.
The Khronos Group industry consortium (Beaverton, OR, USA), which manages OpenGL and Vulkan, is now developing the new Kamaros API standard for embedded camera systems. The aim is to increase hardware/software interoperability for embedded vision products. But the graphics and vision market are very different in character, as are camera and GPU hardware architectures, necessitating different approaches to the design of effective open standard APIs.
In this article, we will discuss the design challenges Kamaros faces and how the Kamaros working group at Khronos proposes to overcome them and bring the benefits of interoperability to embedded vision.
3D Graphics Industry Growth Fueled by Open Standards
The 3D graphics industry enjoyed explosive growth in the 1990s and 2000s with the introduction of the first GPUs, specialized devices designed to process data-intensive graphics more efficiently than a CPU. GPU hardware accelerated 3D rendering, quickly becoming widely adopted in a range of applications from video games to professional design, visualization and film production. The pervasive adoption of GPUs was fueled by the industry’s early adoption of the OpenGL standard in 1992. OpenGL provides an abstracted model of the graphics pipeline and an API for accessing any GPU that provides a driver that maps that API to its own hardware architecture.
This enables hardware vendors to innovate hardware implementations without exposing proprietary technology while enabling applications to easily deploy on a wide hardware installed base, which in turn fuels the demand for innovative graphics hardware in more markets and platforms.
The Khronos Group was also instrumental in extending the availability of 3D graphics to additional markets, including mobile devices with the OpenGL ES standard and the World Wide Web with the WebGL API—direct descendants of OpenGL. Khronos’ new generation Vulkan API has further evolved OpenGL for modern graphics workloads.
Portability and easy access to GPU acceleration through open API standards continue to fuel adoption of 3D technology, even as graphics hardware goes through significant rounds of innovation.
Could the embedded vision market enjoy similar benefits if open standards lower barriers to accessing innovative camera, sensor, and ISP (image signal processor) hardware?
The Embedded Vision Industry may be Missing an Opportunity for Additional Growth
The embedded vision industry has experienced its own rapid growth surge, with millions of cameras being deployed in drones, robots, smart devices and autonomous machines, along with AI to make sense of visual data. However, the historical path taken by the embedded vision market has resulted in camera systems today typically exposing their functionality through proprietary APIs and software development kits (SDKs) tied to specific hardware vendors. This creates a significant burden on software developers and system integrators when trying to switch or upgrade camera systems. The lack of an open standard API makes it costly to leverage different camera technologies as large portions of a vision processing stack often need to be rewritten to target new camera hardware.
Additionally, the lack of a thoughtfully designed open standard API that provides functional portability through abstracting details of the underlying camera hardware has meant that camera vendors are often not able to enable software developers to fully utilize their camera's capabilities if they do not wish to disclose proprietary sensor and processing pipeline details.
The industry’s need to fill this gap in the open standards landscape is becoming more urgent, as cameras are being deployed in a growing number of increasingly complex systems, often with multiple cameras and close integration with sophisticated downstream processing that demands precise control over the generation of the stream of images from the camera. This situation is fueling a growing reckoning across the embedded vision industry that an open standard API is needed to drive innovation and reduce costs.
The Khronos Group and Kamaros
For over 20 years, Khronos has created open, royalty-free API standards that enable software applications and frameworks to harness the power of dedicated silicon processors, often using parallel processing, to accelerate the performance of demanding use cases such as 3D graphics, augmented and virtual reality, vision processing and machine learning/inferencing.
The new Kamaros API standard will fit within a family of active Khronos standards providing software control over the multiple pipeline stages of embedded vision hardware typically used in consumer, industrial, automotive and scientific applications. This includes OpenCL and OpenVX for accelerating vision and inferencing operations, which complement the proposed functionality of Kamaros that is being designed to provide portable access to cameras, sensors and ISP acceleration.
The Goals of the Kamaros API
API standards for accessing acceleration hardware, or GPUs, in embedded markets can enable numerous benefits, including:
- Cross-platform software portability
- Decoupled software and hardware development for easier deployment and integration of new components
- Cross-generation reusability
- Field upgradability
Such standards reduce costs, shorten time to market, and lower the barriers to using advanced techniques, such as parallel processing, for faster inferencing and vision processing in compelling real-world products.
Any successful API that provides portable access to hardware, including cameras, must offer significant benefits to both the implementers and users of the standard to drive wide adoption and support. For software vendors, an effective open standard for embedded cameras should provide cross-platform portability of engines and applications for wider deployment and lower development costs across diverse hardware. Meanwhile, hardware vendors should enable software developers to discover and effectively make use of the full potential of their camera system capabilities without needing to expose implementation details—enabling hardware innovation and differentiation while gaining access to a larger ecosystem of libraries and applications.
Kamaros Design
Kamaros is an application-facing, user-space API. It is being designed to be deployed on diverse operating systems and underlying hardware architectures, and used by applications, frameworks, and libraries. Kamaros will be callable through multiple language bindings, including C, C++ and Python. Like other modern Khronos APIs, Kamaros will use a command dispatch system through loadable layers to enable developers to use installable layers for validation, profiling and debugging. Kamaros is also expected to use Vulkan-style resource management, synchronization and command queues, with a focus on minimalism and efficiency to meet the highly sensitive latency and throughput requirements of embedded vision workloads.
The Kamaros API will allow applications to identify and select available cameras in a system, discover camera properties, and configure processing pipelines and settings via pipeline templates that enable applications to make full use of hardware capabilities, including multiple input and output streams.
Kamaros is an abstracted API that does not require developers to design code to conform to proprietary hardware implementation details. Therefore, it can be used portably across different devices without platform-specific code. Once sensors and processing pipelines are configured, Kamaros enables frames to be requested using per-frame dynamic controls, with all applied settings and any exceptions returned in per-frame metadata.
The use of pipeline templates enables Kamaros to address the needs of a broad spectrum of camera vendors and developers. Simple templates configured for common use cases will be widely supported, enabling broad portability for applications that wish to start acquiring frames after just a few API calls. Developers who wish to use camera capabilities can use the Kamaros API to precisely configure processing pipelines to suit their needs. Camera vendors can also publish sophisticated pipeline templates that expose advanced or custom pipelines that have been tested and tuned for efficiency.
Get Involved
Any organization is welcome to join Khronos for a voice and a vote in the development of Kamaros and other Khronos standards.
The 3D graphics industry has benefited tremendously from open standard APIs like OpenGL and Vulkan that enable applications to access GPU acceleration across different hardware vendors. However, the rapidly emerging world of embedded vision lacks standard camera system APIs, resulting in interoperability inefficiencies. Intended for first release in 2024, Khronos' Kamaros API for embedded camera systems could have a similarly transformative impact as OpenGL did for graphics, offering the chance for the embedded vision industry to enjoy the benefits of open standards that effectively connect software to silicon.
