Documentation: Add architecture documentation
The documentation is copied mostly verbatim from the website, with small modifications to the ascii art diagrams to make them compile. Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Acked-by: Jacopo Mondi <jacopo@jmondi.org> Acked-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
This commit is contained in:
Laurent Pinchart
parent
73fec989ae
commit
58a31afc0c
3 changed files with 385 additions and 0 deletions
383
Documentation/docs.rst
Normal file
383
Documentation/docs.rst
Normal file
|
|
@ -0,0 +1,383 @@
|
|||
*************
|
||||
Documentation
|
||||
*************
|
||||
|
||||
Feature Requirements
|
||||
====================
|
||||
|
||||
Device enumeration
|
||||
------------------
|
||||
|
||||
The library shall support enumerating all camera devices available in the
|
||||
system, including both fixed cameras and hotpluggable cameras. It shall
|
||||
support cameras plugged and unplugged after the initialization of the
|
||||
library, and shall offer a mechanism to notify applications of camera plug
|
||||
and unplug.
|
||||
|
||||
The following types of cameras shall be supported:
|
||||
|
||||
* Internal cameras designed for point-and-shoot still image and video
|
||||
capture usage, either controlled directly by the CPU, or exposed through
|
||||
an internal USB bus as a UVC device.
|
||||
|
||||
* External UVC cameras designed for video conferencing usage.
|
||||
|
||||
Other types of camera, including analog cameras, depth cameras, thermal
|
||||
cameras, external digital picture or movie cameras, are out of scope for
|
||||
this project.
|
||||
|
||||
A hardware device that includes independent camera sensors, such as front
|
||||
and back sensors in a phone, shall be considered as multiple camera devices
|
||||
for the purpose of this library.
|
||||
|
||||
Independent Camera Devices
|
||||
--------------------------
|
||||
|
||||
When multiple cameras are present in the system and are able to operate
|
||||
independently from each other, the library shall expose them as multiple
|
||||
camera devices and support parallel operation without any additional usage
|
||||
restriction apart from the limitations inherent to the hardware (such as
|
||||
memory bandwidth, CPU usage or number of CSI-2 receivers for instance).
|
||||
|
||||
Independent processes shall be able to use independent cameras devices
|
||||
without interfering with each other. A single camera device shall be
|
||||
usable by a single process at a time.
|
||||
|
||||
Multiple streams support
|
||||
------------------------
|
||||
|
||||
The library shall support multiple video streams running in parallel
|
||||
for each camera device, within the limits imposed by the system.
|
||||
|
||||
Per frame controls
|
||||
------------------
|
||||
|
||||
The library shall support controlling capture parameters for each stream
|
||||
on a per-frame basis, on a best effort basis based on the capabilities of the
|
||||
hardware and underlying software stack (including kernel drivers and
|
||||
firmware). It shall apply capture parameters to the frame they target, and
|
||||
report the value of the parameters that have effectively been used for each
|
||||
captured frame.
|
||||
|
||||
When a camera device supports multiple streams, the library shall allow both
|
||||
control of each stream independently, and control of multiple streams
|
||||
together. Streams that are controlled together shall be synchronized. No
|
||||
synchronization is required for streams controlled independently.
|
||||
|
||||
Capability Enumeration
|
||||
----------------------
|
||||
|
||||
The library shall expose capabilities of each camera device in a way that
|
||||
allows applications to discover those capabilities dynamically. Applications
|
||||
shall be allowed to cache capabilities for as long as they are using the
|
||||
library. If capabilities can change at runtime, the library shall offer a
|
||||
mechanism to notify applications of such changes. Applications shall not
|
||||
cache capabilities in long term storage between runs.
|
||||
|
||||
Capabilities shall be discovered dynamically at runtime from the device when
|
||||
possible, and may come, in part or in full, from platform configuration
|
||||
data.
|
||||
|
||||
Device Profiles
|
||||
---------------
|
||||
|
||||
The library may define different camera device profiles, each with a minimum
|
||||
set of required capabilities. Applications may use those profiles to quickly
|
||||
determine the level of features exposed by a device without parsing the full
|
||||
list of capabilities. Camera devices may implement additional capabilities
|
||||
on top of the minimum required set for the profile they expose.
|
||||
|
||||
3A and Image Enhancement Algorithms
|
||||
-----------------------------------
|
||||
|
||||
The camera devices shall implement auto exposure, auto gain and auto white
|
||||
balance. Camera devices that include a focus lens shall implement auto
|
||||
focus. Additional image enhancement algorithms, such as noise reduction or
|
||||
video stabilization, may be implemented.
|
||||
|
||||
All algorithms may be implemented in hardware or firmware outside of the
|
||||
library, or in software in the library. They shall all be controllable by
|
||||
applications.
|
||||
|
||||
The library shall be architectured to isolate the 3A and image enhancement
|
||||
algorithms in a component with a documented API, respectively called the 3A
|
||||
component and the 3A API. The 3A API shall be stable, and shall allow both
|
||||
open-source and closed-source implementations of the 3A component.
|
||||
|
||||
The library may include statically-linked open-source 3A components, and
|
||||
shall support dynamically-linked open-source and closed-source 3A
|
||||
components.
|
||||
|
||||
Closed-source 3A Component Sandboxing
|
||||
-------------------------------------
|
||||
|
||||
For security purposes, it may be desired to run closed-source 3A components
|
||||
in a separate process. The 3A API would in such a case be transported over
|
||||
IPC. The 3A API shall make it possible to use any IPC mechanism that
|
||||
supports passing file descriptors.
|
||||
|
||||
The library may implement an IPC mechanism, and shall support third-party
|
||||
platform-specific IPC mechanisms through the implementation of a
|
||||
platform-specific 3A API wrapper. No modification to the library shall be
|
||||
needed to use such third-party IPC mechanisms.
|
||||
|
||||
The 3A component shall not directly access any device node on the system.
|
||||
Such accesses shall instead be performed through the 3A API. The library
|
||||
shall validate all accesses and restrict them to what is absolutely required
|
||||
by 3A components.
|
||||
|
||||
V4L2 Compatibility Layer
|
||||
------------------------
|
||||
|
||||
The project shall support traditional V4L2 application through an additional
|
||||
libcamera wrapper library. The wrapper library shall trap all accesses to
|
||||
camera devices through LD_PRELOAD, and route them through libcamera to
|
||||
emulate a high-level V4L2 camera device. It shall expose camera device
|
||||
features on a best-effort basis, and aim for the level of features
|
||||
traditionally available from a UVC camera designed for video conferencing.
|
||||
|
||||
Android Camera HAL v3 Compatibility
|
||||
-----------------------------------
|
||||
|
||||
The library API shall expose all the features required to implement an
|
||||
Android Camera HAL v3 on top of libcamera. Some features of the HAL may be
|
||||
omitted as long as they can be implemented separately in the HAL, such as
|
||||
JPEG encoding, or YUV reprocessing.
|
||||
|
||||
|
||||
Camera Stack
|
||||
============
|
||||
|
||||
::
|
||||
|
||||
| a c / +-------------+ +-------------+ +-------------+ +-------------+
|
||||
| p a | | Native | | Framework | | Native | | Android |
|
||||
| p t | | V4L2 | | Application | | libcamera | | Camera |
|
||||
| l i | | Application | | (gstreamer) | | Application | | Framework |
|
||||
| i o \ +-------------+ +-------------+ +-------------+ +-------------+
|
||||
| n ^ ^ ^ ^
|
||||
| | | | |
|
||||
| l a | | | |
|
||||
| i d v v | v
|
||||
| b a / +-------------+ +-------------+ | +-------------+
|
||||
| c p | | V4L2 | | Camera | | | Android |
|
||||
| a t | | Compat. | | Framework | | | Camera |
|
||||
| m a | | | | (gstreamer) | | | HAL |
|
||||
| e t \ +-------------+ +-------------+ | +-------------+
|
||||
| r i ^ ^ | ^
|
||||
| a o | | | |
|
||||
| n | | | |
|
||||
| / | ,................................................
|
||||
| | | ! : Language : !
|
||||
| l f | | ! : Bindings : !
|
||||
| i r | | ! : (optional) : !
|
||||
| b a | | \...............................................'
|
||||
| c m | | | | |
|
||||
| a e | | | | |
|
||||
| m w | v v v v
|
||||
| e o | +----------------------------------------------------------------+
|
||||
| r r | | |
|
||||
| a k | | libcamera |
|
||||
| | | |
|
||||
| \ +----------------------------------------------------------------+
|
||||
| ^ ^ ^
|
||||
| Userspace | | |
|
||||
| ------------------------ | ---------------- | ---------------- | ---------------
|
||||
| Kernel | | |
|
||||
| v v v
|
||||
| +-----------+ +-----------+ +-----------+
|
||||
| | Media | <--> | Video | <--> | V4L2 |
|
||||
| | Device | | Device | | Subdev |
|
||||
| +-----------+ +-----------+ +-----------+
|
||||
|
||||
The camera stack comprises four software layers. From bottom to top:
|
||||
|
||||
* The kernel drivers control the camera hardware and expose a
|
||||
low-level interface to userspace through the Linux kernel V4L2
|
||||
family of APIs (Media Controller API, V4L2 Video Device API and
|
||||
V4L2 Subdev API).
|
||||
|
||||
* The libcamera framework is the core part of the stack. It
|
||||
handles all control of the camera devices in its core component,
|
||||
libcamera, and exposes a native C++ API to upper layers. Optional
|
||||
language bindings allow interfacing to libcamera from other
|
||||
programming languages.
|
||||
|
||||
Those components live in the same source code repository and
|
||||
all together constitute the libcamera framework.
|
||||
|
||||
* The libcamera adaptation is an umbrella term designating the
|
||||
components that interface to libcamera in other frameworks.
|
||||
Notable examples are a V4L2 compatibility layer, a gstreamer
|
||||
libcamera element, and an Android camera HAL implementation based
|
||||
on libcamera.
|
||||
|
||||
Those components can live in the libcamera project source code
|
||||
in separate repositories, or move to their respective project's
|
||||
repository (for instance the gstreamer libcamera element).</p>
|
||||
|
||||
* The applications and upper level frameworks are based on the
|
||||
libcamera framework or libcamera adaptation, and are outside of
|
||||
the scope of the libcamera project.
|
||||
|
||||
|
||||
libcamera Architecture
|
||||
======================
|
||||
|
||||
::
|
||||
|
||||
| ---------------------------< libcamera Public API >---------------------------
|
||||
| ^ ^
|
||||
| | |
|
||||
| v v
|
||||
| +-------------+ +-------------------------------------------------+
|
||||
| | Camera | | Camera Device |
|
||||
| | Devices | | +---------------------------------------------+ |
|
||||
| | Manager | | | Device-Agnostic | |
|
||||
| +-------------+ | | | |
|
||||
| ^ | | +------------------------+ |
|
||||
| | | | | ~~~~~~~~~~~~~~~~~~~~~ |
|
||||
| | | | | { +---------------+ } |
|
||||
| | | | | } | ////Image//// | { |
|
||||
| | | | | <-> | /Processing// | } |
|
||||
| | | | | } | /Algorithms// | { |
|
||||
| | | | | { +---------------+ } |
|
||||
| | | | | ~~~~~~~~~~~~~~~~~~~~~ |
|
||||
| | | | | ======================== |
|
||||
| | | | | +---------------+ |
|
||||
| | | | | | //Pipeline/// | |
|
||||
| | | | | <-> | ///Handler/// | |
|
||||
| | | | | | ///////////// | |
|
||||
| | | +--------------------+ +---------------+ |
|
||||
| | | Device-Specific |
|
||||
| | +-------------------------------------------------+
|
||||
| | ^ ^
|
||||
| | | |
|
||||
| v v v
|
||||
| +--------------------------------------------------------------------+
|
||||
| | Helpers and Support Classes |
|
||||
| | +-------------+ +-------------+ +-------------+ +-------------+ |
|
||||
| | | MC & V4L2 | | Buffers | | Sandboxing | | Plugins | |
|
||||
| | | Support | | Allocator | | IPC | | Manager | |
|
||||
| | +-------------+ +-------------+ +-------------+ +-------------+ |
|
||||
| | +-------------+ +-------------+ |
|
||||
| | | Pipeline | | ... | |
|
||||
| | | Runner | | | |
|
||||
| | +-------------+ +-------------+ |
|
||||
| +--------------------------------------------------------------------+
|
||||
|
|
||||
| /// Device-Specific Components
|
||||
| ~~~ Sandboxing
|
||||
|
||||
While offering a unified API towards upper layers, and presenting
|
||||
itself as a single library, libcamera isn't monolithic. It exposes
|
||||
multiple components through its public API, is built around a set of
|
||||
separate helpers internally, uses device-specific components and can
|
||||
load dynamic plugins.
|
||||
|
||||
Camera Devices Manager
|
||||
The Camera Devices Manager provides a view of available cameras
|
||||
in the system. It performs cold enumeration and runtime camera
|
||||
management, and supports a hotplug notification mechanism in its
|
||||
public API.
|
||||
|
||||
To avoid the cost associated with cold enumeration of all devices
|
||||
at application start, and to arbitrate concurrent access to camera
|
||||
devices, the Camera Devices Manager could later be split to a
|
||||
separate service, possibly with integration in platform-specific
|
||||
device management.
|
||||
|
||||
Camera Device
|
||||
The Camera Device represents a camera device to upper layers. It
|
||||
exposes full control of the device through the public API, and is
|
||||
thus the highest level object exposed by libcamera.
|
||||
|
||||
Camera Device instances are created by the Camera Devices
|
||||
Manager. An optional method to create new instances could be exposed
|
||||
through the public API to speed up initialization when the upper
|
||||
layer knows how to directly address camera devices present in the
|
||||
system.
|
||||
|
||||
Pipeline Handler
|
||||
The Pipeline Handler manages complex pipelines exposed by the kernel drivers
|
||||
through the Media Controller and V4L2 APIs. It abstracts pipeline handling to
|
||||
hide device-specific details to the rest of the library, and implements both
|
||||
pipeline configuration based on stream configuration, and pipeline runtime
|
||||
execution and scheduling when needed by the device.
|
||||
|
||||
This component is device-specific and is part of the libcamera code base. As
|
||||
such it is covered by the same free software license as the rest of libcamera
|
||||
and needs to be contributed upstream by device vendors. The Pipeline Handler
|
||||
lives in the same process as the rest of the library, and has access to all
|
||||
helpers and kernel camera-related devices.</p>
|
||||
|
||||
Image Processing Algorithms
|
||||
Together with the hardware image processing and hardware statistics
|
||||
collection, the Image Processing Algorithms implement 3A (Auto-Exposure,
|
||||
Auto-White Balance and Auto-Focus) and other algorithms. They run on the CPU
|
||||
and interact with the kernel camera devices to control hardware image
|
||||
processing based on the parameters supplied by upper layers, closing the
|
||||
control loop of the ISP.
|
||||
|
||||
This component is device-specific and is loaded as an external plugin. It can
|
||||
be part of the libcamera code base, in which case it is covered by the same
|
||||
license, or provided externally as an open-source or closed-source component.
|
||||
|
||||
The component is sandboxed and can only interact with libcamera through
|
||||
internal APIs specifically marked as such. In particular it will have no
|
||||
direct access to kernel camera devices, and all its accesses to image and
|
||||
metadata will be mediated by dmabuf instances explicitly passed to the
|
||||
component. The component must be prepared to run in a process separate from
|
||||
the main libcamera process, and to have a very restricted view of the system,
|
||||
including no access to networking APIs and limited access to file systems.
|
||||
|
||||
The sandboxing mechanism isn't defined by libcamera. One example
|
||||
implementation will be provided as part of the project, and platforms vendors
|
||||
will be able to provide their own sandboxing mechanism as a plugin.
|
||||
|
||||
libcamera should provide a basic implementation of Image Processing
|
||||
Algorithms, to serve as a reference for the internal API. Device vendors are
|
||||
expected to provide a full-fledged implementation compatible with their
|
||||
Pipeline Handler. One goal of the libcamera project is to create an
|
||||
environment in which the community will be able to compete with the
|
||||
closed-source vendor binaries and develop a high quality open source
|
||||
implementation.
|
||||
|
||||
Helpers and Support Classes
|
||||
While Pipeline Handlers are device-specific, implementations are expected to
|
||||
share code due to usage of identical APIs towards the kernel camera drivers
|
||||
and the Image Processing Algorithms. This includes without limitation handling
|
||||
of the MC and V4L2 APIs, buffer management through dmabuf, and pipeline
|
||||
discovery, configuration and scheduling. Such code will be factored out to
|
||||
helpers when applicable.
|
||||
|
||||
Other parts of libcamera will also benefit from factoring code out to
|
||||
self-contained support classes, even if such code is present only once in the
|
||||
code base, in order to keep the source code clean and easy to read. This
|
||||
should be the case for instance for plugin management.
|
||||
|
||||
|
||||
V4L2 Compatibility Layer
|
||||
------------------------
|
||||
|
||||
V4L2 compatibility is achieved through a shared library that traps all
|
||||
accesses to camera devices and routes them to libcamera to emulate high-level
|
||||
V4L2 camera devices. It is injected in a process address space through
|
||||
LD_PRELOAD and is completely transparent for applications.
|
||||
|
||||
The compatibility layer exposes camera device features on a best-effort basis,
|
||||
and aims for the level of features traditionally available from a UVC camera
|
||||
designed for video conferencing.
|
||||
|
||||
|
||||
Android Camera HAL
|
||||
------------------
|
||||
|
||||
Camera support for Android is achieved through a generic Android
|
||||
camera HAL implementation on top of libcamera. The HAL will implement internally
|
||||
features required by Android and missing from libcamera, such as JPEG encoding
|
||||
support.
|
||||
|
||||
The Android camera HAL implementation will initially target the
|
||||
LIMITED harware level, with support for the FULL level then being gradually
|
||||
implemented.
|
||||
|
|
@ -21,6 +21,7 @@ systems, including traditional Linux distributions, ChromeOS and Android.
|
|||
:caption: Contents:
|
||||
|
||||
coding-style
|
||||
docs
|
||||
contributing
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -44,6 +44,7 @@ if sphinx.found()
|
|||
'coding-style.rst',
|
||||
'conf.py',
|
||||
'contributing.rst',
|
||||
'docs.rst',
|
||||
'index.rst',
|
||||
]
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue