Documentation: Remove libcamera architecture from introduction.rst
The libcamera Architecture section of the introduction is largely a duplicate of the section broken out from docs.rst. Remove it from the introduction.rst file and consolidate anything that wasn't duplicated into libcamera_architecture.rst and feature_requirements.rst. Take the opportunity to also expand the list of Platform Support which is now a bit out of date. Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
This commit is contained in:
Daniel Scally
Laurent Pinchart
parent
c01dfb3650
commit
3fe819eecf
3 changed files with 140 additions and 257 deletions
|
|
@ -90,10 +90,15 @@ on top of the minimum required set for the profile they expose.
|
|||
3A and Image Enhancement Algorithms
|
||||
-----------------------------------
|
||||
|
||||
The camera devices shall implement auto exposure, auto gain and auto white
|
||||
balance. Camera devices that include a focus lens shall implement auto
|
||||
focus. Additional image enhancement algorithms, such as noise reduction or
|
||||
video stabilization, may be implemented.
|
||||
The library shall provide a basic implementation of Image Processing Algorithms
|
||||
to serve as a reference for the internal API. This shall including auto exposure
|
||||
and gain and auto white balance. Camera devices that include a focus lens shall
|
||||
implement auto focus. Additional image enhancement algorithms, such as noise
|
||||
reduction or video stabilization, may be implemented. Device vendors are
|
||||
expected to provide a fully-fledged implementation compatible with their
|
||||
Pipeline Handler. One goal of the libcamera project is to create an environment
|
||||
in which the community will be able to compete with the closed-source vendor
|
||||
biaries and develop a high quality open source implementation.
|
||||
|
||||
All algorithms may be implemented in hardware or firmware outside of the
|
||||
library, or in software in the library. They shall all be controllable by
|
||||
|
|
|
|||
|
|
@ -26,10 +26,8 @@ desirable results from the camera.
|
|||
.. _Media Controller: https://www.linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/mediactl/media-controller.html
|
||||
|
||||
|
||||
In this developers guide, we will explore the internal `Architecture`_ of
|
||||
the libcamera library with its components. The current `Platform Support`_ is
|
||||
detailed, as well as an overview of the `Licensing`_ requirements of the
|
||||
project.
|
||||
In this developers guide the `Licensing`_ requirements of the project are
|
||||
detailed.
|
||||
|
||||
This introduction is followed by a walkthrough tutorial to newcomers wishing to
|
||||
support a new platform with the `Pipeline Handler Writers Guide`_ and for those
|
||||
|
|
@ -41,156 +39,6 @@ provides a tutorial of the key APIs exposed by libcamera.
|
|||
|
||||
.. TODO: Correctly link to the other articles of the guide
|
||||
|
||||
Architecture
|
||||
------------
|
||||
|
||||
While offering a unified API towards upper layers, and presenting itself as a
|
||||
single library, libcamera isn't monolithic. It exposes multiple components
|
||||
through its public API and is built around a set of separate helpers internally.
|
||||
Hardware abstractions are handled through the use of device-specific components
|
||||
where required and dynamically loadable plugins are used to separate image
|
||||
processing algorithms from the core libcamera codebase.
|
||||
|
||||
::
|
||||
|
||||
--------------------------< libcamera Public API >---------------------------
|
||||
^ ^
|
||||
| |
|
||||
v v
|
||||
+-------------+ +---------------------------------------------------+
|
||||
| Camera | | Camera Device |
|
||||
| Manager | | +-----------------------------------------------+ |
|
||||
+-------------+ | | Device-Agnostic | |
|
||||
^ | | | |
|
||||
| | | +--------------------------+ |
|
||||
| | | | ~~~~~~~~~~~~~~~~~~~~~~~ |
|
||||
| | | | { +-----------------+ } |
|
||||
| | | | } | //// Image //// | { |
|
||||
| | | | <-> | / Processing // | } |
|
||||
| | | | } | / Algorithms // | { |
|
||||
| | | | { +-----------------+ } |
|
||||
| | | | ~~~~~~~~~~~~~~~~~~~~~~~ |
|
||||
| | | | ========================== |
|
||||
| | | | +-----------------+ |
|
||||
| | | | | // Pipeline /// | |
|
||||
| | | | <-> | /// Handler /// | |
|
||||
| | | | | /////////////// | |
|
||||
| | +--------------------+ +-----------------+ |
|
||||
| | Device-Specific |
|
||||
| +---------------------------------------------------+
|
||||
| ^ ^
|
||||
| | |
|
||||
v v v
|
||||
+--------------------------------------------------------------------+
|
||||
| Helpers and Support Classes |
|
||||
| +-------------+ +-------------+ +-------------+ +-------------+ |
|
||||
| | MC & V4L2 | | Buffers | | Sandboxing | | Plugins | |
|
||||
| | Support | | Allocator | | IPC | | Manager | |
|
||||
| +-------------+ +-------------+ +-------------+ +-------------+ |
|
||||
| +-------------+ +-------------+ |
|
||||
| | Pipeline | | ... | |
|
||||
| | Runner | | | |
|
||||
| +-------------+ +-------------+ |
|
||||
+--------------------------------------------------------------------+
|
||||
|
||||
/// Device-Specific Components
|
||||
~~~ Sandboxing
|
||||
|
||||
|
||||
Camera Manager
|
||||
The Camera Manager enumerates cameras and instantiates Pipeline Handlers to
|
||||
manage each Camera that libcamera supports. The Camera Manager supports
|
||||
hotplug detection and notification events when supported by the underlying
|
||||
kernel devices.
|
||||
|
||||
There is only ever one instance of the Camera Manager running per application.
|
||||
Each application's instance of the Camera Manager ensures that only a single
|
||||
application can take control of a camera device at once.
|
||||
|
||||
Read the `Camera Manager API`_ documentation for more details.
|
||||
|
||||
.. _Camera Manager API: https://libcamera.org/api-html/classlibcamera_1_1CameraManager.html
|
||||
|
||||
Camera Device
|
||||
The Camera class represents a single item of camera hardware that is capable
|
||||
of producing one or more image streams, and provides the API to interact with
|
||||
the underlying device.
|
||||
|
||||
If a system has multiple instances of the same hardware attached, each has its
|
||||
own instance of the camera class.
|
||||
|
||||
The API exposes full control of the device to upper layers of libcamera through
|
||||
the public API, making it the highest level object libcamera exposes, and the
|
||||
object that all other API operations interact with from configuration to
|
||||
capture.
|
||||
|
||||
Read the `Camera API`_ documentation for more details.
|
||||
|
||||
.. _Camera API: https://libcamera.org/api-html/classlibcamera_1_1Camera.html
|
||||
|
||||
Pipeline Handler
|
||||
The Pipeline Handler manages the complex pipelines exposed by the kernel
|
||||
drivers through the Media Controller and V4L2 APIs. It abstracts pipeline
|
||||
handling to hide device-specific details from the rest of the library, and
|
||||
implements both pipeline configuration based on stream configuration, and
|
||||
pipeline runtime execution and scheduling when needed by the device.
|
||||
|
||||
The Pipeline Handler lives in the same process as the rest of the library, and
|
||||
has access to all helpers and kernel camera-related devices.
|
||||
|
||||
Hardware abstraction is handled by device specific Pipeline Handlers which are
|
||||
derived from the Pipeline Handler base class allowing commonality to be shared
|
||||
among the implementations.
|
||||
|
||||
Derived pipeline handlers create Camera device instances based on the devices
|
||||
they detect and support on the running system, and are responsible for
|
||||
managing the interactions with a camera device.
|
||||
|
||||
More details can be found in the `PipelineHandler API`_ documentation, and the
|
||||
`Pipeline Handler Writers Guide`_.
|
||||
|
||||
.. _PipelineHandler API: https://libcamera.org/api-html/classlibcamera_1_1PipelineHandler.html
|
||||
|
||||
Image Processing Algorithms
|
||||
An image processing algorithm (IPA) component is a loadable plugin that
|
||||
implements 3A (Auto-Exposure, Auto-White Balance, and Auto-Focus) and other
|
||||
algorithms.
|
||||
|
||||
The algorithms run on the CPU and interact with the camera devices through the
|
||||
Pipeline Handler to control hardware image processing based on the parameters
|
||||
supplied by upper layers, maintaining state and closing the control loop
|
||||
of the ISP.
|
||||
|
||||
The component is sandboxed and can only interact with libcamera through the
|
||||
API provided by the Pipeline Handler and an IPA has no direct access to kernel
|
||||
camera devices.
|
||||
|
||||
Open source IPA modules built with libcamera can be run in the same process
|
||||
space as libcamera, however external IPA modules are run in a separate process
|
||||
from the main libcamera process. IPA modules have a restricted view of the
|
||||
system, including no access to networking APIs and limited access to file
|
||||
systems.
|
||||
|
||||
IPA modules are only required for platforms and devices with an ISP controlled
|
||||
by the host CPU. Camera sensors which have an integrated ISP are not
|
||||
controlled through the IPA module.
|
||||
|
||||
Platform Support
|
||||
----------------
|
||||
|
||||
The library currently supports the following hardware platforms specifically
|
||||
with dedicated pipeline handlers:
|
||||
|
||||
- Intel IPU3 (ipu3)
|
||||
- Rockchip RK3399 (rkisp1)
|
||||
- RaspberryPi 3 and 4 (rpi/vc4)
|
||||
|
||||
Furthermore, generic platform support is provided for the following:
|
||||
|
||||
- USB video device class cameras (uvcvideo)
|
||||
- iMX7, Allwinner Sun6i (simple)
|
||||
- Virtual media controller driver for test use cases (vimc)
|
||||
|
||||
Licensing
|
||||
---------
|
||||
|
||||
|
|
|
|||
|
|
@ -5,124 +5,136 @@
|
|||
libcamera Architecture
|
||||
======================
|
||||
|
||||
While offering a unified API towards upper layers, and presenting itself as a
|
||||
single library, libcamera isn't monolithic. It exposes multiple components
|
||||
through its public API and is built around a set of separate helpers internally.
|
||||
Hardware abstractions are handled through the use of device-specific components
|
||||
where required and dynamically loadable plugins are used to separate image
|
||||
processing algorithms from the core libcamera codebase.
|
||||
|
||||
::
|
||||
|
||||
---------------------------< libcamera Public API >---------------------------
|
||||
^ ^
|
||||
| |
|
||||
v v
|
||||
+-------------+ +-------------------------------------------------+
|
||||
| Camera | | Camera Device |
|
||||
| Devices | | +---------------------------------------------+ |
|
||||
| Manager | | | Device-Agnostic | |
|
||||
+-------------+ | | | |
|
||||
^ | | +------------------------+ |
|
||||
| | | | ~~~~~~~~~~~~~~~~~~~~~ |
|
||||
| | | | { +---------------+ } |
|
||||
| | | | } | ////Image//// | { |
|
||||
| | | | <-> | /Processing// | } |
|
||||
| | | | } | /Algorithms// | { |
|
||||
| | | | { +---------------+ } |
|
||||
| | | | ~~~~~~~~~~~~~~~~~~~~~ |
|
||||
| | | | ======================== |
|
||||
| | | | +---------------+ |
|
||||
| | | | | //Pipeline/// | |
|
||||
| | | | <-> | ///Handler/// | |
|
||||
| | | | | ///////////// | |
|
||||
| | +--------------------+ +---------------+ |
|
||||
| | Device-Specific |
|
||||
| +-------------------------------------------------+
|
||||
| ^ ^
|
||||
| | |
|
||||
v v v
|
||||
+--------------------------------------------------------------------+
|
||||
| Helpers and Support Classes |
|
||||
| +-------------+ +-------------+ +-------------+ +-------------+ |
|
||||
| | MC & V4L2 | | Buffers | | Sandboxing | | Plugins | |
|
||||
| | Support | | Allocator | | IPC | | Manager | |
|
||||
| +-------------+ +-------------+ +-------------+ +-------------+ |
|
||||
| +-------------+ +-------------+ |
|
||||
| | Pipeline | | ... | |
|
||||
| | Runner | | | |
|
||||
| +-------------+ +-------------+ |
|
||||
+--------------------------------------------------------------------+
|
||||
--------------------------< libcamera Public API >---------------------------
|
||||
^ ^
|
||||
| |
|
||||
v v
|
||||
+-------------+ +---------------------------------------------------+
|
||||
| Camera | | Camera Device |
|
||||
| Manager | | +-----------------------------------------------+ |
|
||||
+-------------+ | | Device-Agnostic | |
|
||||
^ | | | |
|
||||
| | | +--------------------------+ |
|
||||
| | | | ~~~~~~~~~~~~~~~~~~~~~~~ |
|
||||
| | | | { +-----------------+ } |
|
||||
| | | | } | //// Image //// | { |
|
||||
| | | | <-> | / Processing // | } |
|
||||
| | | | } | / Algorithms // | { |
|
||||
| | | | { +-----------------+ } |
|
||||
| | | | ~~~~~~~~~~~~~~~~~~~~~~~ |
|
||||
| | | | ========================== |
|
||||
| | | | +-----------------+ |
|
||||
| | | | | // Pipeline /// | |
|
||||
| | | | <-> | /// Handler /// | |
|
||||
| | | | | /////////////// | |
|
||||
| | +--------------------+ +-----------------+ |
|
||||
| | Device-Specific |
|
||||
| +---------------------------------------------------+
|
||||
| ^ ^
|
||||
| | |
|
||||
v v v
|
||||
+--------------------------------------------------------------------+
|
||||
| Helpers and Support Classes |
|
||||
| +-------------+ +-------------+ +-------------+ +-------------+ |
|
||||
| | MC & V4L2 | | Buffers | | Sandboxing | | Plugins | |
|
||||
| | Support | | Allocator | | IPC | | Manager | |
|
||||
| +-------------+ +-------------+ +-------------+ +-------------+ |
|
||||
| +-------------+ +-------------+ |
|
||||
| | Pipeline | | ... | |
|
||||
| | Runner | | | |
|
||||
| +-------------+ +-------------+ |
|
||||
+--------------------------------------------------------------------+
|
||||
|
||||
/// Device-Specific Components
|
||||
~~~ Sandboxing
|
||||
/// Device-Specific Components
|
||||
~~~ Sandboxing
|
||||
|
||||
While offering a unified API towards upper layers, and presenting
|
||||
itself as a single library, libcamera isn't monolithic. It exposes
|
||||
multiple components through its public API, is built around a set of
|
||||
separate helpers internally, uses device-specific components and can
|
||||
load dynamic plugins.
|
||||
|
||||
Camera Devices Manager
|
||||
The Camera Devices Manager provides a view of available cameras
|
||||
in the system. It performs cold enumeration and runtime camera
|
||||
management, and supports a hotplug notification mechanism in its
|
||||
public API.
|
||||
Camera Manager
|
||||
The Camera Manager enumerates cameras and instantiates Pipeline Handlers to
|
||||
manage each Camera that libcamera supports. The Camera Manager supports
|
||||
hotplug detection and notification events when supported by the underlying
|
||||
kernel devices.
|
||||
|
||||
To avoid the cost associated with cold enumeration of all devices
|
||||
at application start, and to arbitrate concurrent access to camera
|
||||
devices, the Camera Devices Manager could later be split to a
|
||||
separate service, possibly with integration in platform-specific
|
||||
device management.
|
||||
There is only ever one instance of the Camera Manager running per application.
|
||||
Each application's instance of the Camera Manager ensures that only a single
|
||||
application can take control of a camera device at once.
|
||||
|
||||
Read the `Camera Manager API`_ documentation for more details.
|
||||
|
||||
.. _Camera Manager API: https://libcamera.org/api-html/classlibcamera_1_1CameraManager.html
|
||||
|
||||
Camera Device
|
||||
The Camera Device represents a camera device to upper layers. It
|
||||
exposes full control of the device through the public API, and is
|
||||
thus the highest level object exposed by libcamera.
|
||||
The Camera class represents a single item of camera hardware that is capable
|
||||
of producing one or more image streams, and provides the API to interact with
|
||||
the underlying device.
|
||||
|
||||
Camera Device instances are created by the Camera Devices
|
||||
Manager. An optional function to create new instances could be exposed
|
||||
through the public API to speed up initialization when the upper
|
||||
layer knows how to directly address camera devices present in the
|
||||
system.
|
||||
If a system has multiple instances of the same hardware attached, each has its
|
||||
own instance of the camera class.
|
||||
|
||||
The API exposes full control of the device to upper layers of libcamera through
|
||||
the public API, making it the highest level object libcamera exposes, and the
|
||||
object that all other API operations interact with from configuration to
|
||||
capture.
|
||||
|
||||
Read the `Camera API`_ documentation for more details.
|
||||
|
||||
.. _Camera API: https://libcamera.org/api-html/classlibcamera_1_1Camera.html
|
||||
|
||||
Pipeline Handler
|
||||
The Pipeline Handler manages complex pipelines exposed by the kernel drivers
|
||||
through the Media Controller and V4L2 APIs. It abstracts pipeline handling to
|
||||
hide device-specific details to the rest of the library, and implements both
|
||||
pipeline configuration based on stream configuration, and pipeline runtime
|
||||
execution and scheduling when needed by the device.
|
||||
The Pipeline Handler manages the complex pipelines exposed by the kernel
|
||||
drivers through the Media Controller and V4L2 APIs. It abstracts pipeline
|
||||
handling to hide device-specific details from the rest of the library, and
|
||||
implements both pipeline configuration based on stream configuration, and
|
||||
pipeline runtime execution and scheduling when needed by the device.
|
||||
|
||||
This component is device-specific and is part of the libcamera code base. As
|
||||
such it is covered by the same free software license as the rest of libcamera
|
||||
and needs to be contributed upstream by device vendors. The Pipeline Handler
|
||||
lives in the same process as the rest of the library, and has access to all
|
||||
helpers and kernel camera-related devices.
|
||||
The Pipeline Handler lives in the same process as the rest of the library, and
|
||||
has access to all helpers and kernel camera-related devices.
|
||||
|
||||
Hardware abstraction is handled by device specific Pipeline Handlers which are
|
||||
derived from the Pipeline Handler base class allowing commonality to be shared
|
||||
among the implementations.
|
||||
|
||||
Derived pipeline handlers create Camera device instances based on the devices
|
||||
they detect and support on the running system, and are responsible for
|
||||
managing the interactions with a camera device.
|
||||
|
||||
More details can be found in the `PipelineHandler API`_ documentation, and the
|
||||
:doc:`Pipeline Handler Writers Guide <guides/pipeline-handler>`.
|
||||
|
||||
.. _PipelineHandler API: https://libcamera.org/api-html/classlibcamera_1_1PipelineHandler.html
|
||||
|
||||
Image Processing Algorithms
|
||||
Together with the hardware image processing and hardware statistics
|
||||
collection, the Image Processing Algorithms implement 3A (Auto-Exposure,
|
||||
collection, the Image Processing Algorithms (IPA) implement 3A (Auto-Exposure,
|
||||
Auto-White Balance and Auto-Focus) and other algorithms. They run on the CPU
|
||||
and interact with the kernel camera devices to control hardware image
|
||||
processing based on the parameters supplied by upper layers, closing the
|
||||
control loop of the ISP.
|
||||
and control hardware image processing based on the parameters supplied by
|
||||
upper layers, closing the control loop of the ISP.
|
||||
|
||||
This component is device-specific and is loaded as an external plugin. It can
|
||||
be part of the libcamera code base, in which case it is covered by the same
|
||||
license, or provided externally as an open-source or closed-source component.
|
||||
IPAs are loaded as external plugins named IPA Modules. IPA Modules can be part
|
||||
of the libcamera code base or provided externally by camera vendors as
|
||||
open-source or closed-source components.
|
||||
|
||||
The component is sandboxed and can only interact with libcamera through
|
||||
internal APIs specifically marked as such. In particular it will have no
|
||||
direct access to kernel camera devices, and all its accesses to image and
|
||||
metadata will be mediated by dmabuf instances explicitly passed to the
|
||||
component. The component must be prepared to run in a process separate from
|
||||
the main libcamera process, and to have a very restricted view of the system,
|
||||
including no access to networking APIs and limited access to file systems.
|
||||
Open source IPA Modules built with libcamera are run in the same process space
|
||||
as libcamera. External IPA Modules are run in a separate sandboxed process. In
|
||||
either case, they can only interact with libcamera through the API provided by
|
||||
the Pipeline Handler. They have a restricted view of the system, with no direct
|
||||
access to kernel camera devices, no access to networking APIs, and limited
|
||||
access to file systems. All their accesses to image and metadata are mediated
|
||||
by dmabuf instances explicitly passed by the Pipeline Handler to the IPA
|
||||
Module.
|
||||
|
||||
The sandboxing mechanism isn't defined by libcamera. One example
|
||||
implementation will be provided as part of the project, and platforms vendors
|
||||
will be able to provide their own sandboxing mechanism as a plugin.
|
||||
|
||||
libcamera should provide a basic implementation of Image Processing
|
||||
Algorithms, to serve as a reference for the internal API. Device vendors are
|
||||
expected to provide a full-fledged implementation compatible with their
|
||||
Pipeline Handler. One goal of the libcamera project is to create an
|
||||
environment in which the community will be able to compete with the
|
||||
closed-source vendor binaries and develop a high quality open source
|
||||
implementation.
|
||||
IPA Modules are only required for platforms and devices with an ISP controlled
|
||||
by the host CPU. Camera sensors which have an integrated ISP are not
|
||||
controlled through the IPA Module.
|
||||
|
||||
Helpers and Support Classes
|
||||
While Pipeline Handlers are device-specific, implementations are expected to
|
||||
|
|
@ -136,3 +148,21 @@ Helpers and Support Classes
|
|||
self-contained support classes, even if such code is present only once in the
|
||||
code base, in order to keep the source code clean and easy to read. This
|
||||
should be the case for instance for plugin management.
|
||||
|
||||
Platform Support
|
||||
----------------
|
||||
|
||||
The library currently supports the following hardware platforms specifically
|
||||
with dedicated pipeline handlers:
|
||||
|
||||
- Arm Mali-C55
|
||||
- Intel IPU3 (ipu3)
|
||||
- NXP i.MX8MP (imx8-isi and rkisp1)
|
||||
- RaspberryPi 3, 4 and zero (rpi/vc4)
|
||||
- Rockchip RK3399 (rkisp1)
|
||||
|
||||
Furthermore, generic platform support is provided for the following:
|
||||
|
||||
- USB video device class cameras (uvcvideo)
|
||||
- iMX7, IPU6, Allwinner Sun6i (simple)
|
||||
- Virtual media controller driver for test use cases (vimc)
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue