Mirror/libcamera
1
0
Fork 0
mirror of https://git.libcamera.org/libcamera/libcamera.git synced 2025-07-12 14:59:44 +03:00
Code Issues Projects Releases Packages Wiki Activity Actions

Documentation: Remove libcamera architecture from introduction.rst

Browse source 
The libcamera Architecture section of the introduction is largely a
duplicate of the section broken out from docs.rst. Remove it from the
introduction.rst file and consolidate anything that wasn't duplicated
into libcamera_architecture.rst and feature_requirements.rst. Take the
opportunity to also expand the list of Platform Support which is now a
bit out of date.

Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
This commit is contained in:
Daniel Scally 2024-08-20 14:07:36 +01:00 • committed by Laurent Pinchart
parent c01dfb3650
commit 3fe819eecf
3 changed files with 140 additions and 257 deletions
Download patch file Download diff file Expand all files Collapse all files

13
Documentation/feature_requirements.rst
View file

@ -90,10 +90,15 @@ on top of the minimum required set for the profile they expose.
3A and Image Enhancement Algorithms 3A and Image Enhancement Algorithms
----------------------------------- -----------------------------------
The camera devices shall implement auto exposure, auto gain and auto white The library shall provide a basic implementation of Image Processing Algorithms
balance. Camera devices that include a focus lens shall implement auto to serve as a reference for the internal API. This shall including auto exposure
focus. Additional image enhancement algorithms, such as noise reduction or and gain and auto white balance. Camera devices that include a focus lens shall
video stabilization, may be implemented. implement auto focus. Additional image enhancement algorithms, such as noise
reduction or video stabilization, may be implemented. Device vendors are
expected to provide a fully-fledged implementation compatible with their
Pipeline Handler. One goal of the libcamera project is to create an environment
in which the community will be able to compete with the closed-source vendor
biaries and develop a high quality open source implementation.
All algorithms may be implemented in hardware or firmware outside of the All algorithms may be implemented in hardware or firmware outside of the
library, or in software in the library. They shall all be controllable by library, or in software in the library. They shall all be controllable by

156
Documentation/guides/introduction.rst
View file

@ -26,10 +26,8 @@ desirable results from the camera.
.. _Media Controller: https://www.linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/mediactl/media-controller.html .. _Media Controller: https://www.linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/mediactl/media-controller.html
In this developers guide, we will explore the internal `Architecture`_ of In this developers guide the `Licensing`_ requirements of the project are
the libcamera library with its components. The current `Platform Support`_ is detailed.
detailed, as well as an overview of the `Licensing`_ requirements of the
project.
This introduction is followed by a walkthrough tutorial to newcomers wishing to This introduction is followed by a walkthrough tutorial to newcomers wishing to
support a new platform with the `Pipeline Handler Writers Guide`_ and for those support a new platform with the `Pipeline Handler Writers Guide`_ and for those
@ -41,156 +39,6 @@ provides a tutorial of the key APIs exposed by libcamera.
.. TODO: Correctly link to the other articles of the guide .. TODO: Correctly link to the other articles of the guide
Architecture
------------
While offering a unified API towards upper layers, and presenting itself as a
single library, libcamera isn't monolithic. It exposes multiple components
through its public API and is built around a set of separate helpers internally.
Hardware abstractions are handled through the use of device-specific components
where required and dynamically loadable plugins are used to separate image
processing algorithms from the core libcamera codebase.
::
--------------------------< libcamera Public API >---------------------------
^ ^
| |
v v
+-------------+ +---------------------------------------------------+
| Camera | | Camera Device |
| Manager | | +-----------------------------------------------+ |
+-------------+ | | Device-Agnostic | |
^ | | | |
| | | +--------------------------+ |
| | | | ~~~~~~~~~~~~~~~~~~~~~~~ |
| | | | { +-----------------+ } |
| | | | } | //// Image //// | { |
| | | | <-> | / Processing // | } |
| | | | } | / Algorithms // | { |
| | | | { +-----------------+ } |
| | | | ~~~~~~~~~~~~~~~~~~~~~~~ |
| | | | ========================== |
| | | | +-----------------+ |
| | | | | // Pipeline /// | |
| | | | <-> | /// Handler /// | |
| | | | | /////////////// | |
| | +--------------------+ +-----------------+ |
| | Device-Specific |
| +---------------------------------------------------+
| ^ ^
| | |
v v v
+--------------------------------------------------------------------+
| Helpers and Support Classes |
| +-------------+ +-------------+ +-------------+ +-------------+ |
| | MC & V4L2 | | Buffers | | Sandboxing | | Plugins | |
| | Support | | Allocator | | IPC | | Manager | |
| +-------------+ +-------------+ +-------------+ +-------------+ |
| +-------------+ +-------------+ |
| | Pipeline | | ... | |
| | Runner | | | |
| +-------------+ +-------------+ |
+--------------------------------------------------------------------+
/// Device-Specific Components
~~~ Sandboxing
Camera Manager
The Camera Manager enumerates cameras and instantiates Pipeline Handlers to
manage each Camera that libcamera supports. The Camera Manager supports
hotplug detection and notification events when supported by the underlying
kernel devices.
There is only ever one instance of the Camera Manager running per application.
Each application's instance of the Camera Manager ensures that only a single
application can take control of a camera device at once.
Read the `Camera Manager API`_ documentation for more details.
.. _Camera Manager API: https://libcamera.org/api-html/classlibcamera_1_1CameraManager.html
Camera Device
The Camera class represents a single item of camera hardware that is capable
of producing one or more image streams, and provides the API to interact with
the underlying device.
If a system has multiple instances of the same hardware attached, each has its
own instance of the camera class.
The API exposes full control of the device to upper layers of libcamera through
the public API, making it the highest level object libcamera exposes, and the
object that all other API operations interact with from configuration to
capture.
Read the `Camera API`_ documentation for more details.
.. _Camera API: https://libcamera.org/api-html/classlibcamera_1_1Camera.html
Pipeline Handler
The Pipeline Handler manages the complex pipelines exposed by the kernel
drivers through the Media Controller and V4L2 APIs. It abstracts pipeline
handling to hide device-specific details from the rest of the library, and
implements both pipeline configuration based on stream configuration, and
pipeline runtime execution and scheduling when needed by the device.
The Pipeline Handler lives in the same process as the rest of the library, and
has access to all helpers and kernel camera-related devices.
Hardware abstraction is handled by device specific Pipeline Handlers which are
derived from the Pipeline Handler base class allowing commonality to be shared
among the implementations.
Derived pipeline handlers create Camera device instances based on the devices
they detect and support on the running system, and are responsible for
managing the interactions with a camera device.
More details can be found in the `PipelineHandler API`_ documentation, and the
`Pipeline Handler Writers Guide`_.
.. _PipelineHandler API: https://libcamera.org/api-html/classlibcamera_1_1PipelineHandler.html
Image Processing Algorithms
An image processing algorithm (IPA) component is a loadable plugin that
implements 3A (Auto-Exposure, Auto-White Balance, and Auto-Focus) and other
algorithms.
The algorithms run on the CPU and interact with the camera devices through the
Pipeline Handler to control hardware image processing based on the parameters
supplied by upper layers, maintaining state and closing the control loop
of the ISP.
The component is sandboxed and can only interact with libcamera through the
API provided by the Pipeline Handler and an IPA has no direct access to kernel
camera devices.
Open source IPA modules built with libcamera can be run in the same process
space as libcamera, however external IPA modules are run in a separate process
from the main libcamera process. IPA modules have a restricted view of the
system, including no access to networking APIs and limited access to file
systems.
IPA modules are only required for platforms and devices with an ISP controlled
by the host CPU. Camera sensors which have an integrated ISP are not
controlled through the IPA module.
Platform Support
----------------
The library currently supports the following hardware platforms specifically
with dedicated pipeline handlers:
- Intel IPU3 (ipu3)
- Rockchip RK3399 (rkisp1)
- RaspberryPi 3 and 4 (rpi/vc4)
Furthermore, generic platform support is provided for the following:
- USB video device class cameras (uvcvideo)
- iMX7, Allwinner Sun6i (simple)
- Virtual media controller driver for test use cases (vimc)
Licensing Licensing
--------- ---------

228
Documentation/libcamera_architecture.rst
View file

@ -5,124 +5,136 @@
libcamera Architecture libcamera Architecture
====================== ======================
While offering a unified API towards upper layers, and presenting itself as a
single library, libcamera isn't monolithic. It exposes multiple components
through its public API and is built around a set of separate helpers internally.
Hardware abstractions are handled through the use of device-specific components
where required and dynamically loadable plugins are used to separate image
processing algorithms from the core libcamera codebase.
:: ::
---------------------------< libcamera Public API >--------------------------- --------------------------< libcamera Public API >---------------------------
^ ^ ^ ^
| | | |
v v v v
+-------------+ +-------------------------------------------------+ +-------------+ +---------------------------------------------------+
| Camera | | Camera Device | | Camera | | Camera Device |
| Devices | | +---------------------------------------------+ | | Manager | | +-----------------------------------------------+ |
| Manager | | | Device-Agnostic | | +-------------+ | | Device-Agnostic | |
+-------------+ | | | | ^ | | | |
^ | | +------------------------+ | | | | +--------------------------+ |
| | | | ~~~~~~~~~~~~~~~~~~~~~ | | | | | ~~~~~~~~~~~~~~~~~~~~~~~ |
| | | | { +---------------+ } | | | | | { +-----------------+ } |
| | | | } | ////Image//// | { | | | | | } | //// Image //// | { |
| | | | <-> | /Processing// | } | | | | | <-> | / Processing // | } |
| | | | } | /Algorithms// | { | | | | | } | / Algorithms // | { |
| | | | { +---------------+ } | | | | | { +-----------------+ } |
| | | | ~~~~~~~~~~~~~~~~~~~~~ | | | | | ~~~~~~~~~~~~~~~~~~~~~~~ |
| | | | ======================== | | | | | ========================== |
| | | | +---------------+ | | | | | +-----------------+ |
| | | | | //Pipeline/// | | | | | | | // Pipeline /// | |
| | | | <-> | ///Handler/// | | | | | | <-> | /// Handler /// | |
| | | | | ///////////// | | | | | | | /////////////// | |
| | +--------------------+ +---------------+ | | | +--------------------+ +-----------------+ |
| | Device-Specific | | | Device-Specific |
| +-------------------------------------------------+ | +---------------------------------------------------+
| ^ ^ | ^ ^
| | | | | |
v v v v v v
+--------------------------------------------------------------------+ +--------------------------------------------------------------------+
| Helpers and Support Classes | | Helpers and Support Classes |
| +-------------+ +-------------+ +-------------+ +-------------+ | | +-------------+ +-------------+ +-------------+ +-------------+ |
| | MC & V4L2 | | Buffers | | Sandboxing | | Plugins | | | | MC & V4L2 | | Buffers | | Sandboxing | | Plugins | |
| | Support | | Allocator | | IPC | | Manager | | | | Support | | Allocator | | IPC | | Manager | |
| +-------------+ +-------------+ +-------------+ +-------------+ | | +-------------+ +-------------+ +-------------+ +-------------+ |
| +-------------+ +-------------+ | | +-------------+ +-------------+ |
| | Pipeline | | ... | | | | Pipeline | | ... | |
| | Runner | | | | | | Runner | | | |
| +-------------+ +-------------+ | | +-------------+ +-------------+ |
+--------------------------------------------------------------------+ +--------------------------------------------------------------------+
/// Device-Specific Components /// Device-Specific Components
~~~ Sandboxing ~~~ Sandboxing
While offering a unified API towards upper layers, and presenting
itself as a single library, libcamera isn't monolithic. It exposes
multiple components through its public API, is built around a set of
separate helpers internally, uses device-specific components and can
load dynamic plugins.
Camera Devices Manager Camera Manager
The Camera Devices Manager provides a view of available cameras The Camera Manager enumerates cameras and instantiates Pipeline Handlers to
in the system. It performs cold enumeration and runtime camera manage each Camera that libcamera supports. The Camera Manager supports
management, and supports a hotplug notification mechanism in its hotplug detection and notification events when supported by the underlying
public API. kernel devices.
To avoid the cost associated with cold enumeration of all devices There is only ever one instance of the Camera Manager running per application.
at application start, and to arbitrate concurrent access to camera Each application's instance of the Camera Manager ensures that only a single
devices, the Camera Devices Manager could later be split to a application can take control of a camera device at once.
separate service, possibly with integration in platform-specific
device management. Read the `Camera Manager API`_ documentation for more details.
.. _Camera Manager API: https://libcamera.org/api-html/classlibcamera_1_1CameraManager.html
Camera Device Camera Device
The Camera Device represents a camera device to upper layers. It The Camera class represents a single item of camera hardware that is capable
exposes full control of the device through the public API, and is of producing one or more image streams, and provides the API to interact with
thus the highest level object exposed by libcamera. the underlying device.
Camera Device instances are created by the Camera Devices If a system has multiple instances of the same hardware attached, each has its
Manager. An optional function to create new instances could be exposed own instance of the camera class.
through the public API to speed up initialization when the upper
layer knows how to directly address camera devices present in the The API exposes full control of the device to upper layers of libcamera through
system. the public API, making it the highest level object libcamera exposes, and the
object that all other API operations interact with from configuration to
capture.
Read the `Camera API`_ documentation for more details.
.. _Camera API: https://libcamera.org/api-html/classlibcamera_1_1Camera.html
Pipeline Handler Pipeline Handler
The Pipeline Handler manages complex pipelines exposed by the kernel drivers The Pipeline Handler manages the complex pipelines exposed by the kernel
through the Media Controller and V4L2 APIs. It abstracts pipeline handling to drivers through the Media Controller and V4L2 APIs. It abstracts pipeline
hide device-specific details to the rest of the library, and implements both handling to hide device-specific details from the rest of the library, and
pipeline configuration based on stream configuration, and pipeline runtime implements both pipeline configuration based on stream configuration, and
execution and scheduling when needed by the device. pipeline runtime execution and scheduling when needed by the device.
This component is device-specific and is part of the libcamera code base. As The Pipeline Handler lives in the same process as the rest of the library, and
such it is covered by the same free software license as the rest of libcamera has access to all helpers and kernel camera-related devices.
and needs to be contributed upstream by device vendors. The Pipeline Handler
lives in the same process as the rest of the library, and has access to all Hardware abstraction is handled by device specific Pipeline Handlers which are
helpers and kernel camera-related devices. derived from the Pipeline Handler base class allowing commonality to be shared
among the implementations.
Derived pipeline handlers create Camera device instances based on the devices
they detect and support on the running system, and are responsible for
managing the interactions with a camera device.
More details can be found in the `PipelineHandler API`_ documentation, and the
:doc:`Pipeline Handler Writers Guide <guides/pipeline-handler>`.
.. _PipelineHandler API: https://libcamera.org/api-html/classlibcamera_1_1PipelineHandler.html
Image Processing Algorithms Image Processing Algorithms
Together with the hardware image processing and hardware statistics Together with the hardware image processing and hardware statistics
collection, the Image Processing Algorithms implement 3A (Auto-Exposure, collection, the Image Processing Algorithms (IPA) implement 3A (Auto-Exposure,
Auto-White Balance and Auto-Focus) and other algorithms. They run on the CPU Auto-White Balance and Auto-Focus) and other algorithms. They run on the CPU
and interact with the kernel camera devices to control hardware image and control hardware image processing based on the parameters supplied by
processing based on the parameters supplied by upper layers, closing the upper layers, closing the control loop of the ISP.
control loop of the ISP.
This component is device-specific and is loaded as an external plugin. It can IPAs are loaded as external plugins named IPA Modules. IPA Modules can be part
be part of the libcamera code base, in which case it is covered by the same of the libcamera code base or provided externally by camera vendors as
license, or provided externally as an open-source or closed-source component. open-source or closed-source components.
The component is sandboxed and can only interact with libcamera through Open source IPA Modules built with libcamera are run in the same process space
internal APIs specifically marked as such. In particular it will have no as libcamera. External IPA Modules are run in a separate sandboxed process. In
direct access to kernel camera devices, and all its accesses to image and either case, they can only interact with libcamera through the API provided by
metadata will be mediated by dmabuf instances explicitly passed to the the Pipeline Handler. They have a restricted view of the system, with no direct
component. The component must be prepared to run in a process separate from access to kernel camera devices, no access to networking APIs, and limited
the main libcamera process, and to have a very restricted view of the system, access to file systems. All their accesses to image and metadata are mediated
including no access to networking APIs and limited access to file systems. by dmabuf instances explicitly passed by the Pipeline Handler to the IPA
Module.
The sandboxing mechanism isn't defined by libcamera. One example IPA Modules are only required for platforms and devices with an ISP controlled
implementation will be provided as part of the project, and platforms vendors by the host CPU. Camera sensors which have an integrated ISP are not
will be able to provide their own sandboxing mechanism as a plugin. controlled through the IPA Module.
libcamera should provide a basic implementation of Image Processing
Algorithms, to serve as a reference for the internal API. Device vendors are
expected to provide a full-fledged implementation compatible with their
Pipeline Handler. One goal of the libcamera project is to create an
environment in which the community will be able to compete with the
closed-source vendor binaries and develop a high quality open source
implementation.
Helpers and Support Classes Helpers and Support Classes
While Pipeline Handlers are device-specific, implementations are expected to While Pipeline Handlers are device-specific, implementations are expected to
@ -136,3 +148,21 @@ Helpers and Support Classes
self-contained support classes, even if such code is present only once in the self-contained support classes, even if such code is present only once in the
code base, in order to keep the source code clean and easy to read. This code base, in order to keep the source code clean and easy to read. This
should be the case for instance for plugin management. should be the case for instance for plugin management.
Platform Support
----------------
The library currently supports the following hardware platforms specifically
with dedicated pipeline handlers:
- Arm Mali-C55
- Intel IPU3 (ipu3)
- NXP i.MX8MP (imx8-isi and rkisp1)
- RaspberryPi 3, 4 and zero (rpi/vc4)
- Rockchip RK3399 (rkisp1)
Furthermore, generic platform support is provided for the following:
- USB video device class cameras (uvcvideo)
- iMX7, IPU6, Allwinner Sun6i (simple)
- Virtual media controller driver for test use cases (vimc)