7ebc19752a
Fix some typos and some literals using ` instead of ``. Signed-off-by: Nícolas F. R. A. Prado <nfraprado@collabora.com> Reviewed-by: Sebastian Fricke <sebastian.fricke@posteo.net> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
319 lines
No EOL
16 KiB
ReStructuredText
319 lines
No EOL
16 KiB
ReStructuredText
.. SPDX-License-Identifier: CC-BY-SA-4.0
|
||
|
||
Developers guide to libcamera
|
||
=============================
|
||
|
||
The Linux kernel handles multimedia devices through the 'Linux media' subsystem
|
||
and provides a set of APIs (application programming interfaces) known
|
||
collectively as V4L2 (`Video for Linux 2`_) and the `Media Controller`_ API
|
||
which provide an interface to interact and control media devices.
|
||
|
||
Included in this subsystem are drivers for camera sensors, CSI2 (Camera
|
||
Serial Interface) recievers, and ISPs (Image Signal Processors)
|
||
|
||
The usage of these drivers to provide a functioning camera stack is a
|
||
responsibility that lies in userspace which is commonly implemented separately
|
||
by vendors without a common architecture or API for application developers.
|
||
|
||
libcamera provides a complete camera stack for Linux based systems to abstract
|
||
functionality desired by camera application developers and process the
|
||
configuration of hardware and image control algorithms requried to obtain
|
||
desireable results from the camera.
|
||
|
||
.. _Video for Linux 2: https://www.linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/v4l2.html
|
||
.. _Media Controller: https://www.linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/mediactl/media-controller.html
|
||
|
||
|
||
In this developers guide, we will explore the `Camera Stack`_ and how it is
|
||
can be visualised at a high level, and explore the internal `Architecture`_ of
|
||
the libcamera library with its components. The current `Platform Support`_ is
|
||
detailed, as well as an overview of the `Licensing`_ requirements of the
|
||
project.
|
||
|
||
This introduction is followed by a walkthrough tutorial to newcomers wishing to
|
||
support a new platform with the `Pipeline Handler Writers Guide`_ and for those
|
||
looking to make use of the libcamera native API an `Application Writers Guide`_
|
||
provides a tutorial of the key APIs exposed by libcamera.
|
||
|
||
.. _Pipeline Handler Writers Guide: pipeline-handler.html
|
||
.. _Application Writers Guide: application-developer.html
|
||
|
||
.. TODO: Correctly link to the other articles of the guide
|
||
|
||
Camera Stack
|
||
------------
|
||
|
||
The libcamera library is implemented in userspace, and makes use of underlying
|
||
kernel drivers that directly interact with hardware.
|
||
|
||
Applications can make use of libcamera through the native `libcamera API`_'s or
|
||
through an adaptation layer integrating libcamera into a larger framework.
|
||
|
||
.. _libcamera API: https://www.libcamera.org/api-html/index.html
|
||
|
||
::
|
||
|
||
Application Layer
|
||
/ +--------------+ +--------------+ +--------------+ +--------------+
|
||
| | Native | | Framework | | Native | | Android |
|
||
| | V4L2 | | Application | | libcamera | | Camera |
|
||
| | Application | | (gstreamer) | | Application | | Framework |
|
||
\ +--------------+ +--------------+ +--------------+ +--------------+
|
||
|
||
^ ^ ^ ^
|
||
| | | |
|
||
| | | |
|
||
v v | v
|
||
Adaptation Layer |
|
||
/ +--------------+ +--------------+ | +--------------+
|
||
| | V4L2 | | gstreamer | | | Android |
|
||
| | Compatability| | element | | | Camera |
|
||
| | (preload) | |(libcamerasrc)| | | HAL |
|
||
\ +--------------+ +--------------+ | +--------------+
|
||
|
|
||
^ ^ | ^
|
||
| | | |
|
||
| | | |
|
||
v v v v
|
||
libcamera Framework
|
||
/ +--------------------------------------------------------------------+
|
||
| | |
|
||
| | libcamera |
|
||
| | |
|
||
\ +--------------------------------------------------------------------+
|
||
|
||
^ ^ ^
|
||
Userspace | | |
|
||
--------------------- | ---------------- | ---------------- | ---------------
|
||
Kernel | | |
|
||
v v v
|
||
|
||
+-----------+ +-----------+ +-----------+
|
||
| Media | <--> | Video | <--> | V4L2 |
|
||
| Device | | Device | | Subdev |
|
||
+-----------+ +-----------+ +-----------+
|
||
|
||
The camera stack comprises of four software layers. From bottom to top:
|
||
|
||
* The kernel drivers control the camera hardware and expose a low-level
|
||
interface to userspace through the Linux kernel V4L2 family of APIs
|
||
(Media Controller API, V4L2 Video Device API and V4L2 Subdev API).
|
||
|
||
* The libcamera framework is the core part of the stack. It handles all control
|
||
of the camera devices in its core component, libcamera, and exposes a native
|
||
C++ API to upper layers.
|
||
|
||
* The libcamera adaptation layer is an umbrella term designating the components
|
||
that interface to libcamera in other frameworks. Notable examples are the V4L2
|
||
compatibility layer, the gstreamer libcamera element, and the Android camera
|
||
HAL implementation based on libcamera which are provided as a part of the
|
||
libcamera project.
|
||
|
||
* The applications and upper level frameworks are based on the libcamera
|
||
framework or libcamera adaptation, and are outside of the scope of the
|
||
libcamera project, however example native applications (cam, qcam) are
|
||
provided for testing.
|
||
|
||
|
||
V4L2 Compatibility Layer
|
||
V4L2 compatibility is achieved through a shared library that traps all
|
||
accesses to camera devices and routes them to libcamera to emulate high-level
|
||
V4L2 camera devices. It is injected in a process address space through
|
||
``LD_PRELOAD`` and is completely transparent for applications.
|
||
|
||
The compatibility layer exposes camera device features on a best-effort basis,
|
||
and aims for the level of features traditionally available from a UVC camera
|
||
designed for video conferencing.
|
||
|
||
Android Camera HAL
|
||
Camera support for Android is achieved through a generic Android camera HAL
|
||
implementation on top of libcamera. The HAL implements features required by
|
||
Android and out of scope from libcamera, such as JPEG encoding support.
|
||
|
||
This component is used to provide support for ChromeOS platforms
|
||
|
||
GStreamer element (gstlibcamerasrc)
|
||
A `GStreamer element`_ is provided to allow capture from libcamera supported
|
||
devices through GStreamer pipelines, and connect to other elements for further
|
||
processing.
|
||
|
||
Development of this element is ongoing and is limited to a single stream.
|
||
|
||
Native libcamera API
|
||
Applications can make use of the libcamera API directly using the C++
|
||
API. An example application and walkthrough using the libcamera API can be
|
||
followed in the `Application Writers Guide`_
|
||
|
||
.. _GStreamer element: https://gstreamer.freedesktop.org/documentation/application-development/basics/elements.html
|
||
|
||
Architecture
|
||
------------
|
||
|
||
While offering a unified API towards upper layers, and presenting itself as a
|
||
single library, libcamera isn’t monolithic. It exposes multiple components
|
||
through its public API and is built around a set of separate helpers internally.
|
||
Hardware abstractractions are handled through the use of device-specific
|
||
components where required and dynamically loadable plugins are used to separate
|
||
image processing algorithms from the core libcamera codebase.
|
||
|
||
::
|
||
|
||
--------------------------< libcamera Public API >---------------------------
|
||
^ ^
|
||
| |
|
||
v v
|
||
+-------------+ +---------------------------------------------------+
|
||
| Camera | | Camera Device |
|
||
| Manager | | +-----------------------------------------------+ |
|
||
+-------------+ | | Device-Agnostic | |
|
||
^ | | | |
|
||
| | | +--------------------------+ |
|
||
| | | | ~~~~~~~~~~~~~~~~~~~~~~~ |
|
||
| | | | { +-----------------+ } |
|
||
| | | | } | //// Image //// | { |
|
||
| | | | <-> | / Processing // | } |
|
||
| | | | } | / Algorithms // | { |
|
||
| | | | { +-----------------+ } |
|
||
| | | | ~~~~~~~~~~~~~~~~~~~~~~~ |
|
||
| | | | ========================== |
|
||
| | | | +-----------------+ |
|
||
| | | | | // Pipeline /// | |
|
||
| | | | <-> | /// Handler /// | |
|
||
| | | | | /////////////// | |
|
||
| | +--------------------+ +-----------------+ |
|
||
| | Device-Specific |
|
||
| +---------------------------------------------------+
|
||
| ^ ^
|
||
| | |
|
||
v v v
|
||
+--------------------------------------------------------------------+
|
||
| Helpers and Support Classes |
|
||
| +-------------+ +-------------+ +-------------+ +-------------+ |
|
||
| | MC & V4L2 | | Buffers | | Sandboxing | | Plugins | |
|
||
| | Support | | Allocator | | IPC | | Manager | |
|
||
| +-------------+ +-------------+ +-------------+ +-------------+ |
|
||
| +-------------+ +-------------+ |
|
||
| | Pipeline | | ... | |
|
||
| | Runner | | | |
|
||
| +-------------+ +-------------+ |
|
||
+--------------------------------------------------------------------+
|
||
|
||
/// Device-Specific Components
|
||
~~~ Sandboxing
|
||
|
||
|
||
Camera Manager
|
||
The Camera Manager enumerates cameras and instantiates Pipeline Handlers to
|
||
manage each Camera that libcamera supports. The Camera Manager supports
|
||
hotplug detection and notification events when supported by the underlying
|
||
kernel devices.
|
||
|
||
There is only ever one instance of the Camera Manager running per application.
|
||
Each application's instance of the Camera Manager ensures that only a single
|
||
application can take control of a camera device at once.
|
||
|
||
Read the `Camera Manager API`_ documentation for more details.
|
||
|
||
.. _Camera Manager API: http://libcamera.org/api-html/classlibcamera_1_1CameraManager.html
|
||
|
||
Camera Device
|
||
The Camera class represents a single item of camera hardware that is capable
|
||
of producing one or more image streams, and provides the API to interact with
|
||
the underlying device.
|
||
|
||
If a system has multiple instances of the same hardware attached, each has it's
|
||
own instance of the camera class.
|
||
|
||
The API exposes full control of the device to upper layers of libcamera through
|
||
the public API, making it the highest level object libcamera exposes, and the
|
||
object that all other API operations interact with from configuration to
|
||
capture.
|
||
|
||
Read the `Camera API`_ documentation for more details.
|
||
|
||
.. _Camera API: http://libcamera.org/api-html/classlibcamera_1_1Camera.html
|
||
|
||
Pipeline Handler
|
||
The Pipeline Handler manages the complex pipelines exposed by the kernel
|
||
drivers through the Media Controller and V4L2 APIs. It abstracts pipeline
|
||
handling to hide device-specific details from the rest of the library, and
|
||
implements both pipeline configuration based on stream configuration, and
|
||
pipeline runtime execution and scheduling when needed by the device.
|
||
|
||
The Pipeline Handler lives in the same process as the rest of the library, and
|
||
has access to all helpers and kernel camera-related devices.
|
||
|
||
Hardware abstraction is handled by device specific Pipeline Handlers which are
|
||
derived from the Pipeline Handler base class allowing commonality to be shared
|
||
among the implementations.
|
||
|
||
Derived pipeline handlers create Camera device instances based on the devices
|
||
they detect and support on the running system, and are responsible for
|
||
managing the interactions with a camera device.
|
||
|
||
More details can be found in the `PipelineHandler API`_ documentation, and the
|
||
`Pipeline Handler Writers Guide`_.
|
||
|
||
.. _PipelineHandler API: http://libcamera.org/api-html/classlibcamera_1_1PipelineHandler.html
|
||
|
||
Image Processing Algorithms
|
||
An image processing algorithm (IPA) component is a loadable plugin that
|
||
implements 3A (Auto-Exposure, Auto-White Balance, and Auto-Focus) and other
|
||
algorithms.
|
||
|
||
The algorithms run on the CPU and interact with the camera devices through the
|
||
Pipeline Handler to control hardware image processing based on the parameters
|
||
supplied by upper layers, maintaining state and closing the control loop
|
||
of the ISP.
|
||
|
||
The component is sandboxed and can only interact with libcamera through the
|
||
API provided by the Pipeline Handler and an IPA has no direct access to kernel
|
||
camera devices.
|
||
|
||
Open source IPA modules built with libcamera can be run in the same process
|
||
space as libcamera, however external IPA modules are run in a separate process
|
||
from the main libcamera process. IPA modules have a restricted view of the
|
||
system, including no access to networking APIs and limited access to file
|
||
systems.
|
||
|
||
IPA modules are only required for platforms and devices with an ISP controlled
|
||
by the host CPU. Camera sensors which have an integrated ISP are not
|
||
controlled through the IPA module.
|
||
|
||
Platform Support
|
||
----------------
|
||
|
||
The library currently supports the following hardware platforms specifically
|
||
with dedicated pipeline handlers:
|
||
|
||
- Intel IPU3 (ipu3)
|
||
- Rockchip RK3399 (rkisp1)
|
||
- RaspberryPi 3 and 4 (raspberrypi)
|
||
|
||
Furthermore, generic platform support is provided for the following:
|
||
|
||
- USB video device class cameras (uvcvideo)
|
||
- iMX7, Allwinner Sun6i (simple)
|
||
- Virtual media controller driver for test use cases (vimc)
|
||
|
||
Licensing
|
||
---------
|
||
|
||
The libcamera core, is covered by the `LGPL-2.1-or-later`_ license. Pipeline
|
||
Handlers are a part of the libcamera code base and need to be contributed
|
||
upstream by device vendors. IPA modules included in libcamera are covered by a
|
||
free software license, however third-parties may develop IPA modules outside of
|
||
libcamera and distribute them under a closed-source license, provided they do
|
||
not include source code from the libcamera project.
|
||
|
||
The libcamera project itself contains multiple libraries, applications and
|
||
utilities. Licenses are expressed through SPDX tags in text-based files that
|
||
support comments, and through the .reuse/dep5 file otherwise. A copy of all
|
||
licenses are stored in the LICENSES directory, and a full summary of the
|
||
licensing used throughout the project can be found in the COPYING.rst document.
|
||
|
||
Applications which link dynamically against libcamera and use only the public
|
||
API are an independent work of the authors and have no license restrictions
|
||
imposed upon them from libcamera.
|
||
|
||
.. _LGPL-2.1-or-later: https://spdx.org/licenses/LGPL-2.1-or-later.html |