Documentation: Introduce Camera Sensor Model

Introduce a documentation page about the 'camera sensor model'
implemented by libcamera.

The camera sensor model serves to provide to applications a reference
description of the processing steps that take place in a camera sensor
in order to precisely control the sensor configuration through the
forthcoming SensorConfiguration class.

Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Reviewed-by: Naushir Patuck <naush@raspberrypi.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

.reuse/dep5

@@ -3,6 +3,12 @@ Upstream-Name: libcamera
 Upstream-Contact: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
 Source: https://git.libcamera.org/libcamera/libcamera.git/
 
+Files: Documentation/binning.svg
+       Documentation/camera-sensor-model.rst
+       Documentation/sensor_model.svg
+Copyright: Copyright 2023 Ideas On Board Oy
+License: CC-BY-SA-4.0
+
 Files: Documentation/theme/static/search.png
 Copyright: 2022 Fonticons, Inc.
 License: CC-BY-4.0

Documentation/camera-sensor-model.rst

@@ -0,0 +1,174 @@
+.. SPDX-License-Identifier: CC-BY-SA-4.0
+
+.. _camera-sensor-model:
+
+.. todo: Move to Doxygen-generated documentation
+
+The libcamera camera sensor model
+=================================
+
+libcamera defines an abstract camera sensor model in order to provide
+a description of each of the processing steps that result in image data being
+sent on the media bus and that form the image stream delivered to applications.
+
+Applications should use the abstract camera sensor model defined here to
+precisely control the operations of the camera sensor.
+
+The libcamera camera sensor model targets image sensors producing frames in
+RAW format, delivered through a MIPI CSI-2 compliant bus implementation.
+
+The abstract sensor model maps libcamera components to the characteristics and
+operations of an image sensor, and serves as a reference to model the libcamera
+CameraSensor class and SensorConfiguration classes and operations.
+
+In order to control the configuration of the camera sensor through the
+SensorConfiguration class, applications should understand this model and map it
+to the combination of image sensor and kernel driver in use.
+
+The camera sensor model defined here is based on the *MIPI CCS specification*,
+particularly on *Section 8.2 - Image readout* of *Chapter 8 - Video Timings*.
+
+
+Glossary
+--------
+
+.. glossary::
+
+   Pixel array
+      The full grid of pixels, active and inactive ones
+
+   Pixel array active area
+      The portion(s) of the pixel array that contains valid and readable pixels;
+      corresponds to the libcamera properties::PixelArrayActiveAreas
+
+   Analog crop rectangle
+      The portion of the *pixel array active area* which is read out and passed
+      to further processing stages
+
+   Subsampling
+      Pixel processing techniques that reduce the image size by binning or by
+      skipping adjacent pixels
+
+   Digital crop
+      Crop of the sub-sampled image data before scaling
+
+   Frame output
+      The frame (image) as output on the media bus by the camera sensor
+
+Camera sensor model
+-------------------
+
+The abstract sensor model is described in the following diagram.
+
+.. figure:: sensor_model.svg
+
+
+1. The sensor reads pixels from the *pixel array*. The pixels being read out are
+   selected by the *analog crop rectangle*.
+
+2. The pixels can be subsampled to reduce the image size without affecting the
+   field of view. Two subsampling techniques can be used:
+
+   - Binning: combines adjacent pixels of the same colour by averaging or
+     summing their values, in the analog domain and/or the digital domain.
+
+      .. figure:: binning.svg
+
+
+   - Skipping: skips the read out of a number of adjacent pixels.
+
+      .. figure:: skipping.svg
+
+
+3. The output of the optional sub-sampling stage is then cropped after the
+   conversion of the analogue pixel values in the digital domain.
+
+4. The resulting output frame is sent on the media bus by the sensor.
+
+Camera Sensor configuration parameters
+--------------------------------------
+
+The libcamera camera sensor model defines parameters that allow users to
+control:
+
+1. The image format bit depth
+
+2. The size and position of the  *Analog crop rectangle*
+
+3. The subsampling factors used to downscale the pixel array readout data to a
+   smaller frame size without reducing the image *field of view*. Two
+   configuration parameters are made available to control the downscaling
+   factor:
+
+   - binning
+      A vertical and horizontal binning factor can be specified, the image
+      will be downscaled in its vertical and horizontal sizes by the specified
+      factor.
+
+      .. code-block::
+         :caption: Definition: The horizontal and vertical binning factors
+
+         horizontal_binning = xBin;
+         vertical_binning = yBin;
+
+
+   - skipping
+      Skipping reduces the image resolution by skipping the read-out of a number
+      of adjacent pixels. The skipping factor is specified by the 'increment'
+      number (number of pixels to 'skip') in the vertical and horizontal
+      directions and for even and odd rows and columns.
+
+      .. code-block::
+         :caption: Definition: The horizontal and vertical skipping factors
+
+         horizontal_skipping = (xOddInc + xEvenInc) / 2
+         vertical_skipping = (yOddInc + yEvenInc) / 2
+
+   Different sensors perform the binning and skipping stages in different
+   orders. For the sake of computing the final output image size the order of
+   execution is not relevant. The overall down-scaling factor is obtained by
+   combining the binning and skipping factors.
+
+   .. code-block::
+      :caption: Definition: The total scaling factor (binning + sub-sampling)
+
+      total_horizontal_downscale = horizontal_binning + horizontal_skipping
+      total_vertical_downscale = vertical_binning + vertical_skipping
+
+
+4. The output size is used to specify any additional cropping on the sub-sampled
+   frame.
+
+5. The total line length and frame height (*visibile* pixels + *blankings*) as
+   sent on the MIPI CSI-2 bus.
+
+6. The pixel transmission rate on the MIPI CSI-2 bus.
+
+The above parameters are combined to obtain the following high-level
+configurations:
+
+- **frame output size**
+
+   Obtained by applying a crop to the physical pixel array size in the analog
+   domain, followed by optional binning and sub-sampling (in any order),
+   followed by an optional crop step in the output digital domain.
+
+- **frame rate**
+
+   The combination of the *total frame size*, the image format *bit depth* and
+   the *pixel rate* of the data sent on the MIPI CSI-2 bus allows to compute the
+   image stream frame rate. The equation is the well known:
+
+   .. code-block::
+
+      frame_duration = total_frame_size / pixel_rate
+      frame_rate = 1 / frame_duration
+
+
+   where the *pixel_rate* parameter is the result of the sensor's configuration
+   of the MIPI CSI-2 bus *(the following formula applies to MIPI CSI-2 when
+   used on MIPI D-PHY physical protocol layer only)*
+
+   .. code-block::
+
+      pixel_rate = CSI-2_link_freq * 2 * nr_of_lanes / bits_per_sample

Documentation/index.rst

@@ -23,3 +23,4 @@
    Sensor driver requirements <sensor_driver_requirements>
    Lens driver requirements <lens_driver_requirements>
    Python Bindings <python-bindings>
+   Camera Sensor Model <camera-sensor-model>

Documentation/meson.build

@@ -63,6 +63,7 @@ endif
 
 if sphinx.found()
     docs_sources = [
+        'camera-sensor-model.rst',
         'coding-style.rst',
         'conf.py',
         'contributing.rst',