Mirror/libcamera
1
0
Fork 0
mirror of https://git.libcamera.org/libcamera/libcamera.git synced 2025-07-12 23:09:45 +03:00
Code Issues Projects Releases Packages Wiki Activity Actions

Documentation: design: ae: Document the design for AE controls

Browse source 
Document the design and rationale for the AE-related controls.
Also add documentation for the controls.

Signed-off-by: Paul Elder <paul.elder@ideasonboard.com>
Signed-off-by: Jacopo Mondi <jacopo@jmondi.org>
Reviewed-by: Stefan Klug <stefan.klug@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
This commit is contained in:
Paul Elder 2025-01-13 15:59:36 -06:00 • committed by Laurent Pinchart
parent 3becdbcbe8
commit 128220a139
3 changed files with 324 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

320
Documentation/design/ae.rst Normal file
View file

@ -0,0 +1,320 @@
.. SPDX-License-Identifier: CC-BY-SA-4.0
Design of Exposure and Gain controls
====================================
This document explains the design and rationale of the controls related to
exposure and gain. This includes the all-encompassing auto-exposure (AE), the
manual exposure control, and the manual gain control.
Description of the problem
--------------------------
Sub controls
^^^^^^^^^^^^
There are more than one control that make up total exposure: exposure time,
gain, and aperture (though for now we will not consider aperture). We already
had individual controls for setting the values of manual exposure and manual
gain, but for switching between auto mode and manual mode we only had a
high-level boolean AeEnable control that would set *both* exposure and gain to
auto mode or manual mode; we had no way to set one to auto and the other to
manual.
So, we need to introduce two new controls to act as "levers" to indicate
individually for exposure and gain if the value would come from AEGC or if it
would come from the manual control value.
Aperture priority
^^^^^^^^^^^^^^^^^
We eventually may need to support aperture, and so whatever our solution is for
having only some controls on auto and the others on manual needs to be
extensible.
Flickering when going from auto to manual
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
When a manual exposure or gain value is requested by the application, it costs
a few frames worth of time for them to take effect. This means that during a
transition from auto to manual, there would be flickering in the control values
and the transition won't be smooth.
Take for instance the following flow, where we start on auto exposure (which
for the purposes of the example increments by 1 each frame) and we want to
switch seamlessly to manual exposure, which involves copying the exposure value
computed by the auto exposure algorithm:
::
+-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+
| N | | N+1 | | N+2 | | N+3 | | N+4 | | N+5 | | N+6 |
+-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+
Mode requested: Auto Auto Auto Manual Manual Manual Manual
Exp requested: N/A N/A N/A 2 2 2 2
Set in Frame: N+2 N+3 N+4 N+5 N+6 N+7 N+8
Mode used: Auto Auto Auto Auto Auto Manual Manual
Exp used: 0 1 2 3 4 2 2
As we can see, after frame N+2 completes, we copy the exposure value that was
used for frame N+2 (which was computed by AE algorithm), and queue that value
into request N+3 with manual mode on. However, as it takes two frames for the
exposure to be set, the exposure still changes since it is set by AE, and we
get a flicker in the exposure during the switch from auto to manual.
A solution is to *not submit* any exposure value when manual mode is enabled,
and wait until the manual mode as been "applied" before copying the exposure
value:
::
+-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+
| N | | N+1 | | N+2 | | N+3 | | N+4 | | N+5 | | N+6 |
+-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+
Mode requested: Auto Auto Auto Manual Manual Manual Manual
Exp requested: N/A N/A N/A None None None 5
Set in Frame: N+2 N+3 N+4 N+5 N+6 N+7 N+8
Mode used: Auto Auto Auto Auto Auto Manual Manual
Exp used: 0 1 2 3 4 5 5
In practice, this works. However, libcamera has a policy where once a control
is submitted, its value is saved and does not need to be resubmitted. If the
manual exposure value was set while auto mode was on, in theory the value would
be saved, so when manual mode is enabled, the exposure value that was
previously set would immediately be used. Clearly this solution isn't correct,
but it can serve as the basis for a proper solution, with some more rigorous
rules.
Existing solutions
------------------
Raspberry Pi
^^^^^^^^^^^^
The Raspberry Pi IPA gets around the lack of individual AeEnable controls for
exposure and gain by using magic values. When AeEnable is false, if one of the
manual control values was set to 0 then the value computed by AEGC would be
used for just that control. This solution isn't desirable, as it prevents
that magic value from being used as a valid value.
To get around the flickering issue, when AeEnable is false, the Raspberry Pi
AEGC simply stops updating the values to be set, without restoring the
previously set manual exposure time and gain. This works, but is not a proper
solution.
Android
^^^^^^^
The Android HAL specification requires that exposure and gain (sensitivity)
must both be manual or both be auto. It cannot be that one is manual while the
other is auto, so they simply don't support sub controls.
For the flickering issue, the Android HAL has an AeLock control. To transition
from auto to manual, the application would keep AE on auto, and turn on the
lock. Once the lock has propagated through, then the value can be copied from
the result into the request and the lock disabled and the mode set to manual.
The problem with this solution is, besides the extra complexity, that it is
ambiguous what happens if there is a state transition from manual to locked
(even though it's a state transition that doesn't make sense). If locked is
defined to "use the last automatically computed values" then it could use the
values from the last time it AE was set to auto, or it would be undefined if AE
was never auto (eg. it started out as manual), or if AE is implemented to run
in the background it could just use the current values that are computed. If
locked is defined to "use the last value that was set" there would be less
ambiguity. Still, it's better if we can make it impossible to execute this
nonsensical state transition, and if we can reduce the complexity of having
this extra control or extra setting on a lever.
Summary of goals
----------------
- We need a lock of some sort, to instruct the AEGC to not update output
results
- We need manual modes, to override the values computed by the AEGC
- We need to support seamless transitions from auto to manual, and do so
without flickering
- We need custom minimum values for the manual controls; that is, no magic
values for enabling/disabling auto
- All of these need to be done with AE sub-controls (exposure time, analogue
gain) and be extensible to aperture in the future
Our solution
------------
A diagram of our solution:
::
+----------------------------+-------------+------------------+-----------------+
| INPUT | ALGORITHM | RESULT | OUTPUT |
+----------------------------+-------------+------------------+-----------------+
ExposureTimeMode ExposureTimeMode
---------------------+----------------------------------------+----------------->
0: Auto | |
1: Manual | V
| |\
| | \
| /----------------------------------> | 1| ExposureTime
| | +-------------+ exposure time | | -------------->
\--)--> | | --------------> | 0|
ExposureTime | | | | /
------------------------+--> | | |/
| | AeState
| AEGC | ----------------------------------->
AnalogueGain | |
------------------------+--> | | |\
| | | | \
/--)--> | | --------------> | 0| AnalogueGain
| | +-------------+ analogue gain | | -------------->
| \----------------------------------> | 1|
| | /
| |/
| ^
AnalogueGainMode | | AnalogueGainMode
---------------------+----------------------------------------+----------------->
0: Auto
1: Manual
The diagram is divided in four sections horizontally:
- Input: The values received from the request controls
- Algorithm: The algorithm itself
- Result: The values calculated by the algorithm
- Output: The values reported in result metadata and applied to the device
The four input controls are divided between manual values (ExposureTime and
AnalogueGain), and operation modes (ExposureTimeMode and AnalogueGainMode). The
former are the manual values, the latter control how they're applied. The two
modes are independent from each other, and each can take one of two values:
- Auto (0): The AGC computes the value normally. The AGC result is applied
to the output. The manual value is ignored *and is not retained*.
- Manual (1): The AGC uses the manual value internally. The corresponding
manual control from the request is applied to the output. The AGC result
is ignored.
The AeState control reports the state of the unified AEGC block. If both
ExposureTimeMode and AnalogueGainMode are set to manual then it will report
Idle. If at least one of the two is set to auto, then AeState will report
if the AEGC has Converged or not (Searching). This control replaces the old
AeLocked control, as it was insufficient for reporting the AE state.
There is a caveat to manual mode: the manual control value is not retained if
it is set during auto mode. This means that if manual mode is entered without
also setting the manual value, then it will enter a state similar to "locked",
where the last automatically computed value while the mode was auto will be
used. Once the manual value is set, then that will be used and retained as
usual.
This simulates an auto -> locked -> manual or auto -> manual state transition,
and makes it impossible to do the nonsensical manual -> locked state
transition.
We specifically do not have a "master AE control" like the old AeEnable. This
is because we have the individual mode controls, and if we had a master AE
control it would be a "control that sets other controls", which could easily
get out of control.
With this solution, the earlier example would become:
::
+-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+
| N+2 | | N+3 | | N+4 | | N+5 | | N+6 | | N+7 | | N+8 | | N+9 | | N+10|
+-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+
Mode requested: Auto Manual Manual Manual Manual Manual Manual Manual Manual
Exp requested: N/A None None None None 10 None 10 10
Set in Frame: N+4 N+5 N+6 N+7 N+8 N+9 N+10 N+11 N+12
Mode used: Auto Auto Auto Manual Manual Manual Manual Manual Manual
Exp used: 2 3 4 5 5 5 5 10 10
This example is extended by a few frames to exhibit the simulated "locked"
state. At frame N+5 the application has confirmed that the manual mode has been
entered, but does not provide a manual value until request N+7. Thus, the value
that is used in requests N+5 and N+6 (where the mode is disabled), comes from
the last value that was used when the mode was auto, which comes from frame
N+4.
Then, in N+7, a manual value of 10 is supplied. It takes until frame N+9 for
the exposure to be applied. N+8 does not supply a manual value, but the last
supplied value is retained, so a manual value of 10 is still used and set in
frame N+10.
Although this behavior is the same as what we had with waiting for the manual
mode to propagate (in the section "Description of the problem"), this time it
is correct as we have defined specifically that if a manual value was specified
while the mode was auto, it will not be retained.
Description of the controls
---------------------------
As described above, libcamera offers the following controls related to exposure
and gain:
- AnalogueGain
- AnalogueGainMode
- ExposureTime
- ExposureTimeMode
- AeState
Auto-exposure and auto-gain can be enabled and disabled separately using the
ExposureTimeMode and AnalogueGainMode controls respectively. There is no
overarching AeEnable control.
When the respective mode is set to auto, the respective value that is computed
by the AEGC algorithm is applied to the image sensor. Any value that is
supplied in the manual ExposureTime/AnalogueGain control is ignored and not
retained. Another way to understand this is that when the mode transitions from
auto to manual, the internally stored control value is overwritten with the
last value computed by the auto algorithm.
This means that when we transition from auto to manual without supplying a
manual control value, the last value that was set by the AEGC algorithm will
keep be used. This can be used to do a flickerless transition from auto to
manual as described earlier. If the camera started out in manual mode and no
corresponding value has been supplied yet, then a best-effort default value
shall be set.
The manual control value can be set in the same request as setting the mode to
auto if the desired manual control value is already known.
Transitioning from manual to auto shall be implicitly flickerless, as the AEGC
algorithms are expected to start running from the last manual value.
The AeState metadata reports the state of the AE algorithm. As AE cannot
compute exposure and gain separately, the state of the AE component is
unified. There are three states: Idle, Searching, and Converged.
The state shall be Idle if both ExposureTimeMode and AnalogueGainMode
are set to Manual. If the camera only supports one of the two controls,
then the state shall be Idle if that one control is set to Manual. If
the camera does not support Manual for at least one of the two controls,
then the state will never be Idle, as AE will always be running.
The state shall be Searching if at least one of exposure or gain calculated
by the AE algorithm is used (that is, at least one of the two modes is Auto),
*and* the value(s) have not converged yet.
The state shall be Converged if at least one of exposure or gain calculated
by the AE algorithm is used (that is, at least one of the two modes is Auto),
*and* the value(s) have converged.

4
Documentation/index.rst
View file

@ -23,7 +23,9 @@
SoftwareISP Benchmarking <software-isp-benchmarking> SoftwareISP Benchmarking <software-isp-benchmarking>
Tracing guide <guides/tracing> Tracing guide <guides/tracing>
Design document: AE <design/ae>
.. toctree:: .. toctree::
:hidden: :hidden:
introduction introduction

1
Documentation/meson.build
View file

@ -128,6 +128,7 @@ if sphinx.found()
'coding-style.rst', 'coding-style.rst',
'conf.py', 'conf.py',
'contributing.rst', 'contributing.rst',
'design/ae.rst',
'documentation-contents.rst', 'documentation-contents.rst',
'environment_variables.rst', 'environment_variables.rst',
'feature_requirements.rst', 'feature_requirements.rst',