diff --git a/docs/development/Blackbox Internals.md b/docs/development/Blackbox Internals.md new file mode 100644 index 0000000000..f2b6b88fdc --- /dev/null +++ b/docs/development/Blackbox Internals.md @@ -0,0 +1,674 @@ +# Blackbox logging internals + +The Blackbox is designed to record the raw internal state of the flight controller at near-maximum rate. By logging the +raw inputs and outputs of key flight systems, the Blackbox log aims to allow the offline bench-top simulation, debugging, +and testing of flight control algorithms using data collected from real flights. + +A typical logging regime might capture 30 different state variables (for an average of 28 bytes per frame) at a sample +rate of 900Hz. That's about 25,000 bytes per second, which is 250,000 baud with a typical 8-N-1 serial encoding. + +## References + +Please refer to the source code to clarify anything this document leaves unclear: + +* Cleanflight's Blackbox logger: [blackbox.c](https://github.com/cleanflight/cleanflight/blob/master/src/main/blackbox/blackbox.c), +[blackbox_io.c](https://github.com/cleanflight/cleanflight/blob/master/src/main/blackbox/blackbox_io.c), +[blackbox_fielddefs.h](https://github.com/cleanflight/cleanflight/blob/master/src/main/blackbox/blackbox_fielddefs.h) +* [C implementation of the Blackbox log decoder](https://github.com/cleanflight/blackbox-tools/blob/master/src/parser.c) +* [JavaScript implementation of the Blackbox log decoder](https://github.com/cleanflight/blackbox-log-viewer/blob/master/js/flightlog_parser.js) + +## Logging cycle +Blackbox is designed for flight controllers that are based around the concept of a "main loop". During each main loop +iteration, the flight controller will read some state from sensors, run some flight control algorithms, and produce some +outputs. For each of these loop iterations, a Blackbox "logging iteration" will be executed. This will read data that +was stored during the execution of the main loop and log this data to an attached logging device. The data will include +algorithm inputs such as sensor and RC data, intermediate results from flight control algorithms, and algorithm outputs +such as motor commands. + +## Log frame types +Each event which is recorded to the log is packaged as a "log frame". Blackbox only uses a handful of different types of +log frames. Each frame type is identified by a single capital letter. + +### Main frames: I, P +The most basic kind of logged frames are the "main frames". These record the primary state of the flight controller (RC +input, gyroscopes, flight control algorithm intermediates, motor outputs), and are logged during every logging +iteration. + +Each main frame must contain at least two fields, "loopIteration" which records the index of the current main loop +iteration (starting at zero for the first logged iteration), and "time" which records the timestamp of the beginning of +the main loop in microseconds (this needn't start at zero, on Cleanflight it represents the system uptime). + +There are two kinds of main frames, "I" and "P". "I", or "intra" frames are like video keyframes. They can be decoded +without reference to any previous frame, so they allow log decoding to be resynchronized in the event of log damage. "P" +or "inter" frames use an encoding which references previously logged frames in order to reduce the required datarate. +When one interframe fails to decode, all following interframes will be undecodable up until the next intraframe. + +### GPS frames: G, H +Because the GPS is updated so infrequently, GPS data is logged in its own dedicated frames. These are recorded whenever +the GPS data changes (not necessarily alongside every main frame). Like the main frames, the GPS frames have their own +intra/inter encoding system. + +The "H" or "home" frame records the lat/lon of a reference point. The "G" or "GPS" frame records the current state of +the GPS system (current position, altitude etc.) based on the reference point. The reference point can be updated +(infrequently) during the flight, and is logged whenever it changes. + +To allow "G" frames to continue be decoded in the event that an "H" update is dropped from the log, the "H" frame is +logged periodically even if it has not changed (say, every 10 seconds). This caps the duration of unreadble "G" frames +that will result from a single missed "H" change. + +### Slow frames: S +Some flight controller state is updated very infrequently (on the order of once or twice a minute). Logging the fact +that this data had not been updated during every single logging iteration would be a waste of bandwidth, so these frames +are only logged when the "slow" state actually changes. + +All Slow frames are logged as intraframes. An interframe encoding scheme can't be used for Slow frames, because one +damaged frame causes all subsequent interframes to be undecodable. Because Slow frames are written so infrequently, one +missing Slow frame could invalidate minutes worth of Slow state. + +On Cleanflight, Slow frames are currently used to log data like the user-chosen flight mode and the current failsafe +state. + +### Event frames: E +Some flight controller data is updated so infrequently or exists so transiently that we do not log it as a flight +controller "state". Instead, we log it as a state *transition* . This data is logged in "E" or "event" frames. Each event +frame payload begins with a single byte "event type" field. The format of the rest of the payload is not encoded in the +flight log, so its interpretation is left up to an agreement of the writer and the decoder. + +For example, one event that Cleanflight logs is that the user has adjusted a system setting (such as a PID setting) +using Cleanflight's inflight adjustments feature. The event payload notes which setting was adjusted and the new value +for the setting. + +Because these setting updates are so rare, it would be wasteful to treat the settings as "state" and log the fact that +the setting had not been changed during every logging iteration. It would be infeasible to periodically log the system +settings using an intra/interframe scheme, because the intraframes would be so large. Instead we only log the +transitions as events, accept the small probability that any one of those events will be damaged/absent in the log, and +leave it up to log readers to decide the extent to which they are willing to assume that the state of the setting +between successfully-decoded transition events was truly unchanged. + +## Log field format +For every field in a given frame type, there is an associated name, predictor, and encoding. + +When a field is written, the chosen predictor is computed for the field, then this predictor value is subtracted from +the raw field value. Finally, the encoder is used to transform the value into bytes to be written to the logging device. + +### Field predictors +The job of the predictor is to bring the value to be encoded as close to zero as possible. The predictor may be based +on the values seen for the field in a previous frame, or some other value such as a fixed value or a value recorded in +the log headers. For example, the battery voltage values in "I" intraframes in Cleanflight use a reference voltage that +is logged as part of the headers as a predictor. This assumes that battery voltages will be broadly similar to the +initial pack voltage of the flight (e.g. 4S battery voltages are likely to lie within a small range for the whole +flight). In "P" interframes, the battery voltage will instead use the previously-logged voltage as a predictor, because +the correlation between successive voltage readings is high. + +These predictors are presently available: + +#### Predict zero (0) +This predictor is the null-predictor which doesn't modify the field value at all. It is a common choice for fields +which are already close to zero, or where no better history is available (e.g. in intraframes which may not rely on the +previous value of fields). + +#### Predict last value (1) +This is the most common predictor in interframes. The last-logged value of the field will be used as the predictor, and +subtracted from the raw field value. For fields which don't change very often, this will make their encoded value be +normally zero. Most fields have some time-correlation, so this predictor should reduce the magnitude of all but the +noisiest fields. + +#### Predict straight line (2) +This predictor assumes that the slope between the current measurement and the previous one will be similar to the +slope between the previous measurement and the one before that. This is common for fields which increase at a steady rate, +such as the "time" field. The predictor is `history_age_2 - 2 * history_age_1`. + +#### Predict average 2 (3) +This predictor is the average of the two previously logged values of the field (i.e. `(history_age_1 + history_age_2) / 2` +). It is used when there is significant random noise involved in the field, which means that the average of the recent +history is a better predictor of the next value than the previous value on its own would be (for example, in gyroscope +or motor measurements). + +#### Predict minthrottle (4) +This predictor subtracts the value of "minthrottle" which is included in the log header. In Cleanflight, motors always +lie in the range of `[minthrottle ... maxthrottle]` when the craft is armed, so this predictor is used for the first +motor value in intraframes. + +#### Predict motor[0] (5) +This predictor is set to the value of `motor[0]` which was decoded earlier within the current frame. It is used in +intraframes for every motor after the first one, because the motor commands typically lie in a tight grouping. + +#### Predict increment (6) +This predictor assumes that the field will be incremented by 1 unit for every main loop iteration. This is used to +predict the `loopIteration` field, which increases by 1 for every loop iteration. + +#### Predict home-coord (7) +This predictor is set to the corresponding latitude or longitude field from the GPS home coordinate (which was logged in +a preceding "H" frame). If no preceding "H" frame exists, the value is marked as invalid. + +#### Predict 1500 (8) +This predictor is set to a fixed value of 1500. It is preferred for logging servo values in intraframes, since these +typically lie close to the midpoint of 1500us. + +#### Predict vbatref (9) +This predictor is set to the "vbatref" field written in the log header. It is used when logging intraframe battery +voltages in Cleanflight, since these are expected to be broadly similar to the first battery voltage seen during +arming. + +#### Predict last main-frame time (10) +This predictor is set to the last logged `time` field from the main frame. This is used when predicting timestamps of +non-main frames (e.g. that might be logging the timing of an event that happened during the main loop cycle, like a GPS +reading). + +### Field encoders +The field encoder's job is to use fewer bits to represent values which are closer to zero than for values that are +further from zero. Blackbox supports a range of different encoders, which should be chosen on a per-field basis in order +to minimize the encoded data size. The choice of best encoder is based on the probability distribution of the values +which are to be encoded. For example, if a field is almost always zero, then an encoding should be chosen for it which +can encode that case into a very small number of bits, such as one. Conversely, if a field is normally 8-16 bits large, +it would be wasteful to use an encoder which provided a special short encoded representation for zero values, because +this will increase the encoded length of larger values. + +These encoders are presently available: + +#### Unsigned variable byte (1) +This is the most straightforward encoding. This encoding uses the lower 7 bits of an encoded byte to store the lower 7 +bits of the field's value. The high bit of that encoded byte is set to one if more than 7 bits are required to store the +value. If the value did exceed 7 bits, the lower 7 bits of the value (which were written to the log) are removed from +the value (by right shift), and the encoding process begins again with the new value. + +This can be represented by the following algorithm: + +```c +while (value > 127) { + writeByte((uint8_t) (value | 0x80)); // Set the high bit to mean "more bytes follow" + value >>= 7; +} +writeByte(value); +``` + +Here are some example values encoded using variable-byte encoding: + +| Input value | Output encoding | +| ----------- | --------------- | +| 1 | 0x01 | +| 42 | 0x2A | +| 127 | 0x7F | +| 128 | 0x80 0x01 | +| 129 | 0x81 0x01 | +| 23456 | 0xA0 0xB7 0x01 | + +#### Signed variable byte (0) +This encoding applies a pre-processing step to fold negative values into positive ones, then the resulting unsigned +number is encoded using unsigned variable byte encoding. The folding is accomplished by "ZigZag" encoding, which is +represented by: + +```c +unsigned32 = (signed32 << 1) ^ (signed32 >> 31) +``` + +ZigZag encoding is preferred against simply casting the signed integer to unsigned, because casting would cause small +negative quantities to appear to be very large unsigned integers, causing the encoded length to be similarly large. +ZigZag encoding ensures that values near zero are still near zero after encoding. + +Here are some example integers encoded using ZigZag encoding: + +| Input value | ZigZag encoding | +| ----------- | --------------- | +| 0 | 0 | +| -1 | 1 | +| 1 | 2 | +| -2 | 3 | +| 2147483647 | 4294967294 | +| -2147483648 | 4294967295 | + +#### Neg 14-bit (3) +The value is negated, treated as an unsigned 14 bit integer, then encoded using unsigned variable byte encoding. This +bizarre encoding is used in Cleanflight for battery pack voltages. This is because battery voltages are measured using a +14-bit ADC, with a predictor which is set to the battery voltage during arming, which is expected to be higher than any +voltage experienced during flight. After the predictor is subtracted, the battery voltage will almost certainly be below +zero. + +This results in small encoded values when the voltage is closely below the initial one, at the expense of very large +encoded values if the voltage rises higher than the initial one. + +#### Elias delta unsigned 32-bit (4) +Because this encoding produces a bitstream, this is the only encoding for which the encoded value might not be a whole +number of bytes. If the bitstream isn't aligned on a byte boundary by the time the next non-Elias Delta field arrives, +or the end of the frame is reached, the final byte is padded with zeros byte-align the stream. This encoding requires +more CPU time than the other encodings because of the bit juggling involved in writing the bitstream. + +When this encoder is chosen to encode all of the values in Cleanflight interframes, it saves about 10% bandwidth +compared to using a mixture of the other encodings, but uses too much CPU time to be practical. + +[The basic encoding algorithm is defined on Wikipedia](https://en.wikipedia.org/wiki/Elias_delta_coding). Given these +utility functions: + +```c +/* Write `bitCount` bits from the least-significant end of the `bits` integer to the bitstream. The most-significant bit + * will be written first + */ +void writeBits(uint32_t bits, unsigned int bitCount); + +/* Returns the number of bits needed to hold the top-most 1-bit of the integer 'i'. 'i' must not be zero. */ +unsigned int numBitsToStoreInteger(uint32_t i); +``` + +This is our reference implementation of Elias Delta: + +```c +// Value must be more than zero +void writeU32EliasDeltaInternal(uint32_t value) +{ + unsigned int valueLen, lengthOfValueLen; + + valueLen = numBitsToStoreInteger(value); + lengthOfValueLen = numBitsToStoreInteger(valueLen); + + // Use unary to encode the number of bits we'll need to write the length of the value + writeBits(0, lengthOfValueLen - 1); + + // Now write the length of the value + writeBits(valueLen, lengthOfValueLen); + + // Having now encoded the position of the top bit of value, write its remaining bits + writeBits(value, valueLen - 1); +} +``` + +To this we add a wrapper which allows encoding both the value zero and MAXINT: + +```c +void writeU32EliasDelta(uint32_t value) +{ + /* We can't encode value==0, so we need to add 1 to the value before encoding + * + * That would make it impossible to encode MAXINT, so use 0xFFFFFFFF as an escape + * code with an additional bit to choose between MAXINT-1 or MAXINT. + */ + if (value >= 0xFFFFFFFE) { + // Write the escape code + writeU32EliasDeltaInternal(0xFFFFFFFF); + // Add a one bit after the escape code if we wanted "MAXINT", or a zero if we wanted "MAXINT - 1" + writeBits(value - 0xFFFFFFFE, 1); + } else { + writeU32EliasDeltaInternal(value + 1); + } +} +``` + +Here are some reference encoded bit patterns produced by writeU32EliasDelta: + +| Input value | Encoded bit string | +| ----------- | ------------------ | +| 0 | 1 | +| 1 | 0100 | +| 2 | 0101 | +| 3 | 01100 | +| 4 | 01101 | +| 5 | 01110 | +| 6 | 01111 | +| 7 | 00100000 | +| 8 | 00100001 | +| 9 | 00100010 | +| 10 | 00100011 | +| 11 | 00100100 | +| 12 | 00100101 | +| 13 | 00100110 | +| 14 | 00100111 | +| 15 | 001010000 | +| 225 | 00010001100010 | +| 4294967292 | 000001000001111111111111111111111111111101 | +| 4294967293 | 000001000001111111111111111111111111111110 | +| 4294967294 | 0000010000011111111111111111111111111111110 | +| 4294967295 | 0000010000011111111111111111111111111111111 | + +Note that the very common value of zero encodes to a single bit, medium-sized values like 225 encode to 14 bits (an +overhead of 4 bits over writing a plain 8 bit value), and typical 32-bit values like 4294967293 encode into 42 bits, an +overhead of 10 bits. + +#### Elias delta signed 32-bit (5) +The value is first converted to unsigned using ZigZag encoding, then unsigned Elias-delta encoding is applied. + +#### TAG8_8SVB (6) +First, an 8-bit (one byte) header is written. This header has its bits set to zero when the corresponding field (from a +maximum of 8 fields) is set to zero, otherwise the bit is set to one. The least-signficant bit in the header corresponds +to the first field to be written. This header is followed by the values of only the fields which are non-zero, written +using signed variable byte encoding. + +This encoding is preferred for groups of fields in interframes which are infrequently updated by the flight controller. +This will mean that their predictions are usually perfect, and so the value to be encoded for each field will normally +be zero. This is common for values like RC inputs and barometer readings, which are updated in only a fraction of main +loop iterations. + +For example, given these field values to encode: + +``` +0, 0, 4, 0, 8 +``` + +This would be encoded: + +``` +0b00010100, 0x04, 0x08 +``` + +#### TAG2_3S32 (7) +A 2-bit header is written, followed by 3 signed field values of up to 32 bits each. The header value is based on the +maximum size in bits of the three values to be encoded as follows: + +| Header value | Maximum field value size | Field range | +| ------------ | ------------------------ | -------------------------- | +| 0 | 2 bits | [-2...1] | +| 1 | 4 bits | [-8...7] | +| 2 | 6 bits | [-32...31] | +| 3 | Up to 32 bits | [-2147483648...2147483647] | + +If any of the three values requires more than 6 bits to encode, a second, 6-bit header value is written in the lower +bits of the initial header byte. This second header has 2 bits for each of the encoded values which represents how many +bytes are required to encode that value. The least-significant bits of the header represent the first field which is +encoded. The values for each possibility are as follows: + +| Header value | Field size | Field range | +| ------------ | ---------- | -------------------------- | +| 0 | 1 byte | [-127...128] | +| 1 | 2 bytes | [-32768...32767] | +| 2 | 3 bytes | [-8388608...8388607] | +| 3 | 4 bytes | [-2147483648...2147483647] | + +This header is followed by the actual field values in order, written least-significant byte first, using the byte +lengths specified in the header. + +So bringing it all together, these encoded bit patterns are possible, where "0" and "1" mean bits fixed to be those +values, "A", "B", and "C" represent the first, second and third fields, and "s" represents the bits of the secondary +header in the case that any field is larger than 6 bits: + +``` +00AA BBCC, +0100 AAAA BBBB CCCC +10AA AAAA 00BB BBBB 00CC CCCC +11ss ssss (followed by fields of byte lengths specified in the "s" header) +``` + +This encoding is useful for fields like 3-axis gyroscopes, which are frequently small and typically have similar +magnitudes. + +#### TAG8_4S16 (8) +An 8-bit header is written, followed by 4 signed field values of up to 16 bits each. The 8-bit header value has 2 bits +for each of the encoded fields (the least-significant bits represent the first field) which represent the +number of bits required to encode that field as follows: + +| Header value | Field value size | Field range | +| ------------ | ---------------- | ---------------- | +| 0 | 0 bits | [0...0] | +| 1 | 4 bits | [-8...7] | +| 2 | 8 bits | [-128...127] | +| 3 | 16 bits | [-32768...32767] | + +This header is followed by the actual field values in order, written as if the output stream was a bit-stream, with the +most-significant bit of the first field ending up in the most-significant bits of the first written byte. If the number +of nibbles written is odd, the final byte has its least-significant nibble set to zero. + +For example, given these field values: + +``` +13, 0, 4, 2 +``` + +Choosing from the allowable field value sizes, they may be encoded using this many bits each: + +``` +8, 0, 4, 4 +``` + +The corresponding header values for these lengths would be: + +``` +2, 0, 1, 1 +``` + +So the header and fields would be encoded together as: + +``` +0b01010010, 0x0D, 0x42 +``` + +#### NULL (9) +This encoding does not write any bytes to the file. It is used when the predictor will always perfectly predict the +value of the field, so the remainder is always zero. In practice this is only used for the "loopIteration" field in +interframes, which is always perfectly predictable based on the logged frame's position in the sequence of frames and +the "P interval" setting from the header. + +## Log file structure +A logging session begins with a log start marker, then a header section which describes the format of the log, then the +log payload data, and finally an optional "log end" event ("E" frame). + +A single log file can be comprised of one or more logging sessions. Each session may be preceded and followed by any +amount of non-Blackbox data. This data is ignored by the Blackbox log decoding tools. This allows for the logging device +to be alternately used by the Blackbox and some other system (such as MSP) without requiring the ability to begin a +separate log file for each separate activity. + +### Log start marker +The log start marker is "H Product:Blackbox flight data recorder by Nicholas Sherlock\n". This marker is +used to discover the beginning of the flight log if the log begins partway through a file. Because it is such a long +string, it is not expected to occur by accident in any sequence of random bytes from other log device users. + +### Log header +The header is comprised of a sequence of lines of plain ASCII text. Each header line has the format `H fieldname:value` +and ends with a '\n'. The overall header does not have a terminator to separate it from the log payload +(the header implicitly ends when a line does not begin with an 'H' character). + +The header can contain some of these fields: + +#### Data version (required) +When the interpretation of the Blackbox header changes due to Blackbox specification updates, the log version is +incremented to allow backwards-compatibility in the decoder: + +``` +H Data version:2 +``` + +#### Logging interval +Not every main loop iteration needs to result in a Blackbox logging iteration. When a loop iteration is not logged, +Blackbox is not called, no state is read from the flight controller, and nothing is written to the log. Two header lines +are included to note which main loop iterations will be logged: + +##### I interval +This header notes which of the main loop iterations will record an "I" intraframe to the log. If main loop iterations +with indexes divisible by 32 will be logged as "I" frames, the header will be: + +``` +H I interval: 32 +``` + +The first main loop iteration seen by Blackbox will be numbered with index 0, so the first main loop iteration will +always be logged as an intraframe. + +##### P interval +Not every "P" interframe needs to be logged. Blackbox will log a portion of iterations in order to bring the total +portion of logged main frames to a user-chosen fraction. This fraction is called the logging rate. The smallest possible +logging rate is `(1/I interval)` which corresponds to logging only "I" frames at the "I" interval and discarding all +other loop iterations. The maximum logging rate is `1/1`, where every main loop iteration that is not an "I" frame is +logged as a "P" frame. The header records the logging rate fraction in `numerator/denominator` format like so: + +``` +H P interval:1/2 +``` + +The logging fraction given by `num/denom` should be simplified (i.e. rather than 2/6, a logging rate of 1/3 should +be used). + +Given a logging rate of `num/denom` and an I-frame interval of `I_INTERVAL`, the frame type to log for an iteration +of index `iteration` is given by: + +```c +if (iteration % I_INTERVAL == 0) + return 'I'; + +if ((iteration % I_INTERVAL + num - 1) % denom < num) + return 'P'; + +return '.'; // i.e. don't log this iteration +``` + +For an I-interval of 32, these are the resulting logging patterns at some different P logging rates. + +| Logging rate | Main frame pattern | Actual portion logged | +| ------------ | ----------------------------------------------------------------- | --------------------- | +| 1 / 32 | I...............................I...............................I | 0.03 | +| 1 / 6 | I.....P.....P.....P.....P.....P.I.....P.....P.....P.....P.....P.I | 0.19 | +| 1 / 3 | I..P..P..P..P..P..P..P..P..P..P.I..P..P..P..P..P..P..P..P..P..P.I | 0.34 | +| 1 / 2 | I.P.P.P.P.P.P.P.P.P.P.P.P.P.P.P.I.P.P.P.P.P.P.P.P.P.P.P.P.P.P.P.I | 0.50 | +| 2 / 3 | I.PP.PP.PP.PP.PP.PP.PP.PP.PP.PP.I.PP.PP.PP.PP.PP.PP.PP.PP.PP.PP.I | 0.66 | +| 5 / 6 | I.PPPPP.PPPPP.PPPPP.PPPPP.PPPPP.I.PPPPP.PPPPP.PPPPP.PPPPP.PPPPP.I | 0.81 | +| 1 / 1 | IPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPIPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPI | 1.00 | + + +#### Firmware type (optional) +Because Blackbox records the internal flight controller state, the interpretation of the logged data will depend +on knowing which flight controller recorded it. To accomodate this, the name of the flight controller should be recorded: + +``` +H Firmware type:Cleanflight +``` + +More details should be included to help narrow down the precise flight-controller version (but these are not required): + +``` +H Firmware revision:c49bd40 +H Firmware date:Aug 28 2015 16:49:11 +``` + +#### Field X name (required) +This header is a comma-separated list of the names for the fields in the 'X' frame type: + +``` +H Field I name:loopIteration,time,axisP[0],axisP[1]... +``` + +The decoder assumes that the fields in the 'P' frame type will all have the same names as those in the 'I' frame, so +a "Field P name" header does not need to be supplied. + +#### Field X signed (optional) +This is a comma-separated list of integers which are set to '1' when their corresponding field's value should be +interpreted as signed after decoding, or '0' otherwise: + +``` +H Field I signed:0,0,1,1... +``` + +#### Field X predictor (required) +This is a comma-separated list of integers which specify the predictors for each field in the specified frame type: + +``` +H Field I predictor:0,0,0,0... +``` + +#### Field X encoding (required) +This is a comma-separated list of integers which specify the encoding used for each field in the specified frame type: + +``` +H Field X encoding:1,1,0,0... +``` + +#### vbatref +This header provides the reference voltage that will be used by predictor #9. + +#### minthrottle +This header provides the minimum value sent by Cleanflight to the ESCs when armed, it is used by predictor #4. + +#### Additional headers +The decoder ignores headers that it does not understand, so you can freely add any headers that you require in order to +properly interpret the meaning of the logged values. + +For example, to create a graphical displays of RC sticks and motor percentages, the Blackbox rendering tool requires +the additional headers "rcRate" and "maxthrottle". In order to convert raw gyroscope, accelerometer and voltage readings +into real-world units, the Blackbox decoder requires the calibration constants "gyro.scale", "acc_1G" and "vbatscale". +These headers might look like: + +``` +H rcRate:100 +H maxthrottle:1980 +H gyro.scale:0x3d79c190 +H acc_1G:4096 +H vbatscale:110 +``` + +### Log payload + +The log payload is a concatenated sequence of logged frames. Each frame type which is present in the log payload must +have been previously described in the log header (with Frame X name, etc. headers). Each frame begins with a single +capital letter to specify the type of frame (I, P, etc), which is immediately followed by the frame's field data. There +is no frame length field, checksum, or trailer. + +The field data is encoded by taking an array of raw field data, computing the predictor for each field, subtrating this +predictor from the field, then applying the field encoders to each field in sequence to serialize them to the log. + +For example, imagine that we are encoding three fields in an intraframe, are using zero-predictors for each field (#0), +and are encoding the values using the unsigned variable byte encoding (#1). For these field values: + +``` +1, 2, 3 +``` + +We would encode a frame: + +``` +'I', 0x01, 0x02, 0x03 +``` + +Imagine that we are encoding an array of motor commands in an interframe. We will use the previous motor commands as a +predictor, and encode the resulting values using signed variable byte encoding. The motor command values seen in the +previous logged iteration were: + +``` +1430, 1500, 1470, 1490 +``` + +And the motor commands to be logged in this iteration are: + +``` +1635, 1501, 1469, 1532 +``` + +After subtracting the predictors for each field, we will be left with: + +``` +205, 1, -1, 42 +``` + +We will apply ZigZag encoding to each field, which will give us: + +``` +410, 2, 1, 84 +``` + +We will use unsigned variable byte encoding to write the resulting values to the log, which will give us: + +``` +'P', 0x9A, 0x03, 0x02, 0x01, 0x54 +``` + +### Log end marker +The log end marker is an optional Event ("E") frame of type 0xFF whose payload is the string "End of log\0". The +payload ensures that random data does not look like an end-of-log marker by chance. This event signals the tidy ending +of the log. All following bytes until the next log-begin marker (or end of file) should be ignored by the log +decoder. + +``` +'E', 0xFF, "End of log", 0x00 +``` + +## Log validation +Any damage experienced to the log during recording is overwhelmingly due to subsequences of bytes being dropped by the +logging device due to overflowing buffers. Accordingly, Blackbox logs do not bother to include any checksums (bytes are +not expected to be damaged by the logging device without changing the length of the message). Because of the tight +bandwidth requirements of logging, neither a frame length field nor frame trailer is recorded that would allow for the +detection of missing bytes. + +Instead, the decoder uses a heuristic in order to detect damaged frames. The decoder reads an entire frame from the log +(using the decoder for each field which is the counterpart of the encoder specified in the header), then it checks to +see if the byte immediately following the frame, which should be the beginning of a next frame, is a recognized +frame-type byte (e.g. 'I', 'P', 'E', etc). If that following byte represents a valid frame type, it is assumed that the +decoded frame was the correct length (so was unlikely to have had random ranges of bytes removed from it, which would +have likely altered the frame length). Otherwise, the frame is rejected, and a valid frame-type byte is looked for +immediately after the frame-start byte of the frame that was rejected. A rejected frame causes all subsequent +interframes to be rejected as well, up until the next intraframe. + +A frame is also rejected if the "loopIteration" or "time" fields have made unreasonable leaps forward, or moved at +all backwards. This suffices to detect almost all log corruption. \ No newline at end of file