Mirror/inav
1
0
Fork 0
mirror of https://github.com/iNavFlight/inav.git synced 2025-07-25 17:25:18 +03:00
Code Issues Wiki Activity

Generate CLI settings at build time (#2028)

Browse source 
* Initial commit for the CLI settings compiler

Not very useful for now, only generates settings.c in the same
way the settings were manually written in cli.c

* Move all settings to a YAML file

This will eventually let us compile and pack the settings saving
a lot of memory. For now, the code compiles but it doesn't work
since it uses a byte to index into the word array which has more
than 256 entries.

* Use varint encoding for cli name word indexing

This makes the CLI work again.

* Make clivalue_name_* funcs return bool

Makes more sense than returning uint8_t, even when the compiler will
probably generate exactly the same assembly in both cases.

* Fix invalid field name

Missing a closing ]

* Initial attempt at generating the settings files at build time

Optimize the generator to call into the compiler only once, so
we can afford to call it for each build and, eventually, generate
build-optimized settings.

* Fix build error due to generated files

Due to make's expansion rules, the generated implementation file
wasn't correctly compiled if the build was started when the
generated files didn't exist.

Althogh there's probably a better solution, this should work for
now.

* Generate a per-build settings_generated.{h,c}

This allows us to save a bit more space, since this way the words
array doesn't include words which are not used by the build.

* Remove pgn_t field from cliValueConfig_t

Use a couple of arrays to find the pgn_t for a setting from its
offset in the table. This saves another 384 bytes on NAZE.

* Use only a byte for the field offset in clivalue_t when possible

While compiling the settings, determine if any offset requires a
number bigger than 255. If that's not the case, use a uint8_t
rather than an uint16_t for storing the field offset.

* Add missing header to PG_MODE_ACTIVATION_OPERATOR_CONFIG group

* Fix unbalanced #endif

Introduced when deleting the hardcoded settings from cli.c

* Don't ignore the return value from g.CanUseByteOffsetoff()

CLIVALUE_USE_BYTE_OFFSETOF was always defined regardless of the
maximum offsetof() value found in the settings.

* clivalue_name_*() functions now take a buffer

Requires only CLIVALUE_MAX_NAME_LENGTH bytes in the stack rather
than 2*CLIVALUE_MAX_NAME_LENGTH, since those functions were called
from functions which already had a buffer for the name allocated
but had to allocate their own.

* Remove unneeded clivalue_get_name() call

clivalue_name_exact_match() will already fill the buffer with
the value name.

* Fix off-by-one error in the settings generator

The generated C code wasn't allocating enough space for the '\0'
terminator for setting names

* Fix off-by-one error in the name decoder

CLIVALUE_ENCODED_NAME_MAX_BYTES represents the maximum number of
bytes in an encoded name, not the maximum word index.

* Add missing headers to PG_STATS_CONFIG group

* Make sure the settings are always up to date

* Initial attempt at encoding constants used for min/max settings

Pretty naive approach for now. Saves ~400 bytes on F1 targets.

* Move tool for generating settings to tools/

Also, rename it from settings_gen to just settings.
Delete the .gitignore in src/main/fc and just add all ignored
files in the root .gitignore, since that speeds up git.

* Only print setting stats when the env var V=1

This way we get quiet output unless the Makefile has been invoked
with verbose output.

* Make setting generation rules compatible with gmake 4

Rules were working fine on gmake 3, but failing with gmake 4. These
new rules should work with both of them.

* Fix constant value detection with GCC 7.1

GCC 6.3 emits errors with <42type-suffix> while GCC 7.1
emits the errors with only <42>

* Format uint8_t arrays a bit better

Don't add a comma after the last element

* Sort words and values determiniscally

This will help while checking the upcoming Ruby implementation
of the generator against the previous one using Go.

* Add missing headers for some groups in settings.yaml

* Replace the Go settings generator with a Ruby implementation

This makes it easier to install the required dependencies to
build INAV, since Ruby is installed by default on macOS and
installing it in Linux should be easier than installing Go
and a 3rd party package (for YAML parsing).

* Don't hardcode the value type for each parameter group

Instead, add a value_type field to each group with a default
value of MASTER_VALUE

* Simplify code for adding custom methods to StringIO

* Only resolve types for enabled fields

This fixes issues with some types which are only defined
if the feature for them is enabled (e.g. STATS or NAV).

* Implement print_stats() in the Ruby settings generator

* Rename constant in generated settings

CLIVALUE_ENCODED_NAME_USES_DIRECT_INDEXING =>
CLIVALUE_ENCODED_NAME_USES_BYTE_INDEXING

* Remove old settings generator binary from .gitignore

* Enable DEBUG while generating settings

Travis build is failing, this should help determine why

* Add $TOOLCHAINPATH to $PATH on Travis builds

* Disable DEBUG in settings.rb

Travis build is now failing because the log is too big

* Fix warning when running settings.rb on RB >= 2.4

* Don't print message when generating settings with V=0

* Use a relative path for the temporary dir

Absolute paths cause issues calling out to g++ on Windows

* Add INAV license header to settings.rb

* Add missing header to settings.c

Required since last rebase, it was compiling fine previously

* Remove unneeded extern variable decl from settings.c

Not needed anymore since we're now including settings_generated.c
directly in settings.c to simplify the Makefile.

* Use obj/tmp rather than just tmp for temporary files

* Update devdocs to mention Ruby installation

* Update Dockerfile and Vagrantfile to install Ruby

Required by the settings generator

Fixes #1997
This commit is contained in:
Alberto García Hierro 2017-08-28 16:08:38 +02:00 • committed by Konstantin Sharlaimov
parent 0590c26203
commit a5607bc54c
15 changed files with 2570 additions and 886 deletions
Download patch file Download diff file Expand all files Collapse all files

3
.gitignore vendored
View file

@ -19,3 +19,6 @@ cov-int*
docs/Manual.pdf
README.pdf
# build generated files
/src/main/fc/settings_generated.h
/src/main/fc/settings_generated.c

1
.travis.yml
View file

@ -26,6 +26,7 @@ before_install:
install:
- ./install-toolchain.sh
- export TOOLCHAINPATH=$PWD/gcc-arm-none-eabi-6-2017-q2-update/bin
- export PATH=$TOOLCHAINPATH:$PATH
before_script:
- $TOOLCHAINPATH/arm-none-eabi-gcc --version

2
Dockerfile
View file

@ -9,4 +9,4 @@ RUN mkdir -p /home/src && apt-get update && \
apt-get remove -y binutils-arm-none-eabi gcc-arm-none-eabi && \
add-apt-repository -y ppa:terry.guo/gcc-arm-embedded && \
apt-get update && \
apt-get install -y gcc-arm-none-eabi libnewlib-arm-none-eabi make git gcc
apt-get install -y gcc-arm-none-eabi libnewlib-arm-none-eabi make git gcc ruby

29
Makefile
View file

@ -594,6 +594,7 @@ COMMON_SRC = \
fc/rc_curves.c \
fc/rc_modes.c \
fc/runtime_config.c \
fc/settings.c \
fc/stats.c \
flight/failsafe.c \
flight/hil.c \
@ -913,6 +914,29 @@ CLEAN_ARTIFACTS += $(TARGET_ELF) $(TARGET_OBJS) $(TARGET_MAP)
# Make sure build date and revision is updated on every incremental build
$(OBJECT_DIR)/$(TARGET)/build/version.o : $(TARGET_SRC)
# Settings generator
.PHONY: settings clean-settings
TOOL_DIR = $(ROOT)/tools
SETTINGS_GENERATOR = $(TOOL_DIR)/settings.rb
GENERATED_SETTINGS = $(SRC_DIR)/fc/settings_generated.h $(SRC_DIR)/fc/settings_generated.c
SETTINGS_FILE = $(SRC_DIR)/fc/settings.yaml
$(GENERATED_SETTINGS): $(SETTINGS_GENERATOR) $(SETTINGS_FILE)
# Use a pattern rule, since they're different than normal rules.
# See https://www.gnu.org/software/make/manual/make.html#Pattern-Examples
%generated.h %generated.c:
$(V1) echo "settings.yaml -> settings_generated.h, settings_generated.c" "$(STDOUT)"
$(V1) CFLAGS="$(CFLAGS)" ruby $(SETTINGS_GENERATOR) . $(SETTINGS_FILE)
settings: $(GENERATED_SETTINGS)
clean-settings:
$(V1) $(RM) $(GENERATED_SETTINGS)
# Files that depend on the generated settings
$(OBJECT_DIR)/$(TARGET)/fc/cli.o: settings
$(OBJECT_DIR)/$(TARGET)/fc/settings.o: settings
# List of buildable ELF files and their object dependencies.
# It would be nice to compute these lists, but that seems to be just beyond make.
@ -922,9 +946,9 @@ $(TARGET_HEX): $(TARGET_ELF)
$(TARGET_BIN): $(TARGET_ELF)
$(V0) $(OBJCOPY) -O binary $< $@
$(TARGET_ELF): $(TARGET_OBJS)
$(TARGET_ELF): clean-settings $(TARGET_OBJS)
$(V1) echo Linking $(TARGET)
$(V1) $(CROSS_CC) -o $@ $^ $(LDFLAGS)
$(V1) $(CROSS_CC) -o $@ $(filter %.o, $^) $(LDFLAGS)
$(V0) $(SIZE) $(TARGET_ELF)
# Compile
@ -959,6 +983,7 @@ clean:
$(V0) echo "Cleaning $(TARGET)"
$(V0) rm -f $(CLEAN_ARTIFACTS)
$(V0) rm -rf $(OBJECT_DIR)/$(TARGET)
$(V0) rm -f $(GENERATED_SETTINGS)
$(V0) echo "Cleaning $(TARGET) succeeded."
## clean_test : clean up all temporary / machine-generated files (tests)

2
Vagrantfile vendored
View file

@ -21,6 +21,6 @@ Vagrant.configure(2) do |config|
apt-get remove -y binutils-arm-none-eabi gcc-arm-none-eabi
add-apt-repository ppa:terry.guo/gcc-arm-embedded
apt-get update
apt-get install -y git gcc-arm-none-eabi=4.9.3.2015q3-1trusty1
apt-get install -y git gcc-arm-none-eabi=4.9.3.2015q3-1trusty1 ruby
SHELL
end

3
docs/development/Building in Eclipse.md
View file

@ -8,11 +8,12 @@ Import the project using the wizard **Existing Code as Makefile Project**
Adjust your build option if necessary
![](https://camo.githubusercontent.com/64a1d32400d6be64dd4b5d237df1e7f1b817f61b/687474703a2f2f692e696d6775722e636f6d2f6641306d30784d2e706e67)
Make sure you have a valid ARM toolchain in the path
Make sure you have a valid ARM toolchain and Ruby in the path
![](http://i.imgur.com/dAbscJo.png)
# Long version
* First you need an ARM toolchain. Good choices are **GCC ARM Embedded** (https://launchpad.net/gcc-arm-embedded) or **Yagarto** (http://www.yagarto.de).
* Install Ruby (see the document for your operating system).
* Now download Eclipse and unpack it somewhere. At the time of writing Eclipse 4.2 was the latest stable version.
* To work with ARM projects in Eclipse you need a few plugins:
+ **Eclipse C Development Tools** (CDT) (available via *Help > Install new Software*).

4
docs/development/Building in Mac OS X.md
View file

@ -73,6 +73,10 @@ If `arm-none-eabi-gcc` couldn't be found, go back and check that you entered the
[GNU Tools for ARM Embedded Processors project]: https://launchpad.net/gcc-arm-embedded
[the older releases]: https://launchpad.net/gcc-arm-embedded/+download
## Ruby
Ruby is installed by default on macOS.
## Checkout INAV sourcecode through git
Enter your development directory and clone the [INAV repository][] using the "HTTPS clone URL" which is shown on

8
docs/development/Building in Ubuntu.md
View file

@ -50,6 +50,14 @@ For Ubuntu 12.04 (previous LTS, called Precise Penguin), you should pin:
sudo apt-get install gcc-arm-none-eabi=4.9.3.2014q4-0precise12
```
## Install Ruby
Install the Ruby package for your distribution. On Debian based distributions, you should
install the ruby package
```
sudo apt-get install ruby
```
## Building on Ubuntu
After the ARM toolchain from Terry is installed, you should be able to build from source.

4
docs/development/Building in Windows light.md
View file

@ -38,6 +38,10 @@ download https://launchpad.net/gcc-arm-embedded/4.9/4.9-2015-q2-update/+download
extract it into C:\devtools\gcc-arm-none-eabi-4_9-2015q2-20150609-win32 (folder already there)
##Install Ruby
Install the latest Ruby version using [Ruby Installer](https://rubyinstaller.org).
##Test
Run C:\devtools\shF4.cmd

4
docs/development/Building in Windows.md
View file

@ -51,6 +51,10 @@ add the "bin" subdirectory to the PATH Windows environment variable: ```%PATH%;C
![GNU ARM Toolchain Setup](assets/010.toolchain_path.png)
##Setup Ruby
Install the latest Ruby version using [Ruby Installer](https://rubyinstaller.org).
## Checkout and compile INAV
Head over to the INAV Github page and grab the URL of the GIT Repository: "https://github.com/iNavFlight/inav.git"

931
src/main/fc/cli.c
View file

File diff suppressed because it is too large Load diff

86
src/main/fc/settings.c Normal file
View file

@ -0,0 +1,86 @@
#include <string.h>
#include <stdint.h>
#include "common/string_light.h"
#include "fc/settings_generated.h"
#include "fc/settings.h"
#include "fc/settings_generated.c"
void clivalue_get_name(const clivalue_t *val, char *buf)
{
uint8_t bpos = 0;
uint16_t n = 0;
#ifndef CLIVALUE_ENCODED_NAME_USES_BYTE_INDEXING
uint8_t shift = 0;
#endif
for (uint8_t ii = 0; ii < CLIVALUE_ENCODED_NAME_MAX_BYTES; ii++) {
#ifdef CLIVALUE_ENCODED_NAME_USES_BYTE_INDEXING
n = val->encoded_name[ii];
#else
// Decode a variable size uint
uint16_t b = val->encoded_name[ii];
if (b >= 0x80) {
// More bytes follow
n |= (b&0x7f) << shift;
shift += 7;
continue;
}
// Final byte
n |= b << shift;
#endif
const char *word = cliValueWords[n];
if (!word) {
// No more words
break;
}
if (bpos > 0) {
// Word separator
buf[bpos++] = '_';
}
strcpy(&buf[bpos], word);
bpos += strlen(word);
#ifndef CLIVALUE_ENCODED_NAME_USES_BYTE_INDEXING
// Reset shift and n
shift = 0;
n = 0;
#endif
}
buf[bpos] = '\0';
}
bool clivalue_name_contains(const clivalue_t *val, char *buf, const char *cmdline)
{
clivalue_get_name(val, buf);
return strstr(buf, cmdline) != NULL;
}
bool clivalue_name_exact_match(const clivalue_t *val, char *buf, const char *cmdline, uint8_t var_name_length)
{
clivalue_get_name(val, buf);
return sl_strncasecmp(cmdline, buf, strlen(buf)) == 0 && var_name_length == strlen(buf);
}
pgn_t clivalue_get_pgn(const clivalue_t *val)
{
uint16_t pos = val - (const clivalue_t *)cliValueTable;
uint16_t acc = 0;
for (uint8_t ii = 0; ii < CLIVALUE_PGN_COUNT; ii++) {
acc += cliValuePgnCounts[ii];
if (acc > pos) {
return cliValuePgn[ii];
}
}
return -1;
}
clivalue_min_t clivalue_get_min(const clivalue_t *val)
{
return cliValueMinMaxTable[CLIVALUE_INDEXES_GET_MIN(val)];
}
clivalue_max_t clivalue_get_max(const clivalue_t *val)
{
return cliValueMinMaxTable[CLIVALUE_INDEXES_GET_MAX(val)];
}

77
src/main/fc/settings.h Normal file
View file

@ -0,0 +1,77 @@
#pragma once
#include <stdbool.h>
#include <stdint.h>
#include "config/parameter_group.h"
#include "fc/settings_generated.h"
typedef struct lookupTableEntry_s {
const char * const *values;
const uint8_t valueCount;
} lookupTableEntry_t;
extern const lookupTableEntry_t cliLookupTables[];
#define VALUE_TYPE_OFFSET 0
#define VALUE_SECTION_OFFSET 4
#define VALUE_MODE_OFFSET 6
typedef enum {
// value type, bits 0-3
VAR_UINT8 = (0 << VALUE_TYPE_OFFSET),
VAR_INT8 = (1 << VALUE_TYPE_OFFSET),
VAR_UINT16 = (2 << VALUE_TYPE_OFFSET),
VAR_INT16 = (3 << VALUE_TYPE_OFFSET),
VAR_UINT32 = (4 << VALUE_TYPE_OFFSET),
VAR_FLOAT = (5 << VALUE_TYPE_OFFSET), // 0x05
// value section, bits 4-5
MASTER_VALUE = (0 << VALUE_SECTION_OFFSET),
PROFILE_VALUE = (1 << VALUE_SECTION_OFFSET),
CONTROL_RATE_VALUE = (2 << VALUE_SECTION_OFFSET), // 0x20
// value mode, bits 6-7
MODE_DIRECT = (0 << VALUE_MODE_OFFSET),
MODE_LOOKUP = (1 << VALUE_MODE_OFFSET), // 0x40
} cliValueFlag_e;
#define VALUE_TYPE_MASK (0x0F)
#define VALUE_SECTION_MASK (0x30)
#define VALUE_MODE_MASK (0xC0)
typedef struct cliMinMaxConfig_s {
const uint8_t indexes[CLIVALUE_MIN_MAX_INDEX_BYTES];
} cliMinMaxConfig_t;
typedef struct cliLookupTableConfig_s {
const uint8_t tableIndex;
} cliLookupTableConfig_t;
typedef union {
cliLookupTableConfig_t lookup;
cliMinMaxConfig_t minmax;
} cliValueConfig_t;
typedef struct {
const uint8_t encoded_name[CLIVALUE_ENCODED_NAME_MAX_BYTES];
const uint8_t type; // see cliValueFlag_e
const cliValueConfig_t config;
clivalue_offset_t offset;
} __attribute__((packed)) clivalue_t;
extern const clivalue_t cliValueTable[];
void clivalue_get_name(const clivalue_t *val, char *buf);
bool clivalue_name_contains(const clivalue_t *val, char *buf, const char *cmdline);
bool clivalue_name_exact_match(const clivalue_t *val, char *buf, const char *cmdline, uint8_t var_name_length);
pgn_t clivalue_get_pgn(const clivalue_t *val);
// Returns the minimum valid value for the given clivalue_t. clivalue_min_t
// depends on the target and build options, but will always be a signed
// integer (e.g. intxx_t,)
clivalue_min_t clivalue_get_min(const clivalue_t *val);
// Returns the maximum valid value for the given clivalue_t. clivalue_max_t
// depends on the target and build options, but will always be an unsigned
// integer (e.g. uintxx_t,)
clivalue_max_t clivalue_get_max(const clivalue_t *val);

1452
src/main/fc/settings.yaml Normal file
View file

File diff suppressed because it is too large Load diff

850
tools/settings.rb Normal file
View file

@ -0,0 +1,850 @@
#!/usr/bin/env ruby
# This file is part of INAV.
#
# author: Alberto Garcia Hierro <alberto@garciahierro.com>
#
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this file,
# You can obtain one at http://mozilla.org/MPL/2.0/.
#
# Alternatively, the contents of this file may be used under the terms
# of the GNU General Public License Version 3, as described below:
#
# This file is free software: you may copy, redistribute and/or modify
# it under the terms of the GNU General Public License as published by the
# Free Software Foundation, either version 3 of the License, or (at your
# option) any later version.
#
# This file is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General
# Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see http://www.gnu.org/licenses/.
require 'fileutils'
require 'open3'
require 'set'
require 'shellwords'
require 'stringio'
require 'tmpdir'
require 'yaml'
DEBUG = false
INFO = false
def dputs(s)
puts s if DEBUG
end
def lputs(s)
puts if DEBUG or INFO
end
class Object
def is_number_kind?
self.kind_of?(Integer) || self.kind_of?(Float)
end
end
class String
def is_number?
true if Float(self) rescue false
end
end
class StringIO
def write_byte(b)
self << [b].pack("C*")
end
def write_uvarint(x)
while x >= 0x80
write_byte((x & 0xFF) | 0x80)
x >>= 7
end
write_byte(x)
end
def to_carr
return string.bytes.to_s.sub('[', '{').sub(']', '}')
end
end
class NameEncoder
attr_reader :max_length
def initialize(names, max_length)
@names = names
@max_length = max_length
# Key is word, value is number of uses
@words = Hash.new(0)
# Most used words first
@words_by_usage = []
# Words that shouldn't be split because
# their encoding is too long
@non_split = Set.new
# Key is the name, value is its encoding
@encoded = Hash.new
update_words
encode_names
end
def uses_byte_indexing
@words.length < 255
end
def words
@words_by_usage
end
def estimated_size(settings_count)
size = 0
@words.each do |word, count|
size += word.length + 1
end
return size + @max_length * settings_count
end
def format_encoded_name(name)
encoded = @encoded[name]
raise "Name #{name} was not encoded" if encoded == nil
return encoded.to_carr
end
private
def split_words(name)
if @non_split.include?(name)
return [name]
end
return name.split('_')
end
def update_words
@words.clear
@names.each do |name|
split_words(name).each do |word|
@words[word] += 1
end
end
# Sort by usage, then alphabetically
@words_by_usage = @words.keys().sort do |x, y|
ux = @words[x]
uy = @words[y]
if ux != uy
uy <=> ux
else
x <=> y
end
end
end
def encode_names
@encoded.clear()
@names.each do |name|
buf = StringIO.new
split_words(name).each do |word|
pos = @words_by_usage.find_index(word)
raise "Word #{word} not found in words array" if pos == nil
# Zero indicates end of words, first word in
# the array starts at 1 ([0] is NULL).
p = pos + 1
if uses_byte_indexing
buf.write_byte(p)
else
buf.write_uvarint(p)
end
end
if buf.length > @max_length
# TODO: print in verbose mode
# fmt.Printf("encoding %q took %d bytes (>%d), adding it as single word\n", v, len(data), e.MaxEncodedLength)
@non_split << name
update_words
return encode_names
end
while buf.length < @max_length
buf.write_byte(0)
end
@encoded[name] = buf
end
end
end
class ValueEncoder
attr_reader :values
def initialize(values, constants)
min = 0
max = 0
valuesHash = Hash.new(0)
values.each do |v|
min = [min, v].min
max = [max, v].max
valuesHash[v] += 1
end
# Sorted by usage, most used first
@values = valuesHash.keys().sort do |x, y|
ux = valuesHash[x]
uy = valuesHash[y]
if ux != uy
uy <=> ux
else
x <=> y
end
end
@constants = constants
@min = min
@max = max
end
def min_type
[8, 16, 32].each do |x|
if @min >= -(2**(x-1))
return "int#{x}_t"
end
end
raise "cannot represent minimum value #{@min} with int32_t"
end
def max_type
[8, 16, 32].each do |x|
if @max < 2**x
return "uint#{x}_t"
end
end
raise "cannot represent maximum value #{@max} with uint32_t"
end
def index_bytes
bits = Math.log2(@values.length).ceil
bytes = (bits / 8.0).ceil.to_i
if bytes > 1
raise "too many bytes required for value index: #{bytes}"
end
return bytes
end
def encode_values(min, max)
buf = StringIO.new
encode_value(buf, min)
encode_value(buf, max)
return buf.to_carr
end
private
def encode_value(buf, val)
v = val || 0
if !v.is_number_kind?
v = @constants[val]
if v == nil
raise "Could not resolve constant #{val}"
end
end
pos = @values.find_index(v)
if pos < 0
raise "Could not encode value not in array #{v}"
end
buf.write_byte(pos)
end
end
OFF_ON_TABLE = Hash["name" => "off_on", "values" => ["OFF", "ON"]]
class Generator
def initialize(src_root, settings_file)
@src_root = src_root
@settings_file = settings_file
@output_dir = File.dirname(settings_file)
@count = 0
@max_name_length = 0
@tables = Hash.new
@used_tables = Set.new
@enabled_tables = Set.new
@data = YAML.load_file(settings_file)
initialize_tables
check_conditions
sanitize_fields
initialize_name_encoder
initialize_value_encoder
end
def write_files
header_file = File.join(@output_dir, "settings_generated.h")
impl_file = File.join(@output_dir, "settings_generated.c")
write_header_file(header_file)
write_impl_file(impl_file)
end
def print_stats
puts "#{@count} settings"
puts "words table has #{@name_encoder.words.length} words"
word_idx = @name_encoder.uses_byte_indexing ? "byte" : "uvarint"
puts "name encoder uses #{word_idx} word indexing"
puts "each setting name uses #{@name_encoder.max_length} bytes"
puts "#{@name_encoder.estimated_size(@count)} bytes estimated for setting name storage"
values_size = @value_encoder.values.length * 4
puts "value storage uses #{values_size} bytes"
value_idx_size = @value_encoder.index_bytes * 2
value_idx_total = value_idx_size * @count
puts "value indexing uses #{value_idx_size} per setting, #{value_idx_total} bytes total"
puts "#{value_idx_size+value_idx_total} bytes estimated for value storage"
buf = StringIO.new
buf << "#include \"fc/settings.h\"\n"
buf << "char (*dummy)[sizeof(clivalue_t)] = 1;\n"
stderr = compile_test_file(buf)
puts "sizeof(clivalue_t) = #{/char \(\*\)\[(\d+)\]/.match(stderr)[1]}"
end
private
def write_header_file(file)
buf = StringIO.new
buf << "#pragma once\n"
# Write clivalue_t size constants
buf << "#define CLIVALUE_MAX_NAME_LENGTH #{@max_name_length+1}\n" # +1 for the terminating '\0'
buf << "#define CLIVALUE_ENCODED_NAME_MAX_BYTES #{@name_encoder.max_length}\n"
if @name_encoder.uses_byte_indexing
buf << "#define CLIVALUE_ENCODED_NAME_USES_BYTE_INDEXING\n"
end
buf << "#define CLIVALUE_TABLE_COUNT #{@count}\n"
offset_type = "uint16_t"
if can_use_byte_offsetof
offset_type = "uint8_t"
end
buf << "typedef #{offset_type} clivalue_offset_t;\n"
pgn_count = 0
foreach_enabled_group do |group|
pgn_count += 1
end
buf << "#define CLIVALUE_PGN_COUNT #{pgn_count}\n"
# Write type definitions and constants for min/max values
buf << "typedef #{@value_encoder.min_type} clivalue_min_t;\n"
buf << "typedef #{@value_encoder.max_type} clivalue_max_t;\n"
buf << "#define CLIVALUE_MIN_MAX_INDEX_BYTES #{@value_encoder.index_bytes*2}\n"
# Write lookup table constants
table_names = ordered_table_names()
buf << "enum {\n"
table_names.each do |name|
buf << "\t#{table_constant_name(name)},\n"
end
buf << "\tLOOKUP_TABLE_COUNT,\n"
buf << "};\n"
# Write table pointers
table_names.each do |name|
buf << "extern const char *#{table_variable_name(name)}[];\n"
end
File.open(file, 'w') {|file| file.write(buf.string)}
end
def write_impl_file(file)
buf = StringIO.new
add_header = ->(h) {
buf << "#include \"#{h}\"\n"
}
add_header.call("platform.h")
add_header.call("config/parameter_group_ids.h")
add_header.call("settings.h")
foreach_enabled_group do |group|
(group["headers"] || []).each do |h|
add_header.call(h)
end
end
# Write PGN arrays
pgn_steps = []
pgns = []
foreach_enabled_group do |group|
count = 0
group["members"].each do |member|
if is_condition_enabled(member["condition"])
count += 1
end
end
pgn_steps << count
pgns << group["name"]
end
buf << "const pgn_t cliValuePgn[] = {\n"
pgns.each do |p|
buf << "\t#{p},\n"
end
buf << "};\n"
buf << "const uint8_t cliValuePgnCounts[] = {\n"
pgn_steps.each do |s|
buf << "\t#{s},\n"
end
buf << "};\n"
# Write word list
buf << "static const char *cliValueWords[] = {\n"
buf << "\tNULL,\n"
@name_encoder.words.each do |w|
buf << "\t#{w.inspect},\n"
end
buf << "};\n"
# Write the tables
table_names = ordered_table_names()
table_names.each do |name|
buf << "const char *#{table_variable_name(name)}[] = {\n"
tbl = @tables[name]
tbl["values"].each do |v|
buf << "\t#{v.inspect},\n"
end
buf << "};\n"
end
buf << "const lookupTableEntry_t cliLookupTables[] = {\n"
table_names.each do |name|
vn = table_variable_name(name)
buf << "\t{ #{vn}, sizeof(#{vn}) / sizeof(char*) },\n"
end
buf << "};\n"
# Write min/max values table
buf << "const uint32_t cliValueMinMaxTable[] = {\n"
@value_encoder.values.each do |v|
buf << "\t#{v},\n"
end
buf << "};\n"
case @value_encoder.index_bytes
when 1
buf << "typedef uint8_t clivalue_min_max_idx_t;\n"
buf << "#define CLIVALUE_INDEXES_GET_MIN(val) (val->config.minmax.indexes[0])\n"
buf << "#define CLIVALUE_INDEXES_GET_MAX(val) (val->config.minmax.indexes[1])\n"
else
raise "can't encode indexed values requiring #{@value_encoder.index_bytes} bytes"
end
# Write clivalues
buf << "const clivalue_t cliValueTable[] = {\n"
last_group = nil
foreach_enabled_member do |group, member|
if group != last_group
last_group = group
buf << "\t// #{group["name"]}\n"
end
buf << "\t{ #{@name_encoder.format_encoded_name(member["name"])}, "
buf << "#{var_type(member["type"])} | #{value_type(group)}"
tbl = member["table"]
if tbl
buf << " | MODE_LOOKUP"
buf << ", .config.lookup = { #{table_constant_name(tbl)} }"
else
enc = @value_encoder.encode_values(member["min"], member["max"])
buf << ", .config.minmax.indexes = #{enc}"
end
buf << ", offsetof(#{group["type"]}, #{member["field"]}) },\n"
end
buf << "};\n"
File.open(file, 'w') {|file| file.write(buf.string)}
end
def var_type(typ)
case typ
when "uint8_t", "bool"
return "VAR_UINT8"
when "int8_t"
return "VAR_INT8"
when "uint16_t"
return "VAR_UINT16"
when "int16_t"
return "VAR_INT16"
when "uint32_t"
return "VAR_UINT32"
when "float"
return "VAR_FLOAT"
else
raise "unknown variable type #{typ.inspect}"
end
end
def value_type(group)
return group["value_type"] || "MASTER_VALUE"
end
def is_condition_enabled(cond)
return !cond || @true_conditions.include?(cond)
end
def foreach_enabled_member
@data["groups"].each do |group|
if is_condition_enabled(group["condition"])
group["members"].each do |member|
if is_condition_enabled(member["condition"])
yield group, member
end
end
end
end
end
def foreach_enabled_group
last = nil
foreach_enabled_member do |group, member|
if last != group
last = group
yield group
end
end
end
def foreach_member
@data["groups"].each do |group|
group["members"].each do |member|
yield group, member
end
end
end
def foreach_group
last = nil
foreach_member do |group, member|
if last != group
last = group
yield group
end
end
end
def initialize_tables
@data["tables"].each do |tbl|
name = tbl["name"]
if @tables.key?(name)
raise "Duplicate table name #{name}"
end
@tables[name] = tbl
end
end
def ordered_table_names
@enabled_tables.to_a().sort()
end
def table_constant_name(name)
upper = name.upcase()
"TABLE_#{upper}"
end
def table_variable_name(name)
"table_#{name}"
end
def can_use_byte_offsetof
buf = StringIO.new
foreach_enabled_member do |group, member|
typ = group["type"]
field = member["field"]
buf << "static_assert(offsetof(#{typ}, #{field}) < 255, \"#{typ}.#{field} is too big\");\n"
end
stderr = compile_test_file(buf)
return !stderr.include?("static assertion failed")
end
def mktmpdir
# Use a temporary dir reachable by relative path
# since g++ in cygwin fails to open files
# with absolute paths
tmp = File.join("obj", "tmp")
FileUtils.mkdir_p(tmp) unless File.directory?(tmp)
value = yield(tmp)
FileUtils.remove_dir(tmp)
value
end
def compile_test_file(prog)
buf = StringIO.new
# cstddef for offsetof()
headers = ["target.h", "platform.h", "cstddef"]
@data["groups"].each do |group|
gh = group["headers"]
if gh
headers.concat gh
end
end
headers.each do |h|
if h
buf << "#include \"#{h}\"\n"
end
end
buf << "\n"
buf << prog.string
mktmpdir do |dir|
file = File.join(dir, "test.cpp")
File.open(file, 'w') {|file| file.write(buf.string)}
cflags = Shellwords.split(ENV["CFLAGS"] || "")
args = ["arm-none-eabi-g++"]
cflags.each do |flag|
# Don't generate temporary files
if flag == "" || flag == "-MMD" || flag == "-MP" || flag.start_with?("-save-temps")
next
end
if flag.start_with? "-std="
flag = "-std=c++11"
end
if flag.start_with? "-D'"
# Cleanup flag. Done by the shell when called from
# it but we must do it ourselves becase we're not
# calling the compiler via shell.
flag = "-D" + flag[3..-2]
end
args << flag
end
args << "-o" << File.join(dir, "test") << file
dputs "Compiling #{buf.string}"
stdout, stderr = Open3.capture3(Shellwords.join(args))
dputs "Output: #{stderr}"
stderr
end
end
def check_conditions
buf = StringIO.new
conditions = Set.new
add_condition = -> (c) {
if c && !conditions.include?(c)
conditions.add(c)
buf << "#ifdef #{c}\n"
buf << "#pragma message(#{c.inspect})\n"
buf << "#endif\n"
end
}
foreach_member do |group, member|
add_condition.call(group["condition"])
add_condition.call(member["condition"])
end
stderr = compile_test_file(buf)
@true_conditions = Set.new
stderr.scan(/#pragma message\(\"(.*)\"\)/).each do |m|
@true_conditions << m[0]
end
end
def sanitize_fields
pending_types = Hash.new
has_booleans = false
foreach_enabled_member do |group, member|
if !group["name"]
raise "Missing group name"
end
if !member["name"]
raise "Missing member name in group #{group["name"]}"
end
table = member["table"]
if table
if !@tables[table]
raise "Member #{member["name"]} references non-existing table #{table}"
end
@used_tables << table
end
if !member["field"]
member["field"] = member["name"]
end
typ = member["type"]
if !typ
pending_types[member] = group
elsif typ == "bool"
has_booleans = true
member["type"] = "uint8_t"
member["table"] = OFF_ON_TABLE["name"]
end
end
if has_booleans
@tables[OFF_ON_TABLE["name"]] = OFF_ON_TABLE
@used_tables << OFF_ON_TABLE["name"]
end
resolve_types pending_types unless !pending_types
foreach_enabled_member do |group, member|
@count += 1
@max_name_length = [@max_name_length, member["name"].length].max
if member["table"]
@enabled_tables << member["table"]
end
end
end
def resolve_types(pending)
# TODO: Loop to avoid reaching the maximum number
# of errors printed by the compiler.
prog = StringIO.new
prog << "int main() {\n"
ii = 0
members = Hash.new
pending.each do |member, group|
var = "var_#{ii}"
members[ii] = member
ii += 1
gt = group["type"]
mf = member["field"]
prog << "#{gt} #{var}; #{var}.#{mf}.__type_detect_;\n"
end
prog << "return 0;\n"
prog << "};\n"
stderr = compile_test_file(prog)
stderr.scan(/var_(\d+).*?', which is of non-class type '(.*)'/).each do |m|
member = members[m[0].to_i]
case m[1]
when "int8_t {aka signed char}"
typ = "int8_t"
when "uint8_t {aka unsigned char}"
typ = "uint8_t"
when "int16_t {aka short int}"
typ = "int16_t"
when "uint16_t {aka short unsigned int}"
typ = "uint16_t"
when "uint32_t {aka long unsigned int}"
typ = "uint32_t"
when "float"
typ = "float"
else
raise "Unknown type #{m[1]} when resolving type for setting #{member["name"]}"
end
dputs "#{member["name"]} type is #{typ}"
member["type"] = typ
end
# Make sure all types have been resolved
foreach_enabled_member do |group, member|
if !member["type"]
raise "Could not resolve type for member #{member["name"]} in group #{group["name"]}"
end
end
end
def initialize_name_encoder
names = []
foreach_enabled_member do |group, member|
names << member["name"]
end
best = nil
(3..7).each do |v|
enc = NameEncoder.new(names, v)
if best == nil || best.estimated_size(@count) > enc.estimated_size(@count)
best = enc
end
end
dputs "Using name encoder with max_length = #{best.max_length}"
@name_encoder = best
end
def initialize_value_encoder
values = []
constants = []
add_value = -> (v) {
v = v || 0
if v.is_number_kind? || (v.class == String && v.is_number?)
values << v.to_i
else
constants << v
end
}
foreach_enabled_member do |group, member|
add_value.call(member["min"])
add_value.call(member["max"])
end
constantValues = resolve_constants(constants)
# Count values used by constants
constants.each do |c|
values << constantValues[c]
end
@value_encoder = ValueEncoder.new(values, constantValues)
end
def resolve_constants(constants)
return nil unless constants.length > 0
s = Set.new
constants.each do |c|
s << c
end
dputs "#{constants.length} constants to resolve"
# Since we're relying on errors rather than
# warnings to find these constants, the compiler
# might reach the maximum number of errors and stop
# compilation, so we might need multiple passes.
re = /required from 'class expr_(.*?)<(.*)>'/
values = Hash.new
while s.length > 0
buf = StringIO.new
buf << "template <int64_t V> class Fail {\n"
# Include V in the static_assert so it's shown
# in the error condition.
buf << "static_assert(V == 42 && 0 == 1, \"FAIL\");\n"
buf << "public:\n"
buf << "Fail() {};\n"
buf << "int64_t v = V\n"
buf << "};\n"
ii = 0
s.each do |c|
cls = "expr_#{c}"
buf << "template <int64_t V> class #{cls}: public Fail<V> {};\n"
buf << "#{cls}<#{c}> var_#{ii};\n"
ii += 1
end
stderr = compile_test_file(buf)
matches = stderr.scan(re)
if matches.length == 0
puts stderr
raise "No more matches looking for constants"
end
matches.each do |m|
c = m[0]
v = m[1]
# gcc 6.3 includes an ul or ll prefix after the
# constant expansion, while gcc 7.1 does not
v = v.tr("ul", "")
nv = v.to_i
values[c] = nv
s.delete(c)
dputs "Constant #{c} resolved to #{nv}"
end
end
values
end
end
def usage
puts "Usage: ruby #{__FILE__} <source_dir> <settings_file>"
end
if __FILE__ == $0
verbose = ENV["V"] == "1"
src_root = ARGV[0]
settings_file = ARGV[1]
if src_root.nil? || settings_file.nil?
usage()
exit(1)
end
gen = Generator.new(src_root, settings_file)
gen.write_files()
if verbose
gen.print_stats()
end
end