* Make sure all Python modules are documented by adding a bit of shell
to .ci/docs.sh
* Remove non-existing module references from .rst
* Fix various warnings from sphinx by adjusting Python docstrings
* Add class member docs to ApkindexBlock
Generalise pmb.helpers.other.cache with a more python decorator.
The Cache decorator takes a list of function arguments to use as cache
keys, keyword args can be used to restrict caching so that it is skipped
entirely unless the attribute has a specific value.
For example, pmb.helpers.pmaports.get() has the decorator:
@Cache("pkgname", subpackages=True)
This means the return value will be cached only when subpackages is
True, otherwise it will always miss.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Cease merging pmbootstrap.cfg into args, implement a Context type to let
us pull globals out of thin air (as an intermediate workaround) and rip
args out of a lot of the codebase.
This is just a first pass, after this we can split all the state that
leaked over into Context into types with narrower scopes (like a
BuildContext(), etc).
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
With the new chroot type, we can now write fancy paths in the pythonic
way. Convert most of the codebase over, as well as adding various other
type hints.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
We use a custom verbose log level in pmbootstrap, unfortunately it isn't
possible to correctly type this due to some limitations in the logging
library [1], [2].
Given that our usecase is fairly simple, we can just wrap the module
with our own so we only have to tell mypy to ignore the error once
instead of at every callsite.
[1]: https://github.com/cryptax/droidlysis/issues/15
[2]: https://github.com/python/typing/discussions/980
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Introduce a new module: pmb.core to contain explicitly typed pmbootstrap
API. The first component being Suffix and SuffixType. This explicitly
defines what suffixes are possible, future changes should aim to further
constrain this API (e.g. by validating against available device
codenames or architectures for buildroot suffixes).
Additionally, migrate the entire codebase over to using pathlib.Path.
This is a relatively new part of the Python standard library that uses a
more object oriented model for path handling. It also uses strong type
hinting and has other features that make it much cleaner and easier to
work with than pure f-strings. The Chroot class overloads the "/"
operator the same way the Path object does, allowing one to write paths
relative to a given chroot as:
builddir = chroot / "home/pmos/build"
The Chroot class also has a string representation ("native", or
"rootfs_valve-jupiter"), and a .path property for directly accessing the
absolute path (as a Path object).
The general idea here is to encapsulate common patterns into type hinted
code, and gradually reduce the amount of assumptions made around the
codebase so that future changes are easier to implement.
As the chroot suffixes are now part of the Chroot class, we also
implement validation for them, this encodes the rules on suffix naming
and will cause a runtime exception if a suffix doesn't follow the rules.
A common pattern in APKBUILDs, is to introduce custom variables prefixed
with underscores that get then used in makedepends and other variables.
For example:
_wlrootsmakedepends="
eudev-dev
# ...
"
makedepends="
# ...
$_wlrootsmakedepends
"
Adjust the APKBUILD parser code, so it parses all top-level variables
and can use them further below when referenced inside other variables.
Before returning the parsed APKBUILD data, remove all variables that are
not in pmbootstrap's list of known APKBUILD parsing attributes (so the
result is the same).
I've compared "pmbootstrap apkbuild_parse" (which parses all APKBUILDs
in the currently checked out pmaports dir), before and after this
change, and the result is the same except for having more variables
successfully replaced.
- Performance Note-
This new implementation is actually faster than the previous one,
because we don't need to iterate through all known keys on each line of
the APKBUILDs. On my machine, average of 3 runs, parsing all APKBUILDs
of current pmaports master takes about half as long as with the previous
implementation.
$ time pmbootstrap -q apkbuild_parse >/dev/null
-> old code: 0.954
-> new code: 0.483
If there are multiple sections to a subpackage declaration, the middle one
(item 1 in the sequence in this case) is the custom function name which we
should use instead of the deduced one.
For example:
$pkgname-something-separate:something_separate:noarch
Here, the subpackage is called $pkgname-something-separate, but function names
cannot contain hyphens, so it's instead given the custom name
something_separate where the hyphen is replaced by an underscore. At the end,
the architecture of the subpackage is also overriden as noarch instead of
whatever architecture the main package has.
However, it is also possible to only override architecture of a subpackage and
not the function name.
In such cases, we get this:
$pkgname-something::noarch
We still have a "middle section" (item 1 in the sequence), but it will be
empty. As such, we not only need to check whether there are more than 1
subpackage part, but also whether that extra subpackage part is an empty string
or not.
This fixes an issue where pmbootstrap would not end up just setting the
properties of subpackages with a set arch but no custom function as
None instead of giving them proper depends, install, pkgdesc, et cetera
properties.
Due to that logging.verbose is "monkeypatched" into logging and not
present by default in the module, Mypy isn't able to pick up that it
exists. As such, just disable the check altogether in this file for
now.
The apkbuild parser could not handle cases where a line ends in a
comment but the value is not quoted.
E.g. this:
pkgver=1.0 # I'm a comment, look at me
was being parsed to a value like this:
1.0 # I'm a comment, look at me
... which is obviously wrong. This strips off any trailing comment on
the line, so it's parsed to the correct value.
fixes#2087
Replace "args.cache" with a global variable in order to
avoid passing "args" to all functions. This is a step to get rid of this
args-passed-to-all-functions pattern in pmbootstrap.
Unmaintained devices are device packages that:
- Are known to be broken in some way without an active maintainer
who can investigate how to fix it, or
- Have not received any updates for a very long time, or
- Are discouraged from using because they are just intended for testing.
An example for this are ports using the downstream kernel for devices
which have a mainline port that is working quite well.
Unmaintained devices are still built by bpo (otherwise it would not make
sense to keep them), but they do not show up in "pmbootstrap init".
However, it is possible to manually select them by entering the name.
pmbootstrap will warn in that case.
Unmaintained packages should have a # Unmaintained: <reason> comment
in the APKBUILD, this comment is displayed in "pmbootstrap init"
so that the user knows why the device should not be used unless they
know what they are doing.
Made changes to limit the line length in following files,
- pmb/parse/_apkbuild.py
- pmb/parse/apkindex.py
- pmb/parse/binfmt_info.py
- pmb/parse/deviceinfo.py
- test/test_parse_apkbuild.py
Added the above files in E501 flake8 command list.
Substitute f-string for string concatenation.
Alpine indicates with arch="", that a package should temporarily not be
built for any architecture. Support this in postmarketOS too by not
complaining in the APKBUILD parser if arch is empty.
Adjust pmb.build.autodetect.arch and pmb.build.menuconfig.get_arch, so
both don't fail with an IndexError when encountering a disabled package.
Co-Authored-By: Luca Weiss <luca@z3ntu.xyz>
In the future, device ports will be located in a subdirectory
below device/... (e.g. device/testing/device-...).
Replace all occurrences of device/* with a glob that checks the
subdirectories instead.
Note: To ensure that this always works properly we should also add some
checks that all devices are indeed located under one of the supported
subdirectories (i.e. testing/community/main).
Change the glob for pmaports to <aports>/**/APKBUILD.
This allows using subdirectories for organization outside of device/
as well.
While at it, also remove unnecessary "#!/usr/bin/env python3" in files
that only get imported, and adjust other empty/comment lines in the
beginnings of the files for consistency.
This makes files easier to read, and makes the pmbootstrap codebase more
consistent with the build.postmarketos.org codebase.
At the moment we have a simple subpkgdesc() function that can only
parse "pkgdesc" from subpackages, without support for any variables.
But we have a quite nice variable parser now that can be extended
to work for subpackages.
Simply put this works by:
- Finding the lines that belong to the subpackage function
- Stripping indentation (tab)
- Parsing relevant attributes similar to the apkbuild() function
The "subpackages" in the parsed APKBUILD are replaced by a dict
of subpkgname: {"pkgdesc": "...", "depends": "..."} which are
parsed from the subpackage function (if found).
This makes it possible to get the "depends" of a subpackage.
At the moment we parse all attributes, split them, and eventually
join them back together for variable replacement.
Replacing variables immediately after parsing (before splitting)
has several advantages:
- No need to handle different value types
(e.g. lists by joining them every time they are accessed)
- Variables like depends="$depends ..." are handled directly
by the variable parser
- APKBUILDs are shell scripts, so we match abuild more closely
if variables defined later do not affect previous attributes
Theoretically it is possible to reference variables before they are defined.
In the shell, these will simply be evaluated to an empty string.
In preparation of replacing variables immediately after parsing attributes,
these variables will be no longer replaced correctly.
We can simplify the code further, and avoid this problem by initializing
the dict with the default values, replacing them with the real values
from the APKBUILD.
This will also avoid a (somewhat unrelated) error in the bootimg test:
File "pmb/parse/_apkbuild.py", line 46, in replace_variable
apkbuild["pkgname"], match.group(0),
KeyError: 'pkgname'
When defining a new kernel subpackage with a "-" in it
(e.g. $pkgname-kernel-mainline-modem), then pmbootstrap is unable to
find the function that builds the subpackage:
ERROR: Could not find subpackage function, no line starts with 'kernel_mainline-modem() {'
This is because it assumes that a $pkgname-kernel-<name> subpackage
is built by a kernel_<name> function, but this does not have to be the case.
We should really respect the name of the subpackage function that
is specified when defining the subpackage, but unfortunately it is
stripped away in cut_off_function_names().
For now let's fix this by replacing "-" with "_", but ideally the
APKBUILD parser API should be refactored to expose the subpackage
function in the future.
Properly ignore comments at the end of lines, instead of assuming that
all lines below belong to the attribute:
subpackages="$pkgname-dev" # $pkgname-lang
Fixes build.postmarketos.org#61, where pmbootstrap would assume that a
random package provides "make", just because the word "make" is written
somewhere below subpackages=" in the APKBUILD and it is parsed
incorrectly.
While at it, also support the ' character for quotations and detect if
a quotation for a value was started, but there is no end quotation sign
in the rest of the file.
I've added tests, and manually checked that this did not introduce any
parsing bugs for all the APKBUILDs in pmaports.git, by running
'pmbootstrap apkbuild_parse' with the old and new code, and diffing the
result.
Properly handle the following two cases in APKBUILDs:
* depends="$depends ..."
* depends="${depends} ..."
First I've attempted to refactor the parsing code to do this in a more
generic way. But I've realized that it would make more sense to retire
the python based APKBUILD parsing approach altogether and finally use a
shell script parser. Let's discuss this in #1801.
Add a new action that lists all aports, for which no binary packages
exist. Only list packages that can be built for the relevant arch
(specified with --arch). This works recursively: when a package can be
built for a certain arch, but one of its dependencies
(or their depends) can not be built for that arch, then don't list it.
This action will be used for the new sr.ht based build infrastructure,
to figure out which packages need to be built ahead of time (so we can
trigger each of them as single build job). Determining the order of the
packages to be built is not determined with pmbootstrap, the serverside
code of build.postmarketos.org takes care of that.
For testing purposes, a single package can also be specified and the
action will list if it can be built for that arch with its
dependencies, and what needs to be built exactly.
Add pmb/helpers/package.py to hold functions that work on both pmaports
and (binary package) repos - in contrary to the existing
pmb/helpers/pmaports.py (see previous commit) and pmb/helpers/repo.py,
which only work with one of those.
Refactoring:
* pmb/helpers/pmaports.py: add a get_list() function, which lists all
aports and use it instead of writing the same glob loop over and over
* add pmb.helpers.pmaports.get(), which finds an APKBUILD and parses it
in one step.
* rename pmb.build._package.check_arch to ...check_arch_abort to
distinguish it from the other check_arch function
