The "pmbootstrap repo_missing" action is used exclusively by bpo. It
calls it only with these arguments:
pmbootstrap repo_missing --built --arch "$ARCH"
A blocker for merging systemd into pmaports master is, that the
current repo_missing code cannot display packages that are both in
extra-repos/systemd and in another path. I have considered just not
supporting this and discussed doing that with Caleb and Clayton, but we
figured it would be a major obstacle in the future to not be able to
easily override packages with systemd specific versions (currently we
need this for 3 packages).
Integrating this into the existing repo_missing code would be hacks upon
hacks. Also the scope of the current repo_missing code has many extra
features that are not used and would be extra effort to carry along:
* Allow specifying a pkgname
* Running without --built
* --overview
So I decided to replace the repo_missing code with a much simpler, more
modern implementation, that does exactly what is needed:
* Duplicate packages in systemd and non-systemd dirs are displayed
* The output always include all packages, no matter if they are already
built or not (same behavior as with --built)
* Removed --overview and selecting specific packages too
* The code for filling "repo" (either "systemd" or None) is more
resilient now, as it can use proper relative paths to the root of
pmaports. Unlike the previous implementation, it will not fail if
subdirs are added to the systemd dir.
I have made sure that the output is exactly the same as before on
current pmaports master.
Related: bpo issue 144
Related: bpo issue 140
Don't attempt to find packages of other arches in repo_missing code.
This leads to unexpected results in general (we want to check if a
binary package exists for only one specific architecture!) - and in case
of master_staging_systemd, it current causes errors because pmbootstrap
tries to fetch armhf, riscv etc. APKINDEXes for which we don't build the
staging repositories.
It was the case that packages were being queued in the wrong order,
since the dependency resolver queues the package before descending into
it's dependencies. This was a bit of a goof since the top level package
was always added last. If we add it first then we can just reverse the
queue and now everything is fine...
Additionally, the logic on when a dependency should be built was a bit
wonky. A case is added for when a dependency exists only in the source
repo and the requisite package isn't marked for build. The solver will
now build the dependency regardless. This is surely an edgecase but
somehow I ran into it, I suspect due to a bug elsewhere...
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Generalise pmb.helpers.other.cache with a more python decorator.
The Cache decorator takes a list of function arguments to use as cache
keys, keyword args can be used to restrict caching so that it is skipped
entirely unless the attribute has a specific value.
For example, pmb.helpers.pmaports.get() has the decorator:
@Cache("pkgname", subpackages=True)
This means the return value will be cached only when subpackages is
True, otherwise it will always miss.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Cease merging pmbootstrap.cfg into args, implement a Context type to let
us pull globals out of thin air (as an intermediate workaround) and rip
args out of a lot of the codebase.
This is just a first pass, after this we can split all the state that
leaked over into Context into types with narrower scopes (like a
BuildContext(), etc).
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
With the new chroot type, we can now write fancy paths in the pythonic
way. Convert most of the codebase over, as well as adding various other
type hints.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
We use a custom verbose log level in pmbootstrap, unfortunately it isn't
possible to correctly type this due to some limitations in the logging
library [1], [2].
Given that our usecase is fairly simple, we can just wrap the module
with our own so we only have to tell mypy to ignore the error once
instead of at every callsite.
[1]: https://github.com/cryptax/droidlysis/issues/15
[2]: https://github.com/python/typing/discussions/980
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Introduce a new module: pmb.core to contain explicitly typed pmbootstrap
API. The first component being Suffix and SuffixType. This explicitly
defines what suffixes are possible, future changes should aim to further
constrain this API (e.g. by validating against available device
codenames or architectures for buildroot suffixes).
Additionally, migrate the entire codebase over to using pathlib.Path.
This is a relatively new part of the Python standard library that uses a
more object oriented model for path handling. It also uses strong type
hinting and has other features that make it much cleaner and easier to
work with than pure f-strings. The Chroot class overloads the "/"
operator the same way the Path object does, allowing one to write paths
relative to a given chroot as:
builddir = chroot / "home/pmos/build"
The Chroot class also has a string representation ("native", or
"rootfs_valve-jupiter"), and a .path property for directly accessing the
absolute path (as a Path object).
The general idea here is to encapsulate common patterns into type hinted
code, and gradually reduce the amount of assumptions made around the
codebase so that future changes are easier to implement.
As the chroot suffixes are now part of the Chroot class, we also
implement validation for them, this encodes the rules on suffix naming
and will cause a runtime exception if a suffix doesn't follow the rules.
While at it, also remove unnecessary "#!/usr/bin/env python3" in files
that only get imported, and adjust other empty/comment lines in the
beginnings of the files for consistency.
This makes files easier to read, and makes the pmbootstrap codebase more
consistent with the build.postmarketos.org codebase.
Remove the recursive check, as it caused an infinite loop. It took a
very long time to complete anyway, even when it worked. The reasoning
for having the recursive check in the first place was, that we would
have device packages with arch=noarch set in the APKBUILD, but which
could only be built for a certain architecture (the device
architecture). Nowadays using arch=noarch in device packages is
forbidden, and we enforce that rule [1]. So the recursive arch check
isn't necessary anymore.
[1] ac6c0a2997
"breeze-icons" depends on "qt5-qtbase-dev", but
"pmbootstrap repo_missing" should return "qt5-qtbase" instead.
This patch fixes it, as one can see with:
$ pmbootstrap repo_missing --built breeze-icons --overview
Add a new action that lists all aports, for which no binary packages
exist. Only list packages that can be built for the relevant arch
(specified with --arch). This works recursively: when a package can be
built for a certain arch, but one of its dependencies
(or their depends) can not be built for that arch, then don't list it.
This action will be used for the new sr.ht based build infrastructure,
to figure out which packages need to be built ahead of time (so we can
trigger each of them as single build job). Determining the order of the
packages to be built is not determined with pmbootstrap, the serverside
code of build.postmarketos.org takes care of that.
For testing purposes, a single package can also be specified and the
action will list if it can be built for that arch with its
dependencies, and what needs to be built exactly.
Add pmb/helpers/package.py to hold functions that work on both pmaports
and (binary package) repos - in contrary to the existing
pmb/helpers/pmaports.py (see previous commit) and pmb/helpers/repo.py,
which only work with one of those.
Refactoring:
* pmb/helpers/pmaports.py: add a get_list() function, which lists all
aports and use it instead of writing the same glob loop over and over
* add pmb.helpers.pmaports.get(), which finds an APKBUILD and parses it
in one step.
* rename pmb.build._package.check_arch to ...check_arch_abort to
distinguish it from the other check_arch function
