Using pmbootstrap v3 with the old config can cause problems, for example
when having $WORK in the pmaports dir instead of the actual work path.
This is not supported anymore by v3 to reduce complexity. The format of
how mirrors are stored in the config also has changed.
Use a separate config file, so users can go back from v3 to 2.3.x if
they need to (for figuring out a regression) and so users won't run into
bugs when moving from 2.3.x to v3.
Remove a bunch of commented out code:
* debug leftovers
* update_work(): a function that used to replace $WORK in all kinds of
arguments to pmbootstrap, with the actual work dir. This seemed like a
good idea when I made this initially, but in hindsight it's clear that
this made args just much more complex. I'm glad that Caleb removed
this. Now $WORK only gets removed in one place, for mount points,
which makes much more sense.
* args.from_argparse and related comment: pmbootstrap used to make a
full copy of its args, so in some cases it could go back before $WORK
was replaced IIRC. This isn't great, but even worse is that this
caused each stack trace to say something like infinite recursion on
args. I'm also glad that Caleb got rid of this :)
The aports argument is a list of paths now, so iterate over each of them
when checking if they exist.
Fix for:
INFO:root:ERROR: 'list' object has no attribute 'exists'
Get rid of config.mirror_alpine and mirrors_postmarketos and make sure
they get migrated over for existing users.
mirrors_postmarketos being a list was always a bit off, but now we have
per-aports mirrors which make a lot more sense for what we're trying to
do with systemd.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Move pmb/parse/arch.py over to core and refactor it as an Arch type,
similar to how Chroot was done. Fix all the uses (that I can find) of
arch in the codebase that need adjusting.
The new Arch type is an Enum, making it clear what architectures can be
represented and making it much easier to reason about. Since we support
~5 (kinda) different representations of an Architecture (Alpine, Kernel,
target triple, platform, and QEMU), we now formalise that the Alpine
format is what we represent internally, with methods to convert to any
of the others as-needed.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Introduce a Deviceinfo class and use it rather than the dictionary. This
gives us sweet sweet autocomplete, and lays the foundation for having a
proper deviceinfo validator in the future.
Additionally, continue refactoring out args...
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Cease merging pmbootstrap.cfg into args, implement a Context type to let
us pull globals out of thin air (as an intermediate workaround) and rip
args out of a lot of the codebase.
This is just a first pass, after this we can split all the state that
leaked over into Context into types with narrower scopes (like a
BuildContext(), etc).
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
We can't totally escape the need for some runtime state defined by args.
To make the migration easier, introduce a global "Context" class and
move some of the read-only global options there.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
With the new chroot type, we can now write fancy paths in the pythonic
way. Convert most of the codebase over, as well as adding various other
type hints.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
The init_cache() function just assigned some default constants, simplify
this by just declaring it that way to begin with, as well as adding type
hints.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Introduce a new module: pmb.core to contain explicitly typed pmbootstrap
API. The first component being Suffix and SuffixType. This explicitly
defines what suffixes are possible, future changes should aim to further
constrain this API (e.g. by validating against available device
codenames or architectures for buildroot suffixes).
Additionally, migrate the entire codebase over to using pathlib.Path.
This is a relatively new part of the Python standard library that uses a
more object oriented model for path handling. It also uses strong type
hinting and has other features that make it much cleaner and easier to
work with than pure f-strings. The Chroot class overloads the "/"
operator the same way the Path object does, allowing one to write paths
relative to a given chroot as:
builddir = chroot / "home/pmos/build"
The Chroot class also has a string representation ("native", or
"rootfs_valve-jupiter"), and a .path property for directly accessing the
absolute path (as a Path object).
The general idea here is to encapsulate common patterns into type hinted
code, and gradually reduce the amount of assumptions made around the
codebase so that future changes are easier to implement.
As the chroot suffixes are now part of the Chroot class, we also
implement validation for them, this encodes the rules on suffix naming
and will cause a runtime exception if a suffix doesn't follow the rules.
Without this, pmbootstrap will give an error if you have a nonexistent
device selected when trying to checksum a package. This can be annoying
when switching between different branches where devices may or may not
exist, and I don't think these initialisation steps are necessary for
the checksum action anyway.
Replace "args.cache" with a global variable in order to
avoid passing "args" to all functions. This is a step to get rid of this
args-passed-to-all-functions pattern in pmbootstrap.
Replace "args.arch_native" with the direct function call in order to
avoid passing "args" to all functions. This is a step to get rid of this
args-passed-to-all-functions pattern in pmbootstrap.
Replace "args.logfd" with "pmb.helpers.logging.logfd" in order to avoid
passing "args" to all functions that only use it to write to logfd. This
is the first step to get rid of this args-passed-to-all-functions
pattern in pmbootstrap.
Allows user to choose one of the mirrors from
https://mirrors.postmarketos.org.
Example:
[1] Mirror 1 (Location 1)
[2] Mirror 2 (Location 2)
[3] Mirror 3 (Location 3)
Select a mirror [1]: 2
Co-Authored-By: Alexey Min <alexey.min@gmail.com>
Do not make the parsed pmaports.cfg from pmaports.git available as
args.pmaports anymore. This de-bloats the args variable a bit.
First I thought that we didn't even need to cache it, but it was pointed
out that later patches do access it frequently to read the current channel
from pmaports.cfg. Therefore it is using a cache now.
Related: #1879, #1855
Prepare to base postmarketOS on Alpine stable by parsing the new
channels.cfg file in pmaports.git, that describes which channel
needs which branches and mirror dirs from postmarketOS and Alpine.
Use the information in pmb.helpers.git.get_branches_official() first,
more is coming in follow-up commits.
Read the file from origin/master, so we get the latest fetched version
even if the last checked out master branch is not up-to-date (think of
currently checked out release branch instead of master, master will
never be updated to point to latest origin/master). Allow to override
the file with a new --config-channels parameter.
Related: https://postmarketos.org/channels.cfg
Prevent "pmbootstrap pull" from failing with:
NOTE: your pmaports folder has version 4, but version 6 is required.
ERROR: Run 'pmbootstrap pull' to update your pmaports.
Fixes: #1900
While at it, also remove unnecessary "#!/usr/bin/env python3" in files
that only get imported, and adjust other empty/comment lines in the
beginnings of the files for consistency.
This makes files easier to read, and makes the pmbootstrap codebase more
consistent with the build.postmarketos.org codebase.
Properly replace ~ with $HOME in all paths. Fix tab completion when
passing a path starting with ~ to pmaports:
$ pmbootstrap --aports ~/src/pmaports/ build linux-<TAB>
Do not fail when an APKINDEX can not be downloaded, print a WARNING
instead. This matches, what apk does. Add a new allow_404 parameter to
pmb.helpers.http.download(), and use it in pmb.helpers.repo.update()
for downloading APKINDEX files. Cache the APKINDEX URLs that gave a 404
for the session, so we do not attempt to download these again.
This is needed for the new binary repository: the initial build is done
without existing APKINDEX files, so we must not fail in that case.
Change the cache format from args.cache["offline_msg_shown"] to
args.cache["pmb.helpers.repo.update"]["offline_msg_shown"]. This is in
preparation for saving more data in the cache of
pmb.helpers.repo.update in the next commit.
Multiple -mp arguments can be used to list multiple mirrors:
$ pmbootstrap -mp=first -mp=second chroot -- cat /etc/apk/repositories
This is needed for the new build infrastructure, so we can have a WIP
repository to which we push packages until all of them are up to date,
and then publish all of them at once. Software like KDE/Plasma Mobile,
which expect a lot of packages to be updated from one version to
another will not end up with a half-way through upgrade that way.
Add a new action that lists all aports, for which no binary packages
exist. Only list packages that can be built for the relevant arch
(specified with --arch). This works recursively: when a package can be
built for a certain arch, but one of its dependencies
(or their depends) can not be built for that arch, then don't list it.
This action will be used for the new sr.ht based build infrastructure,
to figure out which packages need to be built ahead of time (so we can
trigger each of them as single build job). Determining the order of the
packages to be built is not determined with pmbootstrap, the serverside
code of build.postmarketos.org takes care of that.
For testing purposes, a single package can also be specified and the
action will list if it can be built for that arch with its
dependencies, and what needs to be built exactly.
Add pmb/helpers/package.py to hold functions that work on both pmaports
and (binary package) repos - in contrary to the existing
pmb/helpers/pmaports.py (see previous commit) and pmb/helpers/repo.py,
which only work with one of those.
Refactoring:
* pmb/helpers/pmaports.py: add a get_list() function, which lists all
aports and use it instead of writing the same glob loop over and over
* add pmb.helpers.pmaports.get(), which finds an APKBUILD and parses it
in one step.
* rename pmb.build._package.check_arch to ...check_arch_abort to
distinguish it from the other check_arch function
