The unpacking operator can unpack any iterable of any type, unlike the +
operator which only works on some sequences and requires them to be of
the same type usually. Using it allows us to get rid of the line that
disables the "operator" error code for this function.
In 0b4fb9119f (chroot: always run apk static v2 (MR 2423)) we adjusted
install_run_apk() to run apk static on the host and pass in the local
binary repo with "--repository". This function can call apk in two ways,
either with the progress bar handling or without, the second case was
never updated and still ran apk inside the chroot incorrectly and with
an incorrect --repository flag.
Let's finish the job by refactoring helpers/apk.py to support all our
usecases and pointing everything to it, removing the last few situations
where we call "pmb.chroot.root(["apk", ...]).
The apk_with_progress() function is replaced by a generic "run()"
function which takes a boolean to indicate if we should render apk
progress.
Additionally, a new cache_clean() function is added so that "pmbootstrap
zap --pkgs-online-mismatch" can FINALLY be refactored to not rely on a
chroot existing. This requires some hacks but nothing serious, see the
comments in the function for details.
The chroot.init() code is now simplified since handling the --root,
--arch, --cache-dir, and --repository flags is now all done by
apk._prepare_cmd() as and when appropriate.
Lastly, this fixes a (previously unnoticed) bug where apk.static was
actually using /var/cache/apk on your host machine for its cache... This
is definitely not good behaviour....
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
This function better belongs here, especially as it will be used outside
of the context of a chroot() soon.
Additionally, it's adjusted to take the rootfs path as its first
argument rather than a chroot, since it could operate on any rootfs.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Now that we don't need weird apk-tools hacks for systemd, we can
re-implement this optimisation and always run apk static rather than
running apk through the chroot.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Currently the usr-merge script fails, and this leads to a very confusing
error when running pmbootstrap the next time:
(stack trace)
KeyError: 'apk-tools'
Show a more helpful error instead.
Let us install packages without announcing. Useful if there is some more
contextual message display already.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
As we don't call apk from outside of the chroot, the paths need to be
adjusted to be /mnt/pmbootstrap instead of the outside paths.
This should be reverted together with the other TEMP commit after we
upgraded apk to fix this properly (see issue 2388).
Fixes: f5122420 ("TEMP: chroot: apk: run apk through QEMU again (MR 2351)")
This revers b82c4eb167 ("chroot: always run apk static (MR 2252)")
since the current iteration of the apk-tools /usr merge patches
don't handle --root properly and will behave differently based on your
host systems directory structure (specifially for /usr merge).
This commit can be reverted once we fix apk-tools.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
We recently changed how we manage chroots, requiring the user (of the
chroot) to initialize it before doing stuff like installing packages.
There are however still certain edgecases in pmbootstrap where this
doesn't get done (for example qemu/run.py would attempt to install the
kernel package for the device, but we don't initialize the rootfs chroot
in QEMU since that doesn't make sense to do). Check for and catch this
situation explicitly so we don't ruin someones day, but still be loud
about it so we can hopefully catch the remaining instances of this.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
This turns out to have been wrong, since we need to give apk the package
name AND the apk file, it installs the apk and then uninstalls it but
leaves the package installed by name.
Turns out this is necessary, *sigh*. Since we could have one locally
built package depend on another locally built package.
So yeah, re-implement this, but in a slightly lazier / hopefully faster
way.
Bah
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
This got broken during rework. Fix it up, make sure we filter out
packages from to_add that are specified in to_add_local.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
It turns out that apk with the --root argument will still treat local
repositories relative to the host. This makes sense considering the
intended usecase, however as a result the recent change to use
apk.static for all apk commands inside the target root will break the
use of local packages.
To fix this, adjust how we populate /etc/apk/repositories, changing the
default to no longer include the local user repo. Instead we pass
--repository to apk.static with the host path to the repo.
The only time we want to add the user repo to /etc/apk/repositories
inside the chroot is when the user enters the chroot, so a special case
is added to helpers/frontend.py for the chroot command.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
We now have support for having mirrors per-aports repo, drop the
mirrors_postmarketos arg from chroot.init and instead have
repo_bootstrap call apk.update_repository_list() explicitly to exclude
the mirrors for the repository we're going to update.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
We used to recursively descend into the dependencies of the packages we
mark for install, this was needed so that we build all the packages we
want to.
However, the new package builder now does this for us. So simplify and
speed this up a bit by not doing it in apk.install().
Additionally, soft-remove support for passing in packages to delete by
having the package name start with a "!". This was useful at some point,
but we now handle this much better in pmaports.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Generalise pmb.helpers.other.cache with a more python decorator.
The Cache decorator takes a list of function arguments to use as cache
keys, keyword args can be used to restrict caching so that it is skipped
entirely unless the attribute has a specific value.
For example, pmb.helpers.pmaports.get() has the decorator:
@Cache("pkgname", subpackages=True)
This means the return value will be cached only when subpackages is
True, otherwise it will always miss.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Use the new config.mirrors section to handle repository URLs instead of
the old mirror_alpine / mirrors_postmarketos options. This let's us
add the systemd staging repo automatically when on the systemd staging
branch / systemd is enabled.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Move pmb/parse/arch.py over to core and refactor it as an Arch type,
similar to how Chroot was done. Fix all the uses (that I can find) of
arch in the codebase that need adjusting.
The new Arch type is an Enum, making it clear what architectures can be
represented and making it much easier to reason about. Since we support
~5 (kinda) different representations of an Architecture (Alpine, Kernel,
target triple, platform, and QEMU), we now formalise that the Alpine
format is what we represent internally, with methods to convert to any
of the others as-needed.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Cease merging pmbootstrap.cfg into args, implement a Context type to let
us pull globals out of thin air (as an intermediate workaround) and rip
args out of a lot of the codebase.
This is just a first pass, after this we can split all the state that
leaked over into Context into types with narrower scopes (like a
BuildContext(), etc).
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Testing by building postmarketos-initramfs (which installs >100 packages
but is very fast to build, so a worst-case scenario) this results in a
~15-20% speedup (which everything cached and doing multiple back to back
runs). From 32 seconds down to 25.
Doing a full install with --no-image, this takes us from 70 seconds on
my laptop down to 40s!
This also lets us drastically simplify pmb/helpers/apk.py!
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
We currently lazily initialize the chroot's on first use, plus a few
bonus calls to init. However, there are some instances where we actually
don't want the chroot to be initialised (mostly to break recursion
loops).
Simplify the codebase by removing all of this, and just calling
pmb.chroot.init() where it's needed.
In addition, print a warning if init() is called multiple times for one
chroot. This should help us catch these instances if they crop up again.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Defaulting to the native chroot isn't necessarily intuitive. Let's
require this be specified in full.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
With the new chroot type, we can now write fancy paths in the pythonic
way. Convert most of the codebase over, as well as adding various other
type hints.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
We use a custom verbose log level in pmbootstrap, unfortunately it isn't
possible to correctly type this due to some limitations in the logging
library [1], [2].
Given that our usecase is fairly simple, we can just wrap the module
with our own so we only have to tell mypy to ignore the error once
instead of at every callsite.
[1]: https://github.com/cryptax/droidlysis/issues/15
[2]: https://github.com/python/typing/discussions/980
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Introduce a new module: pmb.core to contain explicitly typed pmbootstrap
API. The first component being Suffix and SuffixType. This explicitly
defines what suffixes are possible, future changes should aim to further
constrain this API (e.g. by validating against available device
codenames or architectures for buildroot suffixes).
Additionally, migrate the entire codebase over to using pathlib.Path.
This is a relatively new part of the Python standard library that uses a
more object oriented model for path handling. It also uses strong type
hinting and has other features that make it much cleaner and easier to
work with than pure f-strings. The Chroot class overloads the "/"
operator the same way the Path object does, allowing one to write paths
relative to a given chroot as:
builddir = chroot / "home/pmos/build"
The Chroot class also has a string representation ("native", or
"rootfs_valve-jupiter"), and a .path property for directly accessing the
absolute path (as a Path object).
The general idea here is to encapsulate common patterns into type hinted
code, and gradually reduce the amount of assumptions made around the
codebase so that future changes are easier to implement.
As the chroot suffixes are now part of the Chroot class, we also
implement validation for them, this encodes the rules on suffix naming
and will cause a runtime exception if a suffix doesn't follow the rules.
This is needed to bring up the v24.06 repositories at
build.postmarketos.org. With the latest apk version, apk refuses to
operate if an URL from /etc/apk/repositories cannot be fetched.
Before the repositories are created for the first time, they do not
exist, so we will just set PMB_APK_FORCE_MISSING_REPOSITORIES=1 in bpo
to be not blocked here.
I've also spent significant time on alternative implementations, but
they have problems:
- Let bpo create an empty APKINDEX before building the first package,
but this was a larger code change, leading to lots of adjustments in
the tests, and ultimately it seems it didn't work properly (it seems
apk/abuild doesn't create a valid signed APKINDEX for one that has no
packages).
- Do not set the --mirror-pmOS argument for the "final" repository, only
the "wip" repository, until the "final" repository is available for
the first time. This works fine for x86_64, but not for foreign arch
repositories because then the cross compilers from the x86_64
repository are not available. I've also tried to make a different env
var that ensures we don't write the non-existing repository to
/etc/apk/repositories from within pmbootstrap if initializing a
foreign arch chroot, but then we would find a sane way to do this only
for the "final" repository and not for the "wip" repository which
leads to a lot more complexity than this patch.
So this is not the nicest solution (apk still tries to fetch the indexes
and gets a 404), but it is the simplest one and unblocks us from working
on v24.06. Also it doesn't add more complexity which is important in the
middle of the feature freeze we are currently in.
Related: bpo issue 137
Related: d76213e643
Related: https://postmarketos.org/blog/2024/05/19/pmOS-update-2024-05/#pmbootstrap-230-and-feature-freeze
