Commit graph

2651 commits

Author SHA1 Message Date
Caleb Connolly
fc010bc7c8
config: fixes + handle mirrors migration (MR 2252)
Get rid of config.mirror_alpine and mirrors_postmarketos and make sure
they get migrated over for existing users.

mirrors_postmarketos being a list was always a bit off, but now we have
per-aports mirrors which make a lot more sense for what we're trying to
do with systemd.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:40 +02:00
Caleb Connolly
aedab4df73
helpers: repo: cache urls() (MR 2252)
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:40 +02:00
Caleb Connolly
e087e7c665
helpers: hide channels.cfg and git remote log (MR 2252)
Add a new output=null option to run_core and use it for some git
commands.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:40 +02:00
Caleb Connolly
15ffc2f370
repo_bootstrap: simplify custom mirrors (MR 2252)
We now have support for having mirrors per-aports repo, drop the
mirrors_postmarketos arg from chroot.init and instead have
repo_bootstrap call apk.update_repository_list() explicitly to exclude
the mirrors for the repository we're going to update.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:40 +02:00
Caleb Connolly
807424ee2b
helpers: package: fix potential null access in check_arch() (MR 2252)
pmaports.get() can return None usually, but check_arch() has no handling
for it. Set must_exist=True so we properly error out in this case.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:40 +02:00
Caleb Connolly
84bced2110
qemu: fix config usage (MR 2252)
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:40 +02:00
Caleb Connolly
a7f8623ad8
sideload: fix arch type (MR 2252)
arch is optional

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:40 +02:00
Caleb Connolly
62f3aa8cc4
parse: apkindex: don't set arch if None in providers() (MR 2252)
apkindex_files() handles arch=None just fine.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:40 +02:00
Caleb Connolly
6db7e81a56
helpers: logging: add templates for colors (MR 2252)
Let's make pmbootstrap more colourful! Add some templates to enable
arbitrary colors in log messages.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:40 +02:00
Caleb Connolly
b860b3efbe
chroot: zap: fix zap_pkgs_online_mismatch() (MR 2252)
Since the chroot rework this logic seems a bit off, fix it up.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:40 +02:00
Caleb Connolly
457dfc8691
types: args.arch is optional (MR 2252)
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:40 +02:00
Caleb Connolly
6857882cf0
build: finish builder rework (MR 2252)
Rename build.package() to build.packages() and take a list of packages
to build, since every caller was inside a for loop this simplifies
usage and let's us give nicer log output by doing all the builds first,
so log messages don't get lost in the middle.

Behaviour is cleaned up so this shouuuuld work pretty well now. It
properly descends into dependencies and will build dependencies even if
the package given doesn't need building. Technically this was only done
before during install where the dependencies were recursed in
chroot.apk.install().

It probably makes the most sense to have a mode where it doesn't build
dependencies but warns the user about it, at least when invoked via
pmbootstrap build.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:39 +02:00
Caleb Connolly
e00d2a8e6d
install: don't call build in create_device_rootfs() (MR 2252)
The call to chroot.apk.install() will automatically build packages, so
we don't need to call it again beforehand.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:39 +02:00
Caleb Connolly
1299c2ef6d
chroot: apk: don't recurse depends (MR 2252)
We used to recursively descend into the dependencies of the packages we
mark for install, this was needed so that we build all the packages we
want to.

However, the new package builder now does this for us. So simplify and
speed this up a bit by not doing it in apk.install().

Additionally, soft-remove support for passing in packages to delete by
having the package name start with a "!". This was useful at some point,
but we now handle this much better in pmaports.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:39 +02:00
Caleb Connolly
185d8bcef5
meta: introduce a Cache decorator (MR 2252)
Generalise pmb.helpers.other.cache with a more python decorator.

The Cache decorator takes a list of function arguments to use as cache
keys, keyword args can be used to restrict caching so that it is skipped
entirely unless the attribute has a specific value.

For example, pmb.helpers.pmaports.get() has the decorator:
@Cache("pkgname", subpackages=True)

This means the return value will be cached only when subpackages is
True, otherwise it will always miss.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:39 +02:00
Caleb Connolly
79cf2e8910
TEMP: disable pigz (MR 2252)
Seems to be having trouble unpacking archives...

pigz: skipping: /var/cache/distfiles/postmarketos-mkinitfs-2.5.0.tar.gz ends with .gz
tar: This does not look like a tar archive

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:39 +02:00
Caleb Connolly
3ad4ba2818
WIP: build: rewrite package builder (MR 2252)
The package builder has long been a pain point since it recurses the
entire dependency tree.

Convert it to run in two stages, first it walks through the package
dependencies, descending into each one and processing them in a queue.
If a package is determined to need building then it gets pushed onto the
build_queue.

Then, it pops each package off the build queue (which is actually a
stack..) and builds it.

This avoids recursion entirely and should open the door to optimisations
in the future.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:39 +02:00
Caleb Connolly
338e0890ba
helpers: status: add missing import (MR 2252)
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:39 +02:00
Caleb Connolly
dbd0e97895
install: get_selected_providers: demote logging (MR 2252)
verbose is probably more suitable here.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:39 +02:00
Caleb Connolly
cc85ef70f5
install: losetup.mount() remove args (MR 2252)
not needed here.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:39 +02:00
Caleb Connolly
ef38c12e39
install: handle chroot unmount/mount properly (MR 2252)
We used to called pmb.chroot.shutdown() with "only_install_related=True"
here to unmount everything in the chroot so we could reliably "df" it to
figure out the rootfs image size.

Since ("pmb.chroot: only init once"), the chroot is no longer
automatically spun back up when we run mkinitfs, and the in-pmbootstrap
marker isn't recreated.

We only actually need to unmount the chroot anyway, so let's restrict it
to just that and remount things again afterwards.

This avoids weird side effects (like the native chroot being
shutdown?!), but this is something we should look into more in the
future since we're still doing a finnicky balancing act to manage chroot
state here.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:39 +02:00
Caleb Connolly
aba3584b02
FIXME: helpers: run: handle Arch in cmd (MR 2252)
We need a better solution here.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:39 +02:00
Caleb Connolly
53a2953a4a
chroot: mount: drop args from remove_mnt_pmbootstrap (MR 2252)
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:39 +02:00
Caleb Connolly
759fdd655b
chroot: mount: skip mounted targets (MR 2252)
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:39 +02:00
Caleb Connolly
b447c20545
build: fix arch formatting in check_build_for_arch (MR 2252)
Use f-strings so it's actually converted.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:39 +02:00
Caleb Connolly
656ff3f5bb
commands: move shutdown command to pmb/commands (MR 2252)
Another one bites the dust

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:39 +02:00
Caleb Connolly
a99ae02068
build: slightly less verbose logging (MR 2252)
Don't log about cached packages or on every call to package().

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:39 +02:00
Caleb Connolly
945d6dd063
types: PmbArgs: remove devicesdhbfvhubsud (MR 2252)
Holdover from cleaning up args.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:39 +02:00
Caleb Connolly
084309e756
parse: arguments: typing improvements (MR 2252)
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:39 +02:00
Caleb Connolly
efeb09bf5d
config: pmaports: log all_channels (MR 2252)
helps debugging

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:39 +02:00
Caleb Connolly
133091248f
repo: use new config.mirrors (MR 2252)
Use the new config.mirrors section to handle repository URLs instead of
the old mirror_alpine / mirrors_postmarketos options. This let's us
add the systemd staging repo automatically when on the systemd staging
branch / systemd is enabled.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:39 +02:00
Caleb Connolly
1a01738d50
config: sanity check via types (MR 2252)
Replace the "sanity_check" code with type checking built into the Config
__setattr__ operator.

This keeps all the Config related code in one place.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:39 +02:00
Caleb Connolly
7a8deb0f5e
config: clean up parsing and add mirrors (MR 2252)
Add a new config section "mirrors", to replace the mirrors_alpine and
mirrors_postmarketos options. This will allow for more flexibility since
we can then handle the systemd staging repo (and others like plasma
nightly) with relative ease.

The loading/saving is fixed and now properly avoids writing out default
values, this way if the defaults are changed the user won't be stuck
with old values in their pmbootstrap.cfg.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:39 +02:00
Caleb Connolly
8e18b16370
aportgen: more type hinting (MR 2252)
Thanks WhyNotHugo for the recommendations.
2024-06-23 12:38:39 +02:00
Caleb Connolly
866e5bcfab
core: add an Arch type (MR 2252)
Move pmb/parse/arch.py over to core and refactor it as an Arch type,
similar to how Chroot was done. Fix all the uses (that I can find) of
arch in the codebase that need adjusting.

The new Arch type is an Enum, making it clear what architectures can be
represented and making it much easier to reason about. Since we support
~5 (kinda) different representations of an Architecture (Alpine, Kernel,
target triple, platform, and QEMU), we now formalise that the Alpine
format is what we represent internally, with methods to convert to any
of the others as-needed.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:39 +02:00
Caleb Connolly
505165dc13
parse: apkindex: fix cache keys (MR 2252)
Using Paths as cache keys is not really ideal, plus it's the absolute
path on the host, weird.

For now, add a cache_key() function to get just the part of the path we
care about and use it as the key. This probably ought to be revisisted
later.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:39 +02:00
Caleb Connolly
450bc6873d
core: move Config type (MR 2252)
Keeping the Config class in types seemed kinda weird and was just done
as a workaround to some cyclical imports. But now things are more in
shape let's move it to core.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:39 +02:00
Caleb Connolly
b43724fee4
build: multiple binary repos (MR 2252)
Use a different binary repo depending on the source repository for the
package. This makes it possible to split out systemd packages into their
own repository.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:39 +02:00
Caleb Connolly
560cea46ea
treewide: migrate to pkgrepo API (MR 2252)
Make use of the new pmb.core.pkgrepo API to handle multiple aports and
extra-repos.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:39 +02:00
Caleb Connolly
8a61d67053
chroot: run: support running multiple commands with one call (MR 2252)
Building the command strings and entering the chroot is a
not-insubstantial amount of overhead. Implement support for running
multiple commands with a new pmb.chroot.rootm() function.

TODO: add alternative for chroot.user and run.root/user.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:39 +02:00
Caleb Connolly
3a74018f89
chroot: fix shutdown (MR 2252)
Something was going wrong here, confused

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:39 +02:00
Caleb Connolly
3473422639
chroot: fix args.user (MR 2252)
config.user and args.user are distinct, don't combine them.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:38 +02:00
Caleb Connolly
bc0b6e39e8
config: fix load/save/ask (MR 2252)
Fix all the config code so pmbootstrap init works again...

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:38 +02:00
Caleb Connolly
3a796d10f8
core: add pkgrepo helpers (MR 2252)
Introduce helpers for interacting with pkgrepos (e.g. pmaports, and
others). These allow pmbootstrap to handle multiple source repositories.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:38 +02:00
Caleb Connolly
0a73bc298b
parse: deviceinfo: default flash_method (MR 2252)
Not sure if we should allow flash_method to be unset, but eh, simplify
usage by adding the none fallback here.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:38 +02:00
Caleb Connolly
d8d55c3e52
aportgen: fix fork_alpine flag (MR 2252)
Broke during rework

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:38 +02:00
Oliver Smith
34ee1cff69
pmb.helpers.pmaports.get/find: deduplicate code (MR 2299) (MR 2252)
find() gets the pmaports directory of a package, get() parses the
APKBUILD inside that directory.

Instead of having the extra code path in get() for not searching
subpackages, which duplicated part of find(), add a subpackages
parameter to get().

Besides the deduplication done now, this is in preparation for code that
will extend get() to handle the extra-repos/systemd directory - without
this patch, the changes would need to be duplicated in both functions.
2024-06-23 12:38:38 +02:00
Caleb Connolly
f422b5c7ca
commands: port repo_bootstrap (MR 2252)
The repo_bootstrap command is totally standalone and has it's own state,
making it a good candidate for the pmb.commands submodule. Port it over
and move the require_bootstrap() helper function over to
pmb/helpers/pmaports.py

We also fix the call to pmb.build.package() which broke during rework.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:38 +02:00
Caleb Connolly
97bd8b96ec
parse: deviceinfo: make Deviceinfo a class (MR 2252)
Introduce a Deviceinfo class and use it rather than the dictionary. This
gives us sweet sweet autocomplete, and lays the foundation for having a
proper deviceinfo validator in the future.

Additionally, continue refactoring out args...

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:38 +02:00
Caleb Connolly
b51d31acab
flasher: variables: rename vars to fvars (MR 2252)
vars is a reserved keyword in python, use fvars instead.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:38 +02:00