We plan to ship systemd with split /usr until the /usr merge is complete
in Alpine. Let's not drop all our code yet but just forcefully disable
it.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Ensure we get our fork of apk straight away without having to run `apk
upgrade -a` immediately afterwards.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
* Cache the function so we only do the check once per-run
* check all chroot's at once so the message will be printed near the
start of output (rather than halfway through after building packages
or something).
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
This option will make pmbootstrap automatically zap chroots that are
initialized for the wrong channel, making it much faster to switch
between edge and systemd.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
We now have support for having mirrors per-aports repo, drop the
mirrors_postmarketos arg from chroot.init and instead have
repo_bootstrap call apk.update_repository_list() explicitly to exclude
the mirrors for the repository we're going to update.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Generalise pmb.helpers.other.cache with a more python decorator.
The Cache decorator takes a list of function arguments to use as cache
keys, keyword args can be used to restrict caching so that it is skipped
entirely unless the attribute has a specific value.
For example, pmb.helpers.pmaports.get() has the decorator:
@Cache("pkgname", subpackages=True)
This means the return value will be cached only when subpackages is
True, otherwise it will always miss.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
We used to called pmb.chroot.shutdown() with "only_install_related=True"
here to unmount everything in the chroot so we could reliably "df" it to
figure out the rootfs image size.
Since ("pmb.chroot: only init once"), the chroot is no longer
automatically spun back up when we run mkinitfs, and the in-pmbootstrap
marker isn't recreated.
We only actually need to unmount the chroot anyway, so let's restrict it
to just that and remount things again afterwards.
This avoids weird side effects (like the native chroot being
shutdown?!), but this is something we should look into more in the
future since we're still doing a finnicky balancing act to manage chroot
state here.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Move pmb/parse/arch.py over to core and refactor it as an Arch type,
similar to how Chroot was done. Fix all the uses (that I can find) of
arch in the codebase that need adjusting.
The new Arch type is an Enum, making it clear what architectures can be
represented and making it much easier to reason about. Since we support
~5 (kinda) different representations of an Architecture (Alpine, Kernel,
target triple, platform, and QEMU), we now formalise that the Alpine
format is what we represent internally, with methods to convert to any
of the others as-needed.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Cease merging pmbootstrap.cfg into args, implement a Context type to let
us pull globals out of thin air (as an intermediate workaround) and rip
args out of a lot of the codebase.
This is just a first pass, after this we can split all the state that
leaked over into Context into types with narrower scopes (like a
BuildContext(), etc).
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Testing by building postmarketos-initramfs (which installs >100 packages
but is very fast to build, so a worst-case scenario) this results in a
~15-20% speedup (which everything cached and doing multiple back to back
runs). From 32 seconds down to 25.
Doing a full install with --no-image, this takes us from 70 seconds on
my laptop down to 40s!
This also lets us drastically simplify pmb/helpers/apk.py!
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
We currently lazily initialize the chroot's on first use, plus a few
bonus calls to init. However, there are some instances where we actually
don't want the chroot to be initialised (mostly to break recursion
loops).
Simplify the codebase by removing all of this, and just calling
pmb.chroot.init() where it's needed.
In addition, print a warning if init() is called multiple times for one
chroot. This should help us catch these instances if they crop up again.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
We currently do stuff in the chroot before merging /usr, this could lead
to weird side effects due to the recursive nature of chroot.init being
called every time we run a command. Let's reorder a bit to reduce the
risk of weirdness here.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
We don't need to re-init the chroot this often, cache on first init and
skip all subsequent ones. Even though we take a shortcut if the chroot
already exists we still do a bunch of additional checks which we only
really need to do once.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
With the new chroot type, we can now write fancy paths in the pythonic
way. Convert most of the codebase over, as well as adding various other
type hints.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
We use a custom verbose log level in pmbootstrap, unfortunately it isn't
possible to correctly type this due to some limitations in the logging
library [1], [2].
Given that our usecase is fairly simple, we can just wrap the module
with our own so we only have to tell mypy to ignore the error once
instead of at every callsite.
[1]: https://github.com/cryptax/droidlysis/issues/15
[2]: https://github.com/python/typing/discussions/980
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Introduce a new module: pmb.core to contain explicitly typed pmbootstrap
API. The first component being Suffix and SuffixType. This explicitly
defines what suffixes are possible, future changes should aim to further
constrain this API (e.g. by validating against available device
codenames or architectures for buildroot suffixes).
Additionally, migrate the entire codebase over to using pathlib.Path.
This is a relatively new part of the Python standard library that uses a
more object oriented model for path handling. It also uses strong type
hinting and has other features that make it much cleaner and easier to
work with than pure f-strings. The Chroot class overloads the "/"
operator the same way the Path object does, allowing one to write paths
relative to a given chroot as:
builddir = chroot / "home/pmos/build"
The Chroot class also has a string representation ("native", or
"rootfs_valve-jupiter"), and a .path property for directly accessing the
absolute path (as a Path object).
The general idea here is to encapsulate common patterns into type hinted
code, and gradually reduce the amount of assumptions made around the
codebase so that future changes are easier to implement.
As the chroot suffixes are now part of the Chroot class, we also
implement validation for them, this encodes the rules on suffix naming
and will cause a runtime exception if a suffix doesn't follow the rules.
This is needed to bring up the v24.06 repositories at
build.postmarketos.org. With the latest apk version, apk refuses to
operate if an URL from /etc/apk/repositories cannot be fetched.
Before the repositories are created for the first time, they do not
exist, so we will just set PMB_APK_FORCE_MISSING_REPOSITORIES=1 in bpo
to be not blocked here.
I've also spent significant time on alternative implementations, but
they have problems:
- Let bpo create an empty APKINDEX before building the first package,
but this was a larger code change, leading to lots of adjustments in
the tests, and ultimately it seems it didn't work properly (it seems
apk/abuild doesn't create a valid signed APKINDEX for one that has no
packages).
- Do not set the --mirror-pmOS argument for the "final" repository, only
the "wip" repository, until the "final" repository is available for
the first time. This works fine for x86_64, but not for foreign arch
repositories because then the cross compilers from the x86_64
repository are not available. I've also tried to make a different env
var that ensures we don't write the non-existing repository to
/etc/apk/repositories from within pmbootstrap if initializing a
foreign arch chroot, but then we would find a sane way to do this only
for the "final" repository and not for the "wip" repository which
leads to a lot more complexity than this patch.
So this is not the nicest solution (apk still tries to fetch the indexes
and gets a 404), but it is the simplest one and unblocks us from working
on v24.06. Also it doesn't add more complexity which is important in the
middle of the feature freeze we are currently in.
Related: bpo issue 137
Related: d76213e643
Related: https://postmarketos.org/blog/2024/05/19/pmOS-update-2024-05/#pmbootstrap-230-and-feature-freeze
Prepare to remove the outdated chroot check from "pmbootstrap status".
Display it when the user enters a stale chroot instead. This way the
user is more likely to see it, and we can make "pmbootstrap status" more
minimal (by removing all checks, in future patches).
Related: issue 1903
Upgrade packages in the chroot, in case alpine-base, apk etc. have been
built from source with pmbootstrap. We build it from source for systemd
currently, and sometimes it is useful to do that to debug apk.
Add this argument, so we can disable the pmOS repository during
"pmbootstrap repo_bootstrap" for the chroot we build in.
If building natively, it will be disabled in the "native" chroot.
If building for a foreign arch, it will be disabled in the
"buildroot_$arch" chroot, but still be enabled in the native chroot so
we have the cross compilers available.
The name of the argument is a bit long, but it is consistent with the
argument of the same name in pmb.helpers.repo.urls() (to which it gets
passed).
Replace "args.arch_native" with the direct function call in order to
avoid passing "args" to all functions. This is a step to get rid of this
args-passed-to-all-functions pattern in pmbootstrap.
Move code to install Alpine and postmarketOS keys into its own function,
so it can be called outside of pmb.chroot.init too (next patch).
Describe why this is needed in the first place and, while at it, tweak
the function to only copy the key if it does not exist in the target
directory.
Remove /in-pmbootstrap inside chroots in "pmbootstrap shutdown" instead
of having it at a specific part of "pmbootstrap install".
Reasoning:
* With current approach, it didn't get removed in the on-device
installer chroot.
* This is less error prone than calling it multiple times in
"pmbootstrap install"
Touch the file /in-pmbootstrap in chroots so that we can avoid
performing automated actions that should only happen on a real device
(like flashing the kernel).
Whenever initializing new chroots, save the date in $WORK/workdir.cfg.
Add pmb.config.workdir.chroots_outdated() to check if it's time to zap
the chroots or not (since we don't update them automatically). Mark them
as outdated after two days.
This will be the first check in "pmbootstrap status" (future patches).
Related: #1829
While at it, also remove unnecessary "#!/usr/bin/env python3" in files
that only get imported, and adjust other empty/comment lines in the
beginnings of the files for consistency.
This makes files easier to read, and makes the pmbootstrap codebase more
consistent with the build.postmarketos.org codebase.
When initializing the building chroots, some symlinks are created to
/mnt/pmbootstrap* dirs. With upcoming rust-related symlinks, they will
point to subdirs in /mnt/pmbootstrap-rust, such as
/mnt/pmbootstrap-rust/registry/cache. Create this directory structure if
it does not exist.
Overview:
In order to execute foreign arch binaries on the host system, we are
using the Linux kernel's binfmt_misc feature in combination with
static builds of QEMU. Before this patch, the statically compiled
QEMU binaries were taken from Debian (mostly because I did not realize
that Alpine ships them as well). Now we can use the ones from the aport.
Benefits:
This allows us to easily update and patch the QEMU executables, we
don't need to be in sync with Debian's versions anymore.
Alpine's package is more modular, so we can save some download,
install, zap time, as well as disk space: setting up an armhf chroot
with pmbootstrap took ~102 MB before, now it's ~18 MB.
Detailed changes:
* Remove `cross/qemu-user-static-repack` aport
* Add `data/qemu-user-binfmt.txt` with the binfmt_misc flags for ELF
binaries of various arches (extracted from Debian's packaging)
* When parsing that file, don't write verbose messages to
`pmbootstrap log` anymore, only to the verbose log (can be enabled
with `pmbootstrap -v`)
* Rename `pmb.parse.arch.alpine_to_debian()` to ...`alpine_to_qemu()`
* Rename `arch_debian` to `arch_qemu`
* change "pmbootstrap kconfig_check" to "pmbootstrap kconfig check"
* change "pmbootstrap menuconfig" to "pmbootstrap kconfig edit [-x|-g]"
(with legacy alias, because the first syntax was referenced to a lot)
* enable X11 interfaces: -x: xconfig, -g: gconfig
* new function to copy the xauthority file:
pmb.chroot.other.copy_xauthority()
* remove menufconfig() function from the kernel template and all kernel
aports ([skip ci] because it would rebuild all kernels and run out of
time). Alpine has dropped this as well, and it wouldn't work with the
new code anyway.
Device nodes in the chroots get created in a tmpfs, so they can be
created even if the filesystem where the chroot resides does not
support device nodes (#1317). In "pmbootstrap shutdown" we umount the
`dev` folder, which means all device nodes that were created inside
this folder are gone. This commit changes the code to actually recreate
the device nodes when using the chroot again.
Details:
* move `pmb.chroot.init.create_device_nodes` to
`pmb.chroot.mount.crete_device_nodes`
* don't call it in `pmb.chroot.init()` anymore, but in
`pmb.chroot.mount_dev_tmpfs()`
* Create the `null` device as well (`apk --initdb` also creates it on
`init`, but we don't call it after `shutdown`)
