Cease merging pmbootstrap.cfg into args, implement a Context type to let
us pull globals out of thin air (as an intermediate workaround) and rip
args out of a lot of the codebase.
This is just a first pass, after this we can split all the state that
leaked over into Context into types with narrower scopes (like a
BuildContext(), etc).
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Testing by building postmarketos-initramfs (which installs >100 packages
but is very fast to build, so a worst-case scenario) this results in a
~15-20% speedup (which everything cached and doing multiple back to back
runs). From 32 seconds down to 25.
Doing a full install with --no-image, this takes us from 70 seconds on
my laptop down to 40s!
This also lets us drastically simplify pmb/helpers/apk.py!
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
We currently lazily initialize the chroot's on first use, plus a few
bonus calls to init. However, there are some instances where we actually
don't want the chroot to be initialised (mostly to break recursion
loops).
Simplify the codebase by removing all of this, and just calling
pmb.chroot.init() where it's needed.
In addition, print a warning if init() is called multiple times for one
chroot. This should help us catch these instances if they crop up again.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Defaulting to the native chroot isn't necessarily intuitive. Let's
require this be specified in full.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
We currently do stuff in the chroot before merging /usr, this could lead
to weird side effects due to the recursive nature of chroot.init being
called every time we run a command. Let's reorder a bit to reduce the
risk of weirdness here.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
We don't need to re-init the chroot this often, cache on first init and
skip all subsequent ones. Even though we take a shortcut if the chroot
already exists we still do a bunch of additional checks which we only
really need to do once.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
This is multithreaded, and by consequence much much faster than default
gzip. Since we switched to running gzip from the native chroot we also
changed from running it in the fastest mode, now we're running it with
-9 it can be pretty slow on any kind of mobile hardware.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
With the new chroot type, we can now write fancy paths in the pythonic
way. Convert most of the codebase over, as well as adding various other
type hints.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
We use a custom verbose log level in pmbootstrap, unfortunately it isn't
possible to correctly type this due to some limitations in the logging
library [1], [2].
Given that our usecase is fairly simple, we can just wrap the module
with our own so we only have to tell mypy to ignore the error once
instead of at every callsite.
[1]: https://github.com/cryptax/droidlysis/issues/15
[2]: https://github.com/python/typing/discussions/980
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Introduce a new module: pmb.core to contain explicitly typed pmbootstrap
API. The first component being Suffix and SuffixType. This explicitly
defines what suffixes are possible, future changes should aim to further
constrain this API (e.g. by validating against available device
codenames or architectures for buildroot suffixes).
Additionally, migrate the entire codebase over to using pathlib.Path.
This is a relatively new part of the Python standard library that uses a
more object oriented model for path handling. It also uses strong type
hinting and has other features that make it much cleaner and easier to
work with than pure f-strings. The Chroot class overloads the "/"
operator the same way the Path object does, allowing one to write paths
relative to a given chroot as:
builddir = chroot / "home/pmos/build"
The Chroot class also has a string representation ("native", or
"rootfs_valve-jupiter"), and a .path property for directly accessing the
absolute path (as a Path object).
The general idea here is to encapsulate common patterns into type hinted
code, and gradually reduce the amount of assumptions made around the
codebase so that future changes are easier to implement.
As the chroot suffixes are now part of the Chroot class, we also
implement validation for them, this encodes the rules on suffix naming
and will cause a runtime exception if a suffix doesn't follow the rules.
It's nice to know how much space gets free'd when zapping, but I often
find myself with a lot of chroot's and other junk, and on my laptop
running "du" across all of this takes quite a few seconds. As this is
purely cosmetic, it doesn't justify taking such a long time.
Remove the size calculation code to substantially speed up zap.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
This is needed to bring up the v24.06 repositories at
build.postmarketos.org. With the latest apk version, apk refuses to
operate if an URL from /etc/apk/repositories cannot be fetched.
Before the repositories are created for the first time, they do not
exist, so we will just set PMB_APK_FORCE_MISSING_REPOSITORIES=1 in bpo
to be not blocked here.
I've also spent significant time on alternative implementations, but
they have problems:
- Let bpo create an empty APKINDEX before building the first package,
but this was a larger code change, leading to lots of adjustments in
the tests, and ultimately it seems it didn't work properly (it seems
apk/abuild doesn't create a valid signed APKINDEX for one that has no
packages).
- Do not set the --mirror-pmOS argument for the "final" repository, only
the "wip" repository, until the "final" repository is available for
the first time. This works fine for x86_64, but not for foreign arch
repositories because then the cross compilers from the x86_64
repository are not available. I've also tried to make a different env
var that ensures we don't write the non-existing repository to
/etc/apk/repositories from within pmbootstrap if initializing a
foreign arch chroot, but then we would find a sane way to do this only
for the "final" repository and not for the "wip" repository which
leads to a lot more complexity than this patch.
So this is not the nicest solution (apk still tries to fetch the indexes
and gets a 404), but it is the simplest one and unblocks us from working
on v24.06. Also it doesn't add more complexity which is important in the
middle of the feature freeze we are currently in.
Related: bpo issue 137
Related: d76213e643
Related: https://postmarketos.org/blog/2024/05/19/pmOS-update-2024-05/#pmbootstrap-230-and-feature-freeze
Prepare to remove the outdated chroot check from "pmbootstrap status".
Display it when the user enters a stale chroot instead. This way the
user is more likely to see it, and we can make "pmbootstrap status" more
minimal (by removing all checks, in future patches).
Related: issue 1903
Upgrade packages in the chroot, in case alpine-base, apk etc. have been
built from source with pmbootstrap. We build it from source for systemd
currently, and sometimes it is useful to do that to debug apk.
Add this argument, so we can disable the pmOS repository during
"pmbootstrap repo_bootstrap" for the chroot we build in.
If building natively, it will be disabled in the "native" chroot.
If building for a foreign arch, it will be disabled in the
"buildroot_$arch" chroot, but still be enabled in the native chroot so
we have the cross compilers available.
The name of the argument is a bit long, but it is consistent with the
argument of the same name in pmb.helpers.repo.urls() (to which it gets
passed).
Fix "pmbootstrap chroot" and others not passing the proxy environment
variables correctly. Thanks to notfound405 for pointing this out!
Instead of only preserving proxy environment variables in
pmb.helpers.run_core, which should never be called directly, do it in
the calling functions:
* pmb.helpers.run.user
* pmb.helpers.run.root
* pmb.chroot.root
* pmb.chroot.user
This fixes that the environment variables were only really passed by
pmb.helpers.run.user, because the other functions would result in
something like:
HTTP_PROXY=mytestproxy sudo env -i /usr/bin/sh -c '…'
This is needed to either elevate to root, or to elevate to root first
and then enter the chroot as root or user. Due to the "env -i", the
environment intentionally gets cleaned, but unintentionally also removes
the proxy environment variables that were explicitly set.
By adjusting the functions, they now run a variant of:
sudo env -i /usr/bin/sh -c 'HTTP_PROXY=mytestproxy …'
The escaping is simplified in this example, run "pmbootstrap -v" to see
the not very readable, but proper escaping with shutil.quote().
Remove the previous test for preserving the environment variables in
pmb.helpers.run_core (as it should never be called directly), and test
instead the new behavior.
Fixes: issue 2299
Fixes: 13c4ac42 ("pmb.helpers.run_core: fix proxy env var logic")
This fixes an issue when pmb is run on an Alpine/pmOS host and apk.static sees
that /etc/apk/interactive is set on the host. It's really annoying to have the
build pause so apk can prompt when building chroots, so this uses a flag to apk
to disable interactive mode. I can't think of any situations where we would
actually want to prompt users when building chroots, by then all user
configuration should have been completed via pmb init/config.
Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Co-developed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20231110185320.8358-1-clayton@craftyguy.net%3E
Cache the compiler output of rust code with sccache, like we use ccache
for c code.
I've considered using sccache to completely replace ccache since it can
cache output of C/C++ code too. But let's not do it for now since ccache
doesn't need to run a daemon in the background that needs to be stopped
when shutting down / zapping. Also it would need more refactoring.
Reviewed-by: Luca Weiss <luca@z3ntu.xyz>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230806184729.4891-5-ollieparanoid@postmarketos.org%3E
With this code path, pmbootstrap would start a distccd + sshd in the
native chroot, and configure it so it runs the cross compiler. The
foreign arch chroots would then call this cross compiler from localhost
by calling the distcc client instead of gcc.
This code has been obsoleted by the much simpler crossdirect in 2019.
Let's finally remove it.
Fixes: issue 2179
Reviewed-by: Luca Weiss <luca@z3ntu.xyz>
Reviewed-by: Clayton Craft <clayton@craftyguy.net>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230613161437.570196-4-ollieparanoid@postmarketos.org%3E
Set the env var, so python programs running inside pmbootstrap chroot
don't buffer their output and only end up printing everything when they
are done. I've seen this with meson. This is bad for usability because
we don't see output, but also a problem because pmbootstrap kills
commands if they don't print any output for some time (default: 15 min).
Check if the pkgnames are sane in install_run_apk, right before running
apk. This makes sure that we really run it on all arguments that are
supposed to be packages / files and not options to apk.
Previously pmbootstrap would only show the packages that are about to be
installed. In case all packages were already installed, this would lead
to weird empty install messages:
(rootfs_asus-me176c) install
Show all packages that we want to install, even if they are already
installed in the given chroot.
(rootfs_asus-me176c) install device-asus-me176c
Previously to this patch, pmbootstrap would pass a full dependency tree
to "apk add". It would use a virtual package to ensure only the right
packages get added to /etc/apk/world. For example:
apk add -u --virtual .pmbootstrap postmarketos-base device-asus-me176c \
postmarketos-ui-sxmo-de-sway device-asus-me176c-nonfree-firmware w3m \
sfeed clickclack firefox-esr font-noto font-noto-emoji gnome-icon-theme \
imv megapixels mobile-config-firefox ttyescape postmarketos-base-nofde \
eudev openssh postmarketos-mkinitfs postmarketos-mvcfg postmarketos-keys \
...
Instead of doing that, only pass the packages we want to install and let
apk figure out the dependencies. Most of the time we can even avoid
using the virtual package now.
== Remaining edge case: locally built packages
apk will only upgrade a package with the same pkgver + pkgrel but a
different build date if the full path to an apk file gets passed as
argument. So if the user built a package locally that will be installed,
or one of its dependencies then we still need to use a virtual package
and possibly pass a dependency to apk. Replace
replace_aports_packages_with_path() with packages_get_locally_built_apks()
to get a list of such packages and adjust install() and
install_run_apk() to use it.
Make the code easier to read by moving split_to_add_del() to a separate
function and do some related refactoring. A future patch will use it
twice in install().
Move "arch = ..." to the top of the function while at it, since it's
needed later in the function in 2 places and is not needed for figuring
out packages_with_depends, to_add and to_del.
Remove "# Add depends to packages" because it's obvious from the
packages_with_depends variable name, and getting to_add/to_del is a
different action that stood under the same comment.
Split out the part that builds the apk commands and runs them out from
install() to a new function install_run_apk(). This makes install()
easier to read.
Since the previous commit that adds install_build(), all packages either
end up in to_add or to_del. Move the check for empty packages to the top
of the function, and directly check the packages variable.
I think it's worth keeping this check because it's shorter to add this
check once here than having it a few times in other place where we may
have or may not have something to install. And so we can avoid printing
an empty "install" message with no packages.
install_is_necessary used to do the following things:
1. Error out if there's no binary package but pmb was invoked as
"pmbootstrap install" and build_pkgs_on_install is disabled.
2. Build the package if necessary.
3. Return if a package "needs to be installed" (Boolean or Float).
The only caller of the function is pmb.chroot.apk.install. It would not
add the package to the long "apk add" command if according to 3. it does
not need to be installed.
When I implemented this a few years ago, I probably thought it would be
useful to not unnecessarily pass packages to apk. But this actually
makes it more complicated and doesn't have a benefit, apk is perfectly
capable of recognizing which packages it had already installed.
Replace the function with a much simpler pmb.chroot.apk.install_build,
which only does 1. and 2. Change the order of the package, arch
arguments to match called functions pmb.parse.apkindex.package and
pmb.build.package.
