We can't totally escape the need for some runtime state defined by args.
To make the migration easier, introduce a global "Context" class and
move some of the read-only global options there.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Testing by building postmarketos-initramfs (which installs >100 packages
but is very fast to build, so a worst-case scenario) this results in a
~15-20% speedup (which everything cached and doing multiple back to back
runs). From 32 seconds down to 25.
Doing a full install with --no-image, this takes us from 70 seconds on
my laptop down to 40s!
This also lets us drastically simplify pmb/helpers/apk.py!
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
We currently lazily initialize the chroot's on first use, plus a few
bonus calls to init. However, there are some instances where we actually
don't want the chroot to be initialised (mostly to break recursion
loops).
Simplify the codebase by removing all of this, and just calling
pmb.chroot.init() where it's needed.
In addition, print a warning if init() is called multiple times for one
chroot. This should help us catch these instances if they crop up again.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
This was used for testing the on device installer, but doesn't seem to
be super useful anymore. Let's deprecate it and remove it at some point.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Defaulting to the native chroot isn't necessarily intuitive. Let's
require this be specified in full.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
We currently do stuff in the chroot before merging /usr, this could lead
to weird side effects due to the recursive nature of chroot.init being
called every time we run a command. Let's reorder a bit to reduce the
risk of weirdness here.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
We don't need to re-init the chroot this often, cache on first init and
skip all subsequent ones. Even though we take a shortcut if the chroot
already exists we still do a bunch of additional checks which we only
really need to do once.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
This is multithreaded, and by consequence much much faster than default
gzip. Since we switched to running gzip from the native chroot we also
changed from running it in the fastest mode, now we're running it with
-9 it can be pretty slow on any kind of mobile hardware.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
With the new chroot type, we can now write fancy paths in the pythonic
way. Convert most of the codebase over, as well as adding various other
type hints.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Avoid passing in a boolean to decide if this is the first time the
function has been called and instead just initialise the globals at the
top of the file.
We can figure out the state management if/when we want
to handle doing multiple installs in a single invocation of pmb.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Debug is the default loglevel, demote the "already visited" message to
verbose, as it isn't generally useful.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
We use a custom verbose log level in pmbootstrap, unfortunately it isn't
possible to correctly type this due to some limitations in the logging
library [1], [2].
Given that our usecase is fairly simple, we can just wrap the module
with our own so we only have to tell mypy to ignore the error once
instead of at every callsite.
[1]: https://github.com/cryptax/droidlysis/issues/15
[2]: https://github.com/python/typing/discussions/980
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
According to the type hints for os.cpu_count(), it might return None...
To be safe, add a warning and a fallback in this case (though it seems
incredibly unlikely this would ever be hit)
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
The init_cache() function just assigned some default constants, simplify
this by just declaring it that way to begin with, as well as adding type
hints.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Introduce a new module: pmb.core to contain explicitly typed pmbootstrap
API. The first component being Suffix and SuffixType. This explicitly
defines what suffixes are possible, future changes should aim to further
constrain this API (e.g. by validating against available device
codenames or architectures for buildroot suffixes).
Additionally, migrate the entire codebase over to using pathlib.Path.
This is a relatively new part of the Python standard library that uses a
more object oriented model for path handling. It also uses strong type
hinting and has other features that make it much cleaner and easier to
work with than pure f-strings. The Chroot class overloads the "/"
operator the same way the Path object does, allowing one to write paths
relative to a given chroot as:
builddir = chroot / "home/pmos/build"
The Chroot class also has a string representation ("native", or
"rootfs_valve-jupiter"), and a .path property for directly accessing the
absolute path (as a Path object).
The general idea here is to encapsulate common patterns into type hinted
code, and gradually reduce the amount of assumptions made around the
codebase so that future changes are easier to implement.
As the chroot suffixes are now part of the Chroot class, we also
implement validation for them, this encodes the rules on suffix naming
and will cause a runtime exception if a suffix doesn't follow the rules.
It's nice to know how much space gets free'd when zapping, but I often
find myself with a lot of chroot's and other junk, and on my laptop
running "du" across all of this takes quite a few seconds. As this is
purely cosmetic, it doesn't justify taking such a long time.
Remove the size calculation code to substantially speed up zap.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
HTTP auth is heavily discouraged, and becoming annoying to set up and
use. Let folks reconfigure pmaports origin remote using the SSH URL.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Currently there is very little info from test failures when the test
fails in some call to pmbootstrap. By saving the log file to CI
artifacts on failure, we can make it really easy to find/obtain
important stuff to help with debugging.
This is needed to bring up the v24.06 repositories at
build.postmarketos.org. With the latest apk version, apk refuses to
operate if an URL from /etc/apk/repositories cannot be fetched.
Before the repositories are created for the first time, they do not
exist, so we will just set PMB_APK_FORCE_MISSING_REPOSITORIES=1 in bpo
to be not blocked here.
I've also spent significant time on alternative implementations, but
they have problems:
- Let bpo create an empty APKINDEX before building the first package,
but this was a larger code change, leading to lots of adjustments in
the tests, and ultimately it seems it didn't work properly (it seems
apk/abuild doesn't create a valid signed APKINDEX for one that has no
packages).
- Do not set the --mirror-pmOS argument for the "final" repository, only
the "wip" repository, until the "final" repository is available for
the first time. This works fine for x86_64, but not for foreign arch
repositories because then the cross compilers from the x86_64
repository are not available. I've also tried to make a different env
var that ensures we don't write the non-existing repository to
/etc/apk/repositories from within pmbootstrap if initializing a
foreign arch chroot, but then we would find a sane way to do this only
for the "final" repository and not for the "wip" repository which
leads to a lot more complexity than this patch.
So this is not the nicest solution (apk still tries to fetch the indexes
and gets a 404), but it is the simplest one and unblocks us from working
on v24.06. Also it doesn't add more complexity which is important in the
middle of the feature freeze we are currently in.
Related: bpo issue 137
Related: d76213e643
Related: https://postmarketos.org/blog/2024/05/19/pmOS-update-2024-05/#pmbootstrap-230-and-feature-freeze
This is needed for sphinx autoprogram since that expects an
argparse.ArgumentParser, and arguments() returns some argparse
"Namespace" obj. Useful for sphinx/autoprogram and maybe other things
later that want to get at pmb's full args.
linux-lts and linux-edge in alpine use "utf-8", that might change, but
add the options regardless, so we can be safe (people can always edit them!)
Fixes https://gitlab.com/postmarketOS/pmaports/-/issues/2782
[ci:skip-build]: already built successfully in CI
We don't want executables, suid executables, or devices. We neither
want symbolic links in fat partitions. These have been taken from
systemd
Ref: pmaports#2782
This test is *only* failing in gitlab CI and I haven't been able to
figure out why (see [1]). Everything else (e.g. bpo, building packages
locally) seems to work fine. Since this is blocking pmbootstrap MRs, I'm
going to disable the test until we can figure out wtf is going on with
running it in gitlab CI.
1. https://gitlab.com/postmarketOS/pmbootstrap/-/issues/2346
[ci:skip-build]: already built successfully in CI
