Add a new environment variable that disables apk's caching feature for
space constrained environments such as the bpo image build jobs.
Currently we have a workaround in place in bpo: the apk cache is moved
to a tmpfs. But this is fragile and just disabling the apk cache is a
more elegant solution.
I've decided to make this an env var instead of a full pmbootstrap
option since this option is not relevant for normal users, only for CI
jobs in space constrained environments. Also we already have another
`PMB_APK_` env var.
Related: bpo issue 136
Part-of: https://gitlab.postmarketos.org/postmarketOS/pmbootstrap/-/merge_requests/2553
After extending CI to always run "pmbootstrap zap -a" it revealed
another bug: we did not ensure at this point that apk.static is
available. Rung apk_static.init() to ensure it gets downloaded and
extracted at this point if it is missing.
The tmp dir is owned by root, when it gets created earlier by apk
running as root to store the apk progress fifo. Use another directory to
work around this for now (getting close to the 3.0.0 release), we can
rework the apk progress fifo later on if we want.
Fixes: issue 2491
When running "apk add" we must always have --cache-dir set! We also
expect --no-interactive to be set.
Add some asserts so we don't regress here.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
In 0b4fb9119f (chroot: always run apk static v2 (MR 2423)) we adjusted
install_run_apk() to run apk static on the host and pass in the local
binary repo with "--repository". This function can call apk in two ways,
either with the progress bar handling or without, the second case was
never updated and still ran apk inside the chroot incorrectly and with
an incorrect --repository flag.
Let's finish the job by refactoring helpers/apk.py to support all our
usecases and pointing everything to it, removing the last few situations
where we call "pmb.chroot.root(["apk", ...]).
The apk_with_progress() function is replaced by a generic "run()"
function which takes a boolean to indicate if we should render apk
progress.
Additionally, a new cache_clean() function is added so that "pmbootstrap
zap --pkgs-online-mismatch" can FINALLY be refactored to not rely on a
chroot existing. This requires some hacks but nothing serious, see the
comments in the function for details.
The chroot.init() code is now simplified since handling the --root,
--arch, --cache-dir, and --repository flags is now all done by
apk._prepare_cmd() as and when appropriate.
Lastly, this fixes a (previously unnoticed) bug where apk.static was
actually using /var/cache/apk on your host machine for its cache... This
is definitely not good behaviour....
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
This function better belongs here, especially as it will be used outside
of the context of a chroot() soon.
Additionally, it's adjusted to take the rootfs path as its first
argument rather than a chroot, since it could operate on any rootfs.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Now that we don't need weird apk-tools hacks for systemd, we can
re-implement this optimisation and always run apk static rather than
running apk through the chroot.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Now that we have target-version = "py310" in [tool.ruff] in
pyproject.toml, ruff check complains about using typing.Optional and
typing.Union instead of newer syntax. Run the tool to fix it.
This revers b82c4eb167 ("chroot: always run apk static (MR 2252)")
since the current iteration of the apk-tools /usr merge patches
don't handle --root properly and will behave differently based on your
host systems directory structure (specifially for /usr merge).
This commit can be reverted once we fix apk-tools.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Move pmb/parse/arch.py over to core and refactor it as an Arch type,
similar to how Chroot was done. Fix all the uses (that I can find) of
arch in the codebase that need adjusting.
The new Arch type is an Enum, making it clear what architectures can be
represented and making it much easier to reason about. Since we support
~5 (kinda) different representations of an Architecture (Alpine, Kernel,
target triple, platform, and QEMU), we now formalise that the Alpine
format is what we represent internally, with methods to convert to any
of the others as-needed.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Building the command strings and entering the chroot is a
not-insubstantial amount of overhead. Implement support for running
multiple commands with a new pmb.chroot.rootm() function.
TODO: add alternative for chroot.user and run.root/user.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Cease merging pmbootstrap.cfg into args, implement a Context type to let
us pull globals out of thin air (as an intermediate workaround) and rip
args out of a lot of the codebase.
This is just a first pass, after this we can split all the state that
leaked over into Context into types with narrower scopes (like a
BuildContext(), etc).
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Testing by building postmarketos-initramfs (which installs >100 packages
but is very fast to build, so a worst-case scenario) this results in a
~15-20% speedup (which everything cached and doing multiple back to back
runs). From 32 seconds down to 25.
Doing a full install with --no-image, this takes us from 70 seconds on
my laptop down to 40s!
This also lets us drastically simplify pmb/helpers/apk.py!
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
With the new chroot type, we can now write fancy paths in the pythonic
way. Convert most of the codebase over, as well as adding various other
type hints.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Introduce a new module: pmb.core to contain explicitly typed pmbootstrap
API. The first component being Suffix and SuffixType. This explicitly
defines what suffixes are possible, future changes should aim to further
constrain this API (e.g. by validating against available device
codenames or architectures for buildroot suffixes).
Additionally, migrate the entire codebase over to using pathlib.Path.
This is a relatively new part of the Python standard library that uses a
more object oriented model for path handling. It also uses strong type
hinting and has other features that make it much cleaner and easier to
work with than pure f-strings. The Chroot class overloads the "/"
operator the same way the Path object does, allowing one to write paths
relative to a given chroot as:
builddir = chroot / "home/pmos/build"
The Chroot class also has a string representation ("native", or
"rootfs_valve-jupiter"), and a .path property for directly accessing the
absolute path (as a Path object).
The general idea here is to encapsulate common patterns into type hinted
code, and gradually reduce the amount of assumptions made around the
codebase so that future changes are easier to implement.
As the chroot suffixes are now part of the Chroot class, we also
implement validation for them, this encodes the rules on suffix naming
and will cause a runtime exception if a suffix doesn't follow the rules.
Support branches, so pmbootstrap won't fail if v20.05 is selected:
ERROR: You have an outdated version of the 'apk' package manager installed
(your version: 2.10.5-r1, expected at least: 2.12.1-r0).
Move the logic for this check to pmb.helpers.apk.check_outdated and
adjust the test.
This fixes the CI failure in test_crossdirect_rust, which uses the
stable channel. (My bad for not creating this patch earlier, while at
the same time explaining in the creating pmbootstrap release instructions,
that this minimum apk version should be adjusted.)
Hide progress bars if --details-to-stdout is used, which redirects all
output that would land in the pmbootstrap log to stdout. This caused the
progress bar output to get mixed with the apk output. A new progress bar
would get drawn whenever a new package was installed, without removing
the previous progress bar.
