The init_cache() function just assigned some default constants, simplify
this by just declaring it that way to begin with, as well as adding type
hints.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Introduce a new module: pmb.core to contain explicitly typed pmbootstrap
API. The first component being Suffix and SuffixType. This explicitly
defines what suffixes are possible, future changes should aim to further
constrain this API (e.g. by validating against available device
codenames or architectures for buildroot suffixes).
Additionally, migrate the entire codebase over to using pathlib.Path.
This is a relatively new part of the Python standard library that uses a
more object oriented model for path handling. It also uses strong type
hinting and has other features that make it much cleaner and easier to
work with than pure f-strings. The Chroot class overloads the "/"
operator the same way the Path object does, allowing one to write paths
relative to a given chroot as:
builddir = chroot / "home/pmos/build"
The Chroot class also has a string representation ("native", or
"rootfs_valve-jupiter"), and a .path property for directly accessing the
absolute path (as a Path object).
The general idea here is to encapsulate common patterns into type hinted
code, and gradually reduce the amount of assumptions made around the
codebase so that future changes are easier to implement.
As the chroot suffixes are now part of the Chroot class, we also
implement validation for them, this encodes the rules on suffix naming
and will cause a runtime exception if a suffix doesn't follow the rules.
It's nice to know how much space gets free'd when zapping, but I often
find myself with a lot of chroot's and other junk, and on my laptop
running "du" across all of this takes quite a few seconds. As this is
purely cosmetic, it doesn't justify taking such a long time.
Remove the size calculation code to substantially speed up zap.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
HTTP auth is heavily discouraged, and becoming annoying to set up and
use. Let folks reconfigure pmaports origin remote using the SSH URL.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
This is needed to bring up the v24.06 repositories at
build.postmarketos.org. With the latest apk version, apk refuses to
operate if an URL from /etc/apk/repositories cannot be fetched.
Before the repositories are created for the first time, they do not
exist, so we will just set PMB_APK_FORCE_MISSING_REPOSITORIES=1 in bpo
to be not blocked here.
I've also spent significant time on alternative implementations, but
they have problems:
- Let bpo create an empty APKINDEX before building the first package,
but this was a larger code change, leading to lots of adjustments in
the tests, and ultimately it seems it didn't work properly (it seems
apk/abuild doesn't create a valid signed APKINDEX for one that has no
packages).
- Do not set the --mirror-pmOS argument for the "final" repository, only
the "wip" repository, until the "final" repository is available for
the first time. This works fine for x86_64, but not for foreign arch
repositories because then the cross compilers from the x86_64
repository are not available. I've also tried to make a different env
var that ensures we don't write the non-existing repository to
/etc/apk/repositories from within pmbootstrap if initializing a
foreign arch chroot, but then we would find a sane way to do this only
for the "final" repository and not for the "wip" repository which
leads to a lot more complexity than this patch.
So this is not the nicest solution (apk still tries to fetch the indexes
and gets a 404), but it is the simplest one and unblocks us from working
on v24.06. Also it doesn't add more complexity which is important in the
middle of the feature freeze we are currently in.
Related: bpo issue 137
Related: d76213e643
Related: https://postmarketos.org/blog/2024/05/19/pmOS-update-2024-05/#pmbootstrap-230-and-feature-freeze
This is needed for sphinx autoprogram since that expects an
argparse.ArgumentParser, and arguments() returns some argparse
"Namespace" obj. Useful for sphinx/autoprogram and maybe other things
later that want to get at pmb's full args.
linux-lts and linux-edge in alpine use "utf-8", that might change, but
add the options regardless, so we can be safe (people can always edit them!)
Fixes https://gitlab.com/postmarketOS/pmaports/-/issues/2782
[ci:skip-build]: already built successfully in CI
We don't want executables, suid executables, or devices. We neither
want symbolic links in fat partitions. These have been taken from
systemd
Ref: pmaports#2782
* Replace aports -> pmaports
* Make the "binary repo has newer version" message much shorter, and
mention "pmbootstrap pull". If users didn't mess with their pmaports
repository (checkout a custom branch, make commits), this command
will update to the latest commit and resolve the warning. This also
ties into MR 2294 where I removed a feature from "pmbootstrap status"
that would complain about the last fetch of pmaports.git being too old
and was also recommending "pmbootstrap pull". I think having it here
in the warning makes more sense, as more people will see it.
* Remove obvious / not helpful line:
"# Get package name, version, define start of debug message"
* Remove "old" in "Get old version from APKINDEX" because it may be
newer than the version in pmaports
* Replace "Aports [folder]" with pmaports (in some parts of pmbootstrap
code, pmaports is still referred to as aports, this was one of them)
Give more meaningful names to the variables:
* version_new -> version_pmaports
* version_old -> version_binary
This makes the code less confusing for the case where version_binary is
actually newer than version_pmaports. This is a relict from the time
before there was a binary package repository, in that case the version
from pmaports would always be the newer one, built from source, compared
to the local binary package that was probably built before.
A common pattern in APKBUILDs, is to introduce custom variables prefixed
with underscores that get then used in makedepends and other variables.
For example:
_wlrootsmakedepends="
eudev-dev
# ...
"
makedepends="
# ...
$_wlrootsmakedepends
"
Adjust the APKBUILD parser code, so it parses all top-level variables
and can use them further below when referenced inside other variables.
Before returning the parsed APKBUILD data, remove all variables that are
not in pmbootstrap's list of known APKBUILD parsing attributes (so the
result is the same).
I've compared "pmbootstrap apkbuild_parse" (which parses all APKBUILDs
in the currently checked out pmaports dir), before and after this
change, and the result is the same except for having more variables
successfully replaced.
- Performance Note-
This new implementation is actually faster than the previous one,
because we don't need to iterate through all known keys on each line of
the APKBUILDs. On my machine, average of 3 runs, parsing all APKBUILDs
of current pmaports master takes about half as long as with the previous
implementation.
$ time pmbootstrap -q apkbuild_parse >/dev/null
-> old code: 0.954
-> new code: 0.483
Make the handling of the custom NonBugError and BuildFailedError
exceptions more consistent with the handling of other exceptions, by
printing "ERROR: " infront of the actual error text. Then we don't need
to duplicate that where we raise the errors. pmbootstrap prints "ERROR"
in red.
Reimplement "pmbootstrap status" to be just a simple and useful status
overview. The previous version ran a bunch of checks every time, and
would fail on these even if pmaports was used for normal development:
* "non-official" branch checked out in pmaports
* pmaports.git is not clean
The information about aports.git was also considered not so useful upon
revisiting this command, since it is only used for "pmbootstrap
aportgen". Most users don't need this, and if the user runs this
command, it will tell if aports.git is outdated.
All of the above made the previous version unpleasant to use and I
suspect most people stopped using the command after trying it out a few
times and seeing the irrelevant but loud NOK complaints.
New version:
$ pmbootstrap status
Channel: edge (pmaports: master_staging_systemd)
Device: qemu-amd64 (x86_64, kernel: virt)
UI: console
systemd: no (default for selected UI)
Old version (without --details it only shows NOK checks):
$ pmbootstrap status --details
[00:55:20] *** CONFIG ***
[00:55:20] Device: qemu-amd64 (x86_64, "QEMU amd64")
[00:55:20] Kernel: virt
[00:55:20] User Interface: console
[00:55:20]
[00:55:20] *** GIT REPOS ***
[00:55:20] Path: /home/user/.local/var/pmbootstrap/cache_git
[00:55:20] - aports_upstream (master)
[00:55:20] - pmaports (master)
[00:55:20]
[00:55:20] *** CHECKS ***
[00:55:20] [OK ] Chroots zapped recently (or non-existing)
[00:55:20] [OK ] aports_upstream: on official channel branch
[00:55:20] [OK ] aports_upstream: workdir is clean
[00:55:20] [OK ] aports_upstream: tracking proper remote branch 'origin/master'
[00:55:20] [OK ] aports_upstream: up to date with remote branch
[00:55:20] [OK ] aports_upstream: remote information updated recently (via git fetch/pull)
[00:55:20] [OK ] pmaports: on official channel branch
[00:55:20] [OK ] pmaports: workdir is clean
[00:55:20] [OK ] pmaports: tracking proper remote branch 'origin/master'
[00:55:20] [OK ] pmaports: up to date with remote branch
[00:55:20] [OK ] pmaports: remote information updated recently (via git fetch/pull)
[00:55:20]
[00:55:20] NOTE: chroot is still active (use 'pmbootstrap shutdown' as necessary)
[00:55:20] DONE!
Make sure that we build depends of subpackages, to avoid errors like the
following:
* postmarketos-base has a subpackage postmarketos-base-nftables, which
depends on postmarketos-config-nftables
* when pmbootstrap builds postmarketos-base with all its dependencies,
it did not realize it depends on postmarketos-config-nftables through
the subpackage, but it built it anyway
* when trying to install postmarketos-base-nftables, apk complains that
there is no postmarketos-config-nftables:
ERROR: unable to select packages:
postmarketos-config-nftables (no such package):
required by: postmarketos-base-nftables-32-r2[postmarketos-config-nftables]
This also adds the missing depenendencies to "pmbootstrap repo_missing",
which bpo uses to resolve dependencies.
Fixes: issue 2084
Prepare to remove the outdated chroot check from "pmbootstrap status".
Display it when the user enters a stale chroot instead. This way the
user is more likely to see it, and we can make "pmbootstrap status" more
minimal (by removing all checks, in future patches).
Related: issue 1903
With the !pmb:crossdirect option, the crossdirect compilation method
gets disabled. This means there is no /native directory mounted inside
the foreign arch chroot, all binaries inside the foreign arch chroot are
supposed to run in QEMU. Handle this in the apk_wrapper.sh script that
we use to redirect abuild-apk to /native/usr/bin/abuild-apk.
Fix for (currently with master_staging_systemd branch):
>>> upower: Analyzing dependencies...
/usr/local/bin/abuild-apk: line 11: /native/usr/bin/abuild-apk: not found
When the user attempts to build a package with systemd enabled,
pmbootstrap checks first if the postmarketos-base-systemd package is in
any of the APKINDEX files. Fetch the APKINDEXes before trying to find
the package in them.
I'm running into this currently with bpo, when attempting to build any
package from the master_staging_systemd branch.
Previously, if this was the first time you were connecting to a device,
the output of the ssh command would contain a warning about that the
host was added to the list of trusted hosts. This causes problems as
this gets included in the output where the code previously expected the
string-representation of the architecture to be.
As such, remove the irrelevant output by splitting it into lines and
only using the last line as we assume ssh won't print anything after the
actual output of the invoked command.
Closes https://gitlab.com/postmarketOS/pmbootstrap/-/issues/2335
[ci:skip-build]: already built successfully in CI
I have Nushell set up on my Librem 5, and it is not a POSIX-compatible
shell. As such, the rc=$? variable assignment in this snippet fails.
Running this in a sh subshell works around this problems and will likely
fix it for other non-POSIX shells as well given that /bin/sh should at
least be POSIX-compatible on any sensible system.
This works around an issue where some armhf apps are compiled with
instructions that are not compatible with aarch64, for example apk uses
MCR, and that does not exist on aarch64 causing apk to fail in a chroot
with SIGILL. It's impossible to build armhf images on aarch64 hosts, and
this also fixes a crash when running `pmbootstrap zap -a`, since pmb zap
will init an armhf chroot and blow up with apk generates a SIGILL.
For reference, here's how I arrived at the conclusion that apk (and gdb,
and probably others...) are executing invalid instructions on aarch64
when not using binfmt+qemu emulation:
$ uname -m
aarch64
$ file sbin/apk.static
sbin/apk.static: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), statically linked, BuildID[sha1]=997a8ef97b17cb4951a6220b7807a66fed67bc10, stripped
$ gdb sbin/apk.static
(gdb) r
Starting program: /run/user/10000/foo/sbin/apk.static
Program received signal SIGILL, Illegal instruction.
0x0012843c in ?? ()
(gdb) x /1i $pc
=> 0x12843c: mcr 15, 0, r0, cr7, cr10, {5}
I'm not sure why pmb.parse.arch is being imported here given that it's
the module we already are in. Replace it with pmb.config which this
module actually needs.
Instead of assuming the architecture of the foreign device to be what
the user selected in pmbootstrap init, actually query the architecture
from the device and use that.
This does mean that one extra ssh connection is necessary, which does
slow down the procedure somewhat. However, I think that is worth the
user experience improvement this brings.
Also, the deduction process can be skipped by manually specifying
--arch in the sideload invocation should it fail, or if one really wants
to skip that extra ssh roundtrip.
Related: https://gitlab.com/postmarketOS/pmbootstrap/-/issues/2317
Sometimes it's useful to map an arbitrary machine type to an Alpine
architecture. alpine_native only lets you get the Alpine architecture
mapped to the machine type of the system pmbootstrap is running on. As
such, break out this functionality into a new function that takes the
machine type as a parameter.
The intent of 9a74848f19 was to not print
a stacktrace when this happened. However, I still opted to raise an
exception, so you still get a stacktrace, just with a more helpful
message. There's no need to give the user a big stacktrace here, so
raise a NonBugError instead of a RuntimeError.
NonBugError means that the root cause originates from a bug outside of
pmbootstrap's code base. As such, no need to print a stacktrace or ask
the user to update pmbootstrap.
