We use a custom verbose log level in pmbootstrap, unfortunately it isn't
possible to correctly type this due to some limitations in the logging
library [1], [2].
Given that our usecase is fairly simple, we can just wrap the module
with our own so we only have to tell mypy to ignore the error once
instead of at every callsite.
[1]: https://github.com/cryptax/droidlysis/issues/15
[2]: https://github.com/python/typing/discussions/980
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Introduce a new module: pmb.core to contain explicitly typed pmbootstrap
API. The first component being Suffix and SuffixType. This explicitly
defines what suffixes are possible, future changes should aim to further
constrain this API (e.g. by validating against available device
codenames or architectures for buildroot suffixes).
Additionally, migrate the entire codebase over to using pathlib.Path.
This is a relatively new part of the Python standard library that uses a
more object oriented model for path handling. It also uses strong type
hinting and has other features that make it much cleaner and easier to
work with than pure f-strings. The Chroot class overloads the "/"
operator the same way the Path object does, allowing one to write paths
relative to a given chroot as:
builddir = chroot / "home/pmos/build"
The Chroot class also has a string representation ("native", or
"rootfs_valve-jupiter"), and a .path property for directly accessing the
absolute path (as a Path object).
The general idea here is to encapsulate common patterns into type hinted
code, and gradually reduce the amount of assumptions made around the
codebase so that future changes are easier to implement.
As the chroot suffixes are now part of the Chroot class, we also
implement validation for them, this encodes the rules on suffix naming
and will cause a runtime exception if a suffix doesn't follow the rules.
This test is *only* failing in gitlab CI and I haven't been able to
figure out why (see [1]). Everything else (e.g. bpo, building packages
locally) seems to work fine. Since this is blocking pmbootstrap MRs, I'm
going to disable the test until we can figure out wtf is going on with
running it in gitlab CI.
1. https://gitlab.com/postmarketOS/pmbootstrap/-/issues/2346
[ci:skip-build]: already built successfully in CI
A common pattern in APKBUILDs, is to introduce custom variables prefixed
with underscores that get then used in makedepends and other variables.
For example:
_wlrootsmakedepends="
eudev-dev
# ...
"
makedepends="
# ...
$_wlrootsmakedepends
"
Adjust the APKBUILD parser code, so it parses all top-level variables
and can use them further below when referenced inside other variables.
Before returning the parsed APKBUILD data, remove all variables that are
not in pmbootstrap's list of known APKBUILD parsing attributes (so the
result is the same).
I've compared "pmbootstrap apkbuild_parse" (which parses all APKBUILDs
in the currently checked out pmaports dir), before and after this
change, and the result is the same except for having more variables
successfully replaced.
- Performance Note-
This new implementation is actually faster than the previous one,
because we don't need to iterate through all known keys on each line of
the APKBUILDs. On my machine, average of 3 runs, parsing all APKBUILDs
of current pmaports master takes about half as long as with the previous
implementation.
$ time pmbootstrap -q apkbuild_parse >/dev/null
-> old code: 0.954
-> new code: 0.483
Add the "args" pytest.fixture to each test as argument, so it runs the
related code of initializing logging and the cache (see top of the
test_parse_apkbuild.py). Without this, running the test file without any
prior tests (that do these initializations) is not possible:
$ pytest -xvv test/test_parse_apkbuild.py
…
> if path in pmb.helpers.other.cache["apkbuild"]:
E TypeError: 'NoneType' object is not subscriptable
Reimplement "pmbootstrap status" to be just a simple and useful status
overview. The previous version ran a bunch of checks every time, and
would fail on these even if pmaports was used for normal development:
* "non-official" branch checked out in pmaports
* pmaports.git is not clean
The information about aports.git was also considered not so useful upon
revisiting this command, since it is only used for "pmbootstrap
aportgen". Most users don't need this, and if the user runs this
command, it will tell if aports.git is outdated.
All of the above made the previous version unpleasant to use and I
suspect most people stopped using the command after trying it out a few
times and seeing the irrelevant but loud NOK complaints.
New version:
$ pmbootstrap status
Channel: edge (pmaports: master_staging_systemd)
Device: qemu-amd64 (x86_64, kernel: virt)
UI: console
systemd: no (default for selected UI)
Old version (without --details it only shows NOK checks):
$ pmbootstrap status --details
[00:55:20] *** CONFIG ***
[00:55:20] Device: qemu-amd64 (x86_64, "QEMU amd64")
[00:55:20] Kernel: virt
[00:55:20] User Interface: console
[00:55:20]
[00:55:20] *** GIT REPOS ***
[00:55:20] Path: /home/user/.local/var/pmbootstrap/cache_git
[00:55:20] - aports_upstream (master)
[00:55:20] - pmaports (master)
[00:55:20]
[00:55:20] *** CHECKS ***
[00:55:20] [OK ] Chroots zapped recently (or non-existing)
[00:55:20] [OK ] aports_upstream: on official channel branch
[00:55:20] [OK ] aports_upstream: workdir is clean
[00:55:20] [OK ] aports_upstream: tracking proper remote branch 'origin/master'
[00:55:20] [OK ] aports_upstream: up to date with remote branch
[00:55:20] [OK ] aports_upstream: remote information updated recently (via git fetch/pull)
[00:55:20] [OK ] pmaports: on official channel branch
[00:55:20] [OK ] pmaports: workdir is clean
[00:55:20] [OK ] pmaports: tracking proper remote branch 'origin/master'
[00:55:20] [OK ] pmaports: up to date with remote branch
[00:55:20] [OK ] pmaports: remote information updated recently (via git fetch/pull)
[00:55:20]
[00:55:20] NOTE: chroot is still active (use 'pmbootstrap shutdown' as necessary)
[00:55:20] DONE!
Make sure that we build depends of subpackages, to avoid errors like the
following:
* postmarketos-base has a subpackage postmarketos-base-nftables, which
depends on postmarketos-config-nftables
* when pmbootstrap builds postmarketos-base with all its dependencies,
it did not realize it depends on postmarketos-config-nftables through
the subpackage, but it built it anyway
* when trying to install postmarketos-base-nftables, apk complains that
there is no postmarketos-config-nftables:
ERROR: unable to select packages:
postmarketos-config-nftables (no such package):
required by: postmarketos-base-nftables-32-r2[postmarketos-config-nftables]
This also adds the missing depenendencies to "pmbootstrap repo_missing",
which bpo uses to resolve dependencies.
Fixes: issue 2084
The intent of 9a74848f19 was to not print
a stacktrace when this happened. However, I still opted to raise an
exception, so you still get a stacktrace, just with a more helpful
message. There's no need to give the user a big stacktrace here, so
raise a NonBugError instead of a RuntimeError.
This drops the prompt for using non-free firmware in images. The logic
for searching/installing non-free fw subpackages for devices is kept,
and will always be installed. This is to support the many device
packages in pmaports that still have nonfree-firmware subpackages. Going
forward, device packages can list firmware in `depends=` (for required
fw) or `pmb_recommends` (for optional fw).
nonfree-userland wasn't used in pmaports as far as I could find.
Fix "pmbootstrap chroot" and others not passing the proxy environment
variables correctly. Thanks to notfound405 for pointing this out!
Instead of only preserving proxy environment variables in
pmb.helpers.run_core, which should never be called directly, do it in
the calling functions:
* pmb.helpers.run.user
* pmb.helpers.run.root
* pmb.chroot.root
* pmb.chroot.user
This fixes that the environment variables were only really passed by
pmb.helpers.run.user, because the other functions would result in
something like:
HTTP_PROXY=mytestproxy sudo env -i /usr/bin/sh -c '…'
This is needed to either elevate to root, or to elevate to root first
and then enter the chroot as root or user. Due to the "env -i", the
environment intentionally gets cleaned, but unintentionally also removes
the proxy environment variables that were explicitly set.
By adjusting the functions, they now run a variant of:
sudo env -i /usr/bin/sh -c 'HTTP_PROXY=mytestproxy …'
The escaping is simplified in this example, run "pmbootstrap -v" to see
the not very readable, but proper escaping with shutil.quote().
Remove the previous test for preserving the environment variables in
pmb.helpers.run_core (as it should never be called directly), and test
instead the new behavior.
Fixes: issue 2299
Fixes: 13c4ac42 ("pmb.helpers.run_core: fix proxy env var logic")
Sometimes I want to build an older version of a package from Alpine, and
since package upgrades can involve things like patches and other
externalities just changing the pkgver and running checksum in pmaports
may not be enough. As such, it tends to be easier to revert the change
in the local aports repo and then fork than forking and then trying to
manually revert the changes yourself (since you can't have git do that
for you given that they are distinct repositories).
Prior to this patch, that was not possible since pmbootstrap would
assume older aport version equals outdated aports in general and as such
cancel the whole operation. Instead, just print a warning and helpful
information to make this workflow possible while also warning users that
they may want to update their local aports.
Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20231209111813.37756-1-newbyte@postmarketos.org%3E
This fixes an issue when pmb is run on an Alpine/pmOS host and apk.static sees
that /etc/apk/interactive is set on the host. It's really annoying to have the
build pause so apk can prompt when building chroots, so this uses a flag to apk
to disable interactive mode. I can't think of any situations where we would
actually want to prompt users when building chroots, by then all user
configuration should have been completed via pmb init/config.
Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Co-developed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20231110185320.8358-1-clayton@craftyguy.net%3E
Currently, pmbootstrap checks if either the kernel or the ramdisk in a boot.img
contains the MediaTek header, and if one does, it assumes both do. It hardcodes
the label KERNEL for the kernel and ROOTFS for the ramdisk.
My Amazon Echo Dot (gen 2) has a boot.img where only the kernel has the header,
but not the ramdisk. These changes (as well as those in my new boot-deploy MR)
account for that situation (and any possible label an image has) by splitting
bootimg_mtk_mkimage into two variables for the kernel and the ramdisk labels.
Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/patches/46351
Remove test_crossdirect_rust, the only test in test_crossdirect.py. The
behavior that was tested here with all the "mv /usr/bin/rustc" commands
was specific to version 4 and isn't valid anymore since version 5
(pmaports MR 4234).
The previous behavior was that crossdirect always tried to run rustc
from /native, and if it failed, it would fall back to using qemu. The
new behavior is that it runs rustc from /native only when it expects it
to succeed, and runs rustc via qemu otherwise, without attempting
/native first. As the logic to decide whether to use native rustc or not
is now done beforehand, it does not do the fallback to qemu if native
fails.
Note that the test hasn't been failing because we switch to the v23.06
channel and therefore use the older crossdirect version.
Reviewed-by: Luca Weiss <luca@z3ntu.xyz>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230806184729.4891-6-ollieparanoid@postmarketos.org%3E
Cache the compiler output of rust code with sccache, like we use ccache
for c code.
I've considered using sccache to completely replace ccache since it can
cache output of C/C++ code too. But let's not do it for now since ccache
doesn't need to run a daemon in the background that needs to be stopped
when shutting down / zapping. Also it would need more refactoring.
Reviewed-by: Luca Weiss <luca@z3ntu.xyz>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230806184729.4891-5-ollieparanoid@postmarketos.org%3E
Add the "args" fixture to test_skip_already_built, so it correctly
initializes logging (and indirectly pmb.helpers.other.cache). With this
it becomes possible to run the test file on its own. See "def args" on
top of the file for implementation of the "args" fixture.
Fix for:
$ pytest -xvv test/test_build_package.py
…
> assert pmb.helpers.other.cache["built"] == {} E TypeError: 'NoneType' object is not subscriptable
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230724070509.1885-2-ollieparanoid@postmarketos.org%3E
Fix that the list "cmd" was turned into a string if one of the proxy
vars was set in the environment. Add a test for this code path. Before
this patch:
$ FTP_PROXY=test pmbootstrap -v --details-to-stdout status
…
% cd /home/user/.local/var/pmbootstrap/cache_git/pmaports; git remote -v
run: FTP_PROXY=test ['git', 'remote', '-v']
ERROR: [Errno 2] No such file or directory: "FTP_PROXY=test ['git', 'remote', '-v']"
Fixes: 1a00c04f ("pmb.helpers.run_core: always configure proxy vars if set in environment")
Reviewed-by: Clayton Craft <clayton@craftyguy.net>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230713182337.6185-3-ollieparanoid@postmarketos.org%3E
With this code path, pmbootstrap would start a distccd + sshd in the
native chroot, and configure it so it runs the cross compiler. The
foreign arch chroots would then call this cross compiler from localhost
by calling the distcc client instead of gcc.
This code has been obsoleted by the much simpler crossdirect in 2019.
Let's finally remove it.
Fixes: issue 2179
Reviewed-by: Luca Weiss <luca@z3ntu.xyz>
Reviewed-by: Clayton Craft <clayton@craftyguy.net>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230613161437.570196-4-ollieparanoid@postmarketos.org%3E
Rename the file so it runs last, as it's currently failing a lot of
times on sourcehut. When I looked into it last time, it seemed to be
related to CDN caching and then it worked again so I didn't look further
into it at the time. Move it to the end until this is figured out
properly, so at least we see if all other tests are passing.
Replace the old kconfig check tests with new ones, that do not rely on:
* any kernel config in pmaports
* the currently required kconfig options set in pmb/config/__init__.py
This will make it much easier to change the required options, and moves
the testsuite a bit more towards making it completely independent from
current pmaports state.
Reviewed-by: Clayton Craft <clayton@craftyguy.net>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230312151325.1968-13-ollieparanoid@postmarketos.org%3E
pmOS cannot boot without these two. There was recently someone in the
unofficial postmarketOS Telegram group who used some minimal config and
couldn't figure out why the initramfs couldn't start up, and these
options being disabled turned out to be the culprit. As such, let's
ensure these are enabled to make life easier for people setting up new
kernel configurations.
Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Tested-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230128203716.152281-1-newbyte@postmarketos.org%3E
Always set GOCACHE during build and let it point to pmbootstrap's work
dir. This has a similar effect as using ccache for C/C++.
Set GOMODCACHE conditionally: this is for caching the go modules (git
repositories) that get cloned during a build if they are not bundled
with the source. Usually APKBUILDs should cache them, but when using
pmbootstrap build --src, they would get downloaded over and over again.
Set GOMODCACHE automatically for --src and allow enabling/disabling this
manually with new --go-mod-cache / --no-go-mod-cache.
This speeds up multiple iterations of building the same package
significantly. I'm using it for:
$ pmbootstrap build postmarketos-mkinitfs --src=...
I've verified that using the same GOCACHE dir for multiple architectures
doesn't result in build artifacts for the wrong arch getting used.
Reviewed-by: Clayton Craft <clayton@craftyguy.net>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230223064743.1773-1-ollieparanoid@postmarketos.org%3E
Running abuild on the host directly creates directories in the
aport where it gets built. Interrupting abuild results in those
working-dirs not getting deleted.
We don't want to copy those entries to our builder.
It's only really noticeable if pmbootstrap tries to copy a broken
symlink in src/ and thus fails.
Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20221207205201.22139-1-jane400@bingo-ev.de%3E
