dtbtool-exynos has two options --platform and --subtype that are
embedded in dt.img header and need to match the values that the
bootloader expects. For most devices these values are 0x50a6 and
0x217584da, respectively, but for some they have other values.
Add functionality to parse the dt.img header and extract these values,
and add them to the deviceinfo as
bootimg_qcdt_exynos_{platform,subtype} if they are not equal to the
default values.
Part-of: https://gitlab.postmarketos.org/postmarketOS/pmbootstrap/-/merge_requests/2638
This is just a small detail, but in some way CLI options are similar
to public API. I did this while trying some changes, even if in the
end the issue to solve was not in pmbootstrap, I thought it'd make
sense to contribute this.
Part-of: https://gitlab.postmarketos.org/postmarketOS/pmbootstrap/-/merge_requests/2626
This test includes a binary blob which we want to avoid in our codebase.
It is also better suited as a unit test where it can contribute to code
coverage. Revert this and use a unit test version instead.
This reverts commit d19de946b3.
(Commit message written by Oliver)
The tests relied on a lot of specifics of the old codebase. We decided
that it makes more sense to just remove all tests for now to get this
large patchset of refactorings and improvements done, and then gradually
add new tests / possibly adjust some of the old ones and bring them back
in future patches.
Future tests will not be in the "test" directory, Caleb's reasoning:
> Easier to find relevant tests for a module, and avoid the big flat
> list of test files like we had before. Honestly was just a vibes
> thing after reading the docs.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Tweaked-By: Oliver Smith <ollieparanoid@postmarketos.org>
Cease merging pmbootstrap.cfg into args, implement a Context type to let
us pull globals out of thin air (as an intermediate workaround) and rip
args out of a lot of the codebase.
This is just a first pass, after this we can split all the state that
leaked over into Context into types with narrower scopes (like a
BuildContext(), etc).
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
With the new chroot type, we can now write fancy paths in the pythonic
way. Convert most of the codebase over, as well as adding various other
type hints.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
We use a custom verbose log level in pmbootstrap, unfortunately it isn't
possible to correctly type this due to some limitations in the logging
library [1], [2].
Given that our usecase is fairly simple, we can just wrap the module
with our own so we only have to tell mypy to ignore the error once
instead of at every callsite.
[1]: https://github.com/cryptax/droidlysis/issues/15
[2]: https://github.com/python/typing/discussions/980
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Introduce a new module: pmb.core to contain explicitly typed pmbootstrap
API. The first component being Suffix and SuffixType. This explicitly
defines what suffixes are possible, future changes should aim to further
constrain this API (e.g. by validating against available device
codenames or architectures for buildroot suffixes).
Additionally, migrate the entire codebase over to using pathlib.Path.
This is a relatively new part of the Python standard library that uses a
more object oriented model for path handling. It also uses strong type
hinting and has other features that make it much cleaner and easier to
work with than pure f-strings. The Chroot class overloads the "/"
operator the same way the Path object does, allowing one to write paths
relative to a given chroot as:
builddir = chroot / "home/pmos/build"
The Chroot class also has a string representation ("native", or
"rootfs_valve-jupiter"), and a .path property for directly accessing the
absolute path (as a Path object).
The general idea here is to encapsulate common patterns into type hinted
code, and gradually reduce the amount of assumptions made around the
codebase so that future changes are easier to implement.
As the chroot suffixes are now part of the Chroot class, we also
implement validation for them, this encodes the rules on suffix naming
and will cause a runtime exception if a suffix doesn't follow the rules.
This test is *only* failing in gitlab CI and I haven't been able to
figure out why (see [1]). Everything else (e.g. bpo, building packages
locally) seems to work fine. Since this is blocking pmbootstrap MRs, I'm
going to disable the test until we can figure out wtf is going on with
running it in gitlab CI.
1. https://gitlab.com/postmarketOS/pmbootstrap/-/issues/2346
[ci:skip-build]: already built successfully in CI
A common pattern in APKBUILDs, is to introduce custom variables prefixed
with underscores that get then used in makedepends and other variables.
For example:
_wlrootsmakedepends="
eudev-dev
# ...
"
makedepends="
# ...
$_wlrootsmakedepends
"
Adjust the APKBUILD parser code, so it parses all top-level variables
and can use them further below when referenced inside other variables.
Before returning the parsed APKBUILD data, remove all variables that are
not in pmbootstrap's list of known APKBUILD parsing attributes (so the
result is the same).
I've compared "pmbootstrap apkbuild_parse" (which parses all APKBUILDs
in the currently checked out pmaports dir), before and after this
change, and the result is the same except for having more variables
successfully replaced.
- Performance Note-
This new implementation is actually faster than the previous one,
because we don't need to iterate through all known keys on each line of
the APKBUILDs. On my machine, average of 3 runs, parsing all APKBUILDs
of current pmaports master takes about half as long as with the previous
implementation.
$ time pmbootstrap -q apkbuild_parse >/dev/null
-> old code: 0.954
-> new code: 0.483
Add the "args" pytest.fixture to each test as argument, so it runs the
related code of initializing logging and the cache (see top of the
test_parse_apkbuild.py). Without this, running the test file without any
prior tests (that do these initializations) is not possible:
$ pytest -xvv test/test_parse_apkbuild.py
…
> if path in pmb.helpers.other.cache["apkbuild"]:
E TypeError: 'NoneType' object is not subscriptable
Reimplement "pmbootstrap status" to be just a simple and useful status
overview. The previous version ran a bunch of checks every time, and
would fail on these even if pmaports was used for normal development:
* "non-official" branch checked out in pmaports
* pmaports.git is not clean
The information about aports.git was also considered not so useful upon
revisiting this command, since it is only used for "pmbootstrap
aportgen". Most users don't need this, and if the user runs this
command, it will tell if aports.git is outdated.
All of the above made the previous version unpleasant to use and I
suspect most people stopped using the command after trying it out a few
times and seeing the irrelevant but loud NOK complaints.
New version:
$ pmbootstrap status
Channel: edge (pmaports: master_staging_systemd)
Device: qemu-amd64 (x86_64, kernel: virt)
UI: console
systemd: no (default for selected UI)
Old version (without --details it only shows NOK checks):
$ pmbootstrap status --details
[00:55:20] *** CONFIG ***
[00:55:20] Device: qemu-amd64 (x86_64, "QEMU amd64")
[00:55:20] Kernel: virt
[00:55:20] User Interface: console
[00:55:20]
[00:55:20] *** GIT REPOS ***
[00:55:20] Path: /home/user/.local/var/pmbootstrap/cache_git
[00:55:20] - aports_upstream (master)
[00:55:20] - pmaports (master)
[00:55:20]
[00:55:20] *** CHECKS ***
[00:55:20] [OK ] Chroots zapped recently (or non-existing)
[00:55:20] [OK ] aports_upstream: on official channel branch
[00:55:20] [OK ] aports_upstream: workdir is clean
[00:55:20] [OK ] aports_upstream: tracking proper remote branch 'origin/master'
[00:55:20] [OK ] aports_upstream: up to date with remote branch
[00:55:20] [OK ] aports_upstream: remote information updated recently (via git fetch/pull)
[00:55:20] [OK ] pmaports: on official channel branch
[00:55:20] [OK ] pmaports: workdir is clean
[00:55:20] [OK ] pmaports: tracking proper remote branch 'origin/master'
[00:55:20] [OK ] pmaports: up to date with remote branch
[00:55:20] [OK ] pmaports: remote information updated recently (via git fetch/pull)
[00:55:20]
[00:55:20] NOTE: chroot is still active (use 'pmbootstrap shutdown' as necessary)
[00:55:20] DONE!
Make sure that we build depends of subpackages, to avoid errors like the
following:
* postmarketos-base has a subpackage postmarketos-base-nftables, which
depends on postmarketos-config-nftables
* when pmbootstrap builds postmarketos-base with all its dependencies,
it did not realize it depends on postmarketos-config-nftables through
the subpackage, but it built it anyway
* when trying to install postmarketos-base-nftables, apk complains that
there is no postmarketos-config-nftables:
ERROR: unable to select packages:
postmarketos-config-nftables (no such package):
required by: postmarketos-base-nftables-32-r2[postmarketos-config-nftables]
This also adds the missing depenendencies to "pmbootstrap repo_missing",
which bpo uses to resolve dependencies.
Fixes: issue 2084
The intent of 9a74848f19 was to not print
a stacktrace when this happened. However, I still opted to raise an
exception, so you still get a stacktrace, just with a more helpful
message. There's no need to give the user a big stacktrace here, so
raise a NonBugError instead of a RuntimeError.
This drops the prompt for using non-free firmware in images. The logic
for searching/installing non-free fw subpackages for devices is kept,
and will always be installed. This is to support the many device
packages in pmaports that still have nonfree-firmware subpackages. Going
forward, device packages can list firmware in `depends=` (for required
fw) or `pmb_recommends` (for optional fw).
nonfree-userland wasn't used in pmaports as far as I could find.
Fix "pmbootstrap chroot" and others not passing the proxy environment
variables correctly. Thanks to notfound405 for pointing this out!
Instead of only preserving proxy environment variables in
pmb.helpers.run_core, which should never be called directly, do it in
the calling functions:
* pmb.helpers.run.user
* pmb.helpers.run.root
* pmb.chroot.root
* pmb.chroot.user
This fixes that the environment variables were only really passed by
pmb.helpers.run.user, because the other functions would result in
something like:
HTTP_PROXY=mytestproxy sudo env -i /usr/bin/sh -c '…'
This is needed to either elevate to root, or to elevate to root first
and then enter the chroot as root or user. Due to the "env -i", the
environment intentionally gets cleaned, but unintentionally also removes
the proxy environment variables that were explicitly set.
By adjusting the functions, they now run a variant of:
sudo env -i /usr/bin/sh -c 'HTTP_PROXY=mytestproxy …'
The escaping is simplified in this example, run "pmbootstrap -v" to see
the not very readable, but proper escaping with shutil.quote().
Remove the previous test for preserving the environment variables in
pmb.helpers.run_core (as it should never be called directly), and test
instead the new behavior.
Fixes: issue 2299
Fixes: 13c4ac42 ("pmb.helpers.run_core: fix proxy env var logic")
Sometimes I want to build an older version of a package from Alpine, and
since package upgrades can involve things like patches and other
externalities just changing the pkgver and running checksum in pmaports
may not be enough. As such, it tends to be easier to revert the change
in the local aports repo and then fork than forking and then trying to
manually revert the changes yourself (since you can't have git do that
for you given that they are distinct repositories).
Prior to this patch, that was not possible since pmbootstrap would
assume older aport version equals outdated aports in general and as such
cancel the whole operation. Instead, just print a warning and helpful
information to make this workflow possible while also warning users that
they may want to update their local aports.
Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20231209111813.37756-1-newbyte@postmarketos.org%3E
This fixes an issue when pmb is run on an Alpine/pmOS host and apk.static sees
that /etc/apk/interactive is set on the host. It's really annoying to have the
build pause so apk can prompt when building chroots, so this uses a flag to apk
to disable interactive mode. I can't think of any situations where we would
actually want to prompt users when building chroots, by then all user
configuration should have been completed via pmb init/config.
Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Co-developed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20231110185320.8358-1-clayton@craftyguy.net%3E
Currently, pmbootstrap checks if either the kernel or the ramdisk in a boot.img
contains the MediaTek header, and if one does, it assumes both do. It hardcodes
the label KERNEL for the kernel and ROOTFS for the ramdisk.
My Amazon Echo Dot (gen 2) has a boot.img where only the kernel has the header,
but not the ramdisk. These changes (as well as those in my new boot-deploy MR)
account for that situation (and any possible label an image has) by splitting
bootimg_mtk_mkimage into two variables for the kernel and the ramdisk labels.
Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/patches/46351
Remove test_crossdirect_rust, the only test in test_crossdirect.py. The
behavior that was tested here with all the "mv /usr/bin/rustc" commands
was specific to version 4 and isn't valid anymore since version 5
(pmaports MR 4234).
The previous behavior was that crossdirect always tried to run rustc
from /native, and if it failed, it would fall back to using qemu. The
new behavior is that it runs rustc from /native only when it expects it
to succeed, and runs rustc via qemu otherwise, without attempting
/native first. As the logic to decide whether to use native rustc or not
is now done beforehand, it does not do the fallback to qemu if native
fails.
Note that the test hasn't been failing because we switch to the v23.06
channel and therefore use the older crossdirect version.
Reviewed-by: Luca Weiss <luca@z3ntu.xyz>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230806184729.4891-6-ollieparanoid@postmarketos.org%3E
Cache the compiler output of rust code with sccache, like we use ccache
for c code.
I've considered using sccache to completely replace ccache since it can
cache output of C/C++ code too. But let's not do it for now since ccache
doesn't need to run a daemon in the background that needs to be stopped
when shutting down / zapping. Also it would need more refactoring.
Reviewed-by: Luca Weiss <luca@z3ntu.xyz>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230806184729.4891-5-ollieparanoid@postmarketos.org%3E
