Cease merging pmbootstrap.cfg into args, implement a Context type to let
us pull globals out of thin air (as an intermediate workaround) and rip
args out of a lot of the codebase.
This is just a first pass, after this we can split all the state that
leaked over into Context into types with narrower scopes (like a
BuildContext(), etc).
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
We currently lazily initialize the chroot's on first use, plus a few
bonus calls to init. However, there are some instances where we actually
don't want the chroot to be initialised (mostly to break recursion
loops).
Simplify the codebase by removing all of this, and just calling
pmb.chroot.init() where it's needed.
In addition, print a warning if init() is called multiple times for one
chroot. This should help us catch these instances if they crop up again.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Defaulting to the native chroot isn't necessarily intuitive. Let's
require this be specified in full.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
This is multithreaded, and by consequence much much faster than default
gzip. Since we switched to running gzip from the native chroot we also
changed from running it in the fastest mode, now we're running it with
-9 it can be pretty slow on any kind of mobile hardware.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
With the new chroot type, we can now write fancy paths in the pythonic
way. Convert most of the codebase over, as well as adding various other
type hints.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
We use a custom verbose log level in pmbootstrap, unfortunately it isn't
possible to correctly type this due to some limitations in the logging
library [1], [2].
Given that our usecase is fairly simple, we can just wrap the module
with our own so we only have to tell mypy to ignore the error once
instead of at every callsite.
[1]: https://github.com/cryptax/droidlysis/issues/15
[2]: https://github.com/python/typing/discussions/980
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Introduce a new module: pmb.core to contain explicitly typed pmbootstrap
API. The first component being Suffix and SuffixType. This explicitly
defines what suffixes are possible, future changes should aim to further
constrain this API (e.g. by validating against available device
codenames or architectures for buildroot suffixes).
Additionally, migrate the entire codebase over to using pathlib.Path.
This is a relatively new part of the Python standard library that uses a
more object oriented model for path handling. It also uses strong type
hinting and has other features that make it much cleaner and easier to
work with than pure f-strings. The Chroot class overloads the "/"
operator the same way the Path object does, allowing one to write paths
relative to a given chroot as:
builddir = chroot / "home/pmos/build"
The Chroot class also has a string representation ("native", or
"rootfs_valve-jupiter"), and a .path property for directly accessing the
absolute path (as a Path object).
The general idea here is to encapsulate common patterns into type hinted
code, and gradually reduce the amount of assumptions made around the
codebase so that future changes are easier to implement.
As the chroot suffixes are now part of the Chroot class, we also
implement validation for them, this encodes the rules on suffix naming
and will cause a runtime exception if a suffix doesn't follow the rules.
* Replace aports -> pmaports
* Make the "binary repo has newer version" message much shorter, and
mention "pmbootstrap pull". If users didn't mess with their pmaports
repository (checkout a custom branch, make commits), this command
will update to the latest commit and resolve the warning. This also
ties into MR 2294 where I removed a feature from "pmbootstrap status"
that would complain about the last fetch of pmaports.git being too old
and was also recommending "pmbootstrap pull". I think having it here
in the warning makes more sense, as more people will see it.
* Remove obvious / not helpful line:
"# Get package name, version, define start of debug message"
* Remove "old" in "Get old version from APKINDEX" because it may be
newer than the version in pmaports
* Replace "Aports [folder]" with pmaports (in some parts of pmbootstrap
code, pmaports is still referred to as aports, this was one of them)
Give more meaningful names to the variables:
* version_new -> version_pmaports
* version_old -> version_binary
This makes the code less confusing for the case where version_binary is
actually newer than version_pmaports. This is a relict from the time
before there was a binary package repository, in that case the version
from pmaports would always be the newer one, built from source, compared
to the local binary package that was probably built before.
Make the handling of the custom NonBugError and BuildFailedError
exceptions more consistent with the handling of other exceptions, by
printing "ERROR: " infront of the actual error text. Then we don't need
to duplicate that where we raise the errors. pmbootstrap prints "ERROR"
in red.
Make sure that we build depends of subpackages, to avoid errors like the
following:
* postmarketos-base has a subpackage postmarketos-base-nftables, which
depends on postmarketos-config-nftables
* when pmbootstrap builds postmarketos-base with all its dependencies,
it did not realize it depends on postmarketos-config-nftables through
the subpackage, but it built it anyway
* when trying to install postmarketos-base-nftables, apk complains that
there is no postmarketos-config-nftables:
ERROR: unable to select packages:
postmarketos-config-nftables (no such package):
required by: postmarketos-base-nftables-32-r2[postmarketos-config-nftables]
This also adds the missing depenendencies to "pmbootstrap repo_missing",
which bpo uses to resolve dependencies.
Fixes: issue 2084
With the !pmb:crossdirect option, the crossdirect compilation method
gets disabled. This means there is no /native directory mounted inside
the foreign arch chroot, all binaries inside the foreign arch chroot are
supposed to run in QEMU. Handle this in the apk_wrapper.sh script that
we use to redirect abuild-apk to /native/usr/bin/abuild-apk.
Fix for (currently with master_staging_systemd branch):
>>> upower: Analyzing dependencies...
/usr/local/bin/abuild-apk: line 11: /native/usr/bin/abuild-apk: not found
Only install the abuild-apk wrapper if cpu emulation is required. This
fixes building for x86 on x86_64.
Fix for:
>>> postmarketos-base-ui-gnome: Analyzing dependencies...
/usr/local/bin/abuild-apk: line 11: /native/usr/bin/abuild-apk: not found
Fixes: c5ca06d5 ("pmb: only enable abuild-apk wrapper for buildroot (MR 2246)")
Drop the weird flag file stuff for state management, and just always
mount the source code in, and always unmount it on exit - including in
the error path.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
When running build --force, if a later package is a dependency of an
earlier package, it will be "visited" by the dependency resolver,
erroneously causing it to be skipped when later in the session it's
visited to force build. This is because previously visited packages are
marked as such in a hashmap, but skip_already_built() assumes that a
package which has been visited would have already been built.
Fix this by overriding skip_already_built() if doing a force build. This
works because package dependencies are only built if the APKBUILD
version is newer than the binary repo, even when --force is specified.
So there is no risk of an infinite loop here.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Some packages like kernels are cross compiled from the native chroot.
the apk_wrapper won't work here! So only enable it for buildroot chroots
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Speed up using abuild to install build dependencies by introducing a
wrapper which invokes abuild-apk in the native chroot with
LD_PRELOAD_PATH
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
find_kbuild_output_dir() searches for certain path patterns, which
won't be in the APKBUILD if it uses downstreamkernel_package. Guess
the default kbuild out dir in that case.
Theoretically this might break if some APKBUILD passes weird paths to
downstreamkernel_package, but no kernel package in pmaports today does
that.
Cache the compiler output of rust code with sccache, like we use ccache
for c code.
I've considered using sccache to completely replace ccache since it can
cache output of C/C++ code too. But let's not do it for now since ccache
doesn't need to run a daemon in the background that needs to be stopped
when shutting down / zapping. Also it would need more refactoring.
Reviewed-by: Luca Weiss <luca@z3ntu.xyz>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230806184729.4891-5-ollieparanoid@postmarketos.org%3E
The changes in pmaports!4234 cause cargo build scripts to be built
for and run on the host (native) architecture, so their dependencies
must be present in the native chroot.
pmbootstrap can't tell which of the dependencies specified in
makedepends are actually needed by build scripts, so just duplicate all
of them in the native chroot.
Acked-by: Caleb Connolly <kc@postmarketos.org>
Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230710135001.184769-1-otto.pflueger@abscue.de%3E
With this code path, pmbootstrap would start a distccd + sshd in the
native chroot, and configure it so it runs the cross compiler. The
foreign arch chroots would then call this cross compiler from localhost
by calling the distcc client instead of gcc.
This code has been obsoleted by the much simpler crossdirect in 2019.
Let's finally remove it.
Fixes: issue 2179
Reviewed-by: Luca Weiss <luca@z3ntu.xyz>
Reviewed-by: Clayton Craft <clayton@craftyguy.net>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230613161437.570196-4-ollieparanoid@postmarketos.org%3E
When invoking pmbootstrap build most packages default to the host arch,
however depending on your workflow it might be preferrable to default to
the device arch.
Add a new config option "build_default_device_arch" which when set will
make "pmbootstrap build" prioritise the device arch over the native
arch.
Default to False to preserve the old behaviour and don't ask during
pmbootstrap init as this may not be relevant for most folks.
Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Co-developed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230418-build-default-arch-v5-2-5223fab65867@postmarketos.org%3E
Put it in /tmp and touch it directly from python instead of running
pmb.chroot.root. This way it's slightly faster and doesn't require root
rights. Order the imports alphabetically while at it, and remove very
obvious comments.
Reviewed-by: Caleb Connolly <kc@postmarketos.org>
Tested-by: Caleb Connolly <kc@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230419192042.3951-2-ollieparanoid@postmarketos.org%3E
abuild.conf got moved to /usr/share/abuild/default.conf and
/etc/abuild.conf is now just for user-changes. Adjust
configure_abuild() to append to /etc/abuild.conf if there is no line
defining JOBS
This should be backward compatible
Signed-off-by: Caleb Connolly <kc@postmarketos.org>
Tested-by: Oliver Smith <ollieparanoid@postmarketos.org>
Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Always set GOCACHE during build and let it point to pmbootstrap's work
dir. This has a similar effect as using ccache for C/C++.
Set GOMODCACHE conditionally: this is for caching the go modules (git
repositories) that get cloned during a build if they are not bundled
with the source. Usually APKBUILDs should cache them, but when using
pmbootstrap build --src, they would get downloaded over and over again.
Set GOMODCACHE automatically for --src and allow enabling/disabling this
manually with new --go-mod-cache / --no-go-mod-cache.
This speeds up multiple iterations of building the same package
significantly. I'm using it for:
$ pmbootstrap build postmarketos-mkinitfs --src=...
I've verified that using the same GOCACHE dir for multiple architectures
doesn't result in build artifacts for the wrong arch getting used.
Reviewed-by: Clayton Craft <clayton@craftyguy.net>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230223064743.1773-1-ollieparanoid@postmarketos.org%3E
