In ("build: special case building abuild (MR 2356)") we tried to ensure
that abuild would be built before anything else, however this
implementation failed miserably if the first package given didn't
actually need to be built (since we skip building deps of packages that
aren't flagged for build).
Let's take a different stab at this by actually inserting build_packages
into the build queue if they need building. They're inserted at the end
just before the queue is reversed, so they will always be built first.
This makes the order of pmb.config.build_packages important, enshrine
this with a comment.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
If you build a package with --src but some dep is outdated, force you
to build it (or otherwise handle the situation yourself). We can't guess
what to do here.
This could be made configurable (opt-in) in the future.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
We might have just built some other package which will later be used to
build more packages. Handle all of them and don't special case abuild.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Since abuild is an implicit dependency of everything, we need special
case handling for it. Insert it as a dependency of the first package in
the build queue so that it will be built if outdated, then after
building re-install it in the chroot so all subsequent packages will use
the freshly built abuild.
This is currently used for systemd where we use an abuild fork
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
During rework this got lost, re-incorporate it so packages built with
--src have a relevant pkgver.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
This is a potential security risk. Tighten things up so we only allow
local checksums to mismatch but remote ones must be correct.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
This avoids copying local build artifacts and other things, if they're
in .gitignore then abuild shouldn't need them anyway.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
We can fairly easily patch abuild by appending anything we want to the
APKBUILD file after copying it to the chroot.
Leverage this to override the verify() function so that it doesn't die
when checksums don't match.
Instead let's build the package anyway, but set a flag and print a
warning with instructions on how to generate the checksums.
We should continue to require APKBUILDs in pmaports to have valid
checksums.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
It was the case that packages were being queued in the wrong order,
since the dependency resolver queues the package before descending into
it's dependencies. This was a bit of a goof since the top level package
was always added last. If we add it first then we can just reverse the
queue and now everything is fine...
Additionally, the logic on when a dependency should be built was a bit
wonky. A case is added for when a dependency exists only in the source
repo and the requisite package isn't marked for build. The solver will
now build the dependency regardless. This is surely an edgecase but
somehow I ran into it, I suspect due to a bug elsewhere...
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Rename build.package() to build.packages() and take a list of packages
to build, since every caller was inside a for loop this simplifies
usage and let's us give nicer log output by doing all the builds first,
so log messages don't get lost in the middle.
Behaviour is cleaned up so this shouuuuld work pretty well now. It
properly descends into dependencies and will build dependencies even if
the package given doesn't need building. Technically this was only done
before during install where the dependencies were recursed in
chroot.apk.install().
It probably makes the most sense to have a mode where it doesn't build
dependencies but warns the user about it, at least when invoked via
pmbootstrap build.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
The package builder has long been a pain point since it recurses the
entire dependency tree.
Convert it to run in two stages, first it walks through the package
dependencies, descending into each one and processing them in a queue.
If a package is determined to need building then it gets pushed onto the
build_queue.
Then, it pops each package off the build queue (which is actually a
stack..) and builds it.
This avoids recursion entirely and should open the door to optimisations
in the future.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Move pmb/parse/arch.py over to core and refactor it as an Arch type,
similar to how Chroot was done. Fix all the uses (that I can find) of
arch in the codebase that need adjusting.
The new Arch type is an Enum, making it clear what architectures can be
represented and making it much easier to reason about. Since we support
~5 (kinda) different representations of an Architecture (Alpine, Kernel,
target triple, platform, and QEMU), we now formalise that the Alpine
format is what we represent internally, with methods to convert to any
of the others as-needed.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Use a different binary repo depending on the source repository for the
package. This makes it possible to split out systemd packages into their
own repository.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Cease merging pmbootstrap.cfg into args, implement a Context type to let
us pull globals out of thin air (as an intermediate workaround) and rip
args out of a lot of the codebase.
This is just a first pass, after this we can split all the state that
leaked over into Context into types with narrower scopes (like a
BuildContext(), etc).
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Defaulting to the native chroot isn't necessarily intuitive. Let's
require this be specified in full.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
We use a custom verbose log level in pmbootstrap, unfortunately it isn't
possible to correctly type this due to some limitations in the logging
library [1], [2].
Given that our usecase is fairly simple, we can just wrap the module
with our own so we only have to tell mypy to ignore the error once
instead of at every callsite.
[1]: https://github.com/cryptax/droidlysis/issues/15
[2]: https://github.com/python/typing/discussions/980
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Introduce a new module: pmb.core to contain explicitly typed pmbootstrap
API. The first component being Suffix and SuffixType. This explicitly
defines what suffixes are possible, future changes should aim to further
constrain this API (e.g. by validating against available device
codenames or architectures for buildroot suffixes).
Additionally, migrate the entire codebase over to using pathlib.Path.
This is a relatively new part of the Python standard library that uses a
more object oriented model for path handling. It also uses strong type
hinting and has other features that make it much cleaner and easier to
work with than pure f-strings. The Chroot class overloads the "/"
operator the same way the Path object does, allowing one to write paths
relative to a given chroot as:
builddir = chroot / "home/pmos/build"
The Chroot class also has a string representation ("native", or
"rootfs_valve-jupiter"), and a .path property for directly accessing the
absolute path (as a Path object).
The general idea here is to encapsulate common patterns into type hinted
code, and gradually reduce the amount of assumptions made around the
codebase so that future changes are easier to implement.
As the chroot suffixes are now part of the Chroot class, we also
implement validation for them, this encodes the rules on suffix naming
and will cause a runtime exception if a suffix doesn't follow the rules.
Make the handling of the custom NonBugError and BuildFailedError
exceptions more consistent with the handling of other exceptions, by
printing "ERROR: " infront of the actual error text. Then we don't need
to duplicate that where we raise the errors. pmbootstrap prints "ERROR"
in red.
Make sure that we build depends of subpackages, to avoid errors like the
following:
* postmarketos-base has a subpackage postmarketos-base-nftables, which
depends on postmarketos-config-nftables
* when pmbootstrap builds postmarketos-base with all its dependencies,
it did not realize it depends on postmarketos-config-nftables through
the subpackage, but it built it anyway
* when trying to install postmarketos-base-nftables, apk complains that
there is no postmarketos-config-nftables:
ERROR: unable to select packages:
postmarketos-config-nftables (no such package):
required by: postmarketos-base-nftables-32-r2[postmarketos-config-nftables]
This also adds the missing depenendencies to "pmbootstrap repo_missing",
which bpo uses to resolve dependencies.
Fixes: issue 2084
When running build --force, if a later package is a dependency of an
earlier package, it will be "visited" by the dependency resolver,
erroneously causing it to be skipped when later in the session it's
visited to force build. This is because previously visited packages are
marked as such in a hashmap, but skip_already_built() assumes that a
package which has been visited would have already been built.
Fix this by overriding skip_already_built() if doing a force build. This
works because package dependencies are only built if the APKBUILD
version is newer than the binary repo, even when --force is specified.
So there is no risk of an infinite loop here.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Cache the compiler output of rust code with sccache, like we use ccache
for c code.
I've considered using sccache to completely replace ccache since it can
cache output of C/C++ code too. But let's not do it for now since ccache
doesn't need to run a daemon in the background that needs to be stopped
when shutting down / zapping. Also it would need more refactoring.
Reviewed-by: Luca Weiss <luca@z3ntu.xyz>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230806184729.4891-5-ollieparanoid@postmarketos.org%3E
With this code path, pmbootstrap would start a distccd + sshd in the
native chroot, and configure it so it runs the cross compiler. The
foreign arch chroots would then call this cross compiler from localhost
by calling the distcc client instead of gcc.
This code has been obsoleted by the much simpler crossdirect in 2019.
Let's finally remove it.
Fixes: issue 2179
Reviewed-by: Luca Weiss <luca@z3ntu.xyz>
Reviewed-by: Clayton Craft <clayton@craftyguy.net>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230613161437.570196-4-ollieparanoid@postmarketos.org%3E
Always set GOCACHE during build and let it point to pmbootstrap's work
dir. This has a similar effect as using ccache for C/C++.
Set GOMODCACHE conditionally: this is for caching the go modules (git
repositories) that get cloned during a build if they are not bundled
with the source. Usually APKBUILDs should cache them, but when using
pmbootstrap build --src, they would get downloaded over and over again.
Set GOMODCACHE automatically for --src and allow enabling/disabling this
manually with new --go-mod-cache / --no-go-mod-cache.
This speeds up multiple iterations of building the same package
significantly. I'm using it for:
$ pmbootstrap build postmarketos-mkinitfs --src=...
I've verified that using the same GOCACHE dir for multiple architectures
doesn't result in build artifacts for the wrong arch getting used.
Reviewed-by: Clayton Craft <clayton@craftyguy.net>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230223064743.1773-1-ollieparanoid@postmarketos.org%3E
The plan was to drop this workaround, once alpine releases a new gcc
version, and we regenerate our cross compilers against that version
_and_ drop options="!tracedeps", as latest pmbootstrap aportgen would
do. However !tracedeps wasn't dropped in the latest upgrade. This was
too easy to miss, I'll add a comment to the gcc-cross APKBUILDs so we
see it next time. Let's just update the workaround for this version.
