Defaulting to the native chroot isn't necessarily intuitive. Let's
require this be specified in full.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
We use a custom verbose log level in pmbootstrap, unfortunately it isn't
possible to correctly type this due to some limitations in the logging
library [1], [2].
Given that our usecase is fairly simple, we can just wrap the module
with our own so we only have to tell mypy to ignore the error once
instead of at every callsite.
[1]: https://github.com/cryptax/droidlysis/issues/15
[2]: https://github.com/python/typing/discussions/980
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Introduce a new module: pmb.core to contain explicitly typed pmbootstrap
API. The first component being Suffix and SuffixType. This explicitly
defines what suffixes are possible, future changes should aim to further
constrain this API (e.g. by validating against available device
codenames or architectures for buildroot suffixes).
Additionally, migrate the entire codebase over to using pathlib.Path.
This is a relatively new part of the Python standard library that uses a
more object oriented model for path handling. It also uses strong type
hinting and has other features that make it much cleaner and easier to
work with than pure f-strings. The Chroot class overloads the "/"
operator the same way the Path object does, allowing one to write paths
relative to a given chroot as:
builddir = chroot / "home/pmos/build"
The Chroot class also has a string representation ("native", or
"rootfs_valve-jupiter"), and a .path property for directly accessing the
absolute path (as a Path object).
The general idea here is to encapsulate common patterns into type hinted
code, and gradually reduce the amount of assumptions made around the
codebase so that future changes are easier to implement.
As the chroot suffixes are now part of the Chroot class, we also
implement validation for them, this encodes the rules on suffix naming
and will cause a runtime exception if a suffix doesn't follow the rules.
Make the handling of the custom NonBugError and BuildFailedError
exceptions more consistent with the handling of other exceptions, by
printing "ERROR: " infront of the actual error text. Then we don't need
to duplicate that where we raise the errors. pmbootstrap prints "ERROR"
in red.
Make sure that we build depends of subpackages, to avoid errors like the
following:
* postmarketos-base has a subpackage postmarketos-base-nftables, which
depends on postmarketos-config-nftables
* when pmbootstrap builds postmarketos-base with all its dependencies,
it did not realize it depends on postmarketos-config-nftables through
the subpackage, but it built it anyway
* when trying to install postmarketos-base-nftables, apk complains that
there is no postmarketos-config-nftables:
ERROR: unable to select packages:
postmarketos-config-nftables (no such package):
required by: postmarketos-base-nftables-32-r2[postmarketos-config-nftables]
This also adds the missing depenendencies to "pmbootstrap repo_missing",
which bpo uses to resolve dependencies.
Fixes: issue 2084
When running build --force, if a later package is a dependency of an
earlier package, it will be "visited" by the dependency resolver,
erroneously causing it to be skipped when later in the session it's
visited to force build. This is because previously visited packages are
marked as such in a hashmap, but skip_already_built() assumes that a
package which has been visited would have already been built.
Fix this by overriding skip_already_built() if doing a force build. This
works because package dependencies are only built if the APKBUILD
version is newer than the binary repo, even when --force is specified.
So there is no risk of an infinite loop here.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Cache the compiler output of rust code with sccache, like we use ccache
for c code.
I've considered using sccache to completely replace ccache since it can
cache output of C/C++ code too. But let's not do it for now since ccache
doesn't need to run a daemon in the background that needs to be stopped
when shutting down / zapping. Also it would need more refactoring.
Reviewed-by: Luca Weiss <luca@z3ntu.xyz>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230806184729.4891-5-ollieparanoid@postmarketos.org%3E
With this code path, pmbootstrap would start a distccd + sshd in the
native chroot, and configure it so it runs the cross compiler. The
foreign arch chroots would then call this cross compiler from localhost
by calling the distcc client instead of gcc.
This code has been obsoleted by the much simpler crossdirect in 2019.
Let's finally remove it.
Fixes: issue 2179
Reviewed-by: Luca Weiss <luca@z3ntu.xyz>
Reviewed-by: Clayton Craft <clayton@craftyguy.net>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230613161437.570196-4-ollieparanoid@postmarketos.org%3E
Always set GOCACHE during build and let it point to pmbootstrap's work
dir. This has a similar effect as using ccache for C/C++.
Set GOMODCACHE conditionally: this is for caching the go modules (git
repositories) that get cloned during a build if they are not bundled
with the source. Usually APKBUILDs should cache them, but when using
pmbootstrap build --src, they would get downloaded over and over again.
Set GOMODCACHE automatically for --src and allow enabling/disabling this
manually with new --go-mod-cache / --no-go-mod-cache.
This speeds up multiple iterations of building the same package
significantly. I'm using it for:
$ pmbootstrap build postmarketos-mkinitfs --src=...
I've verified that using the same GOCACHE dir for multiple architectures
doesn't result in build artifacts for the wrong arch getting used.
Reviewed-by: Clayton Craft <clayton@craftyguy.net>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230223064743.1773-1-ollieparanoid@postmarketos.org%3E
The plan was to drop this workaround, once alpine releases a new gcc
version, and we regenerate our cross compilers against that version
_and_ drop options="!tracedeps", as latest pmbootstrap aportgen would
do. However !tracedeps wasn't dropped in the latest upgrade. This was
too easy to miss, I'll add a comment to the gcc-cross APKBUILDs so we
see it next time. Let's just update the workaround for this version.
Install isl25 for the specific gcc versions we currently have in
pmaports master that are missing it; this is easier than building new
packages and going out of sync with Alpine's versioning. This can be
reverted as soon as there is a newer gcc.
Related: pmaports issue 1732
Replace "args.cache" with a global variable in order to
avoid passing "args" to all functions. This is a step to get rid of this
args-passed-to-all-functions pattern in pmbootstrap.
Replace "args.arch_native" with the direct function call in order to
avoid passing "args" to all functions. This is a step to get rid of this
args-passed-to-all-functions pattern in pmbootstrap.
Fix the issue of having the postmarketOS binary repository key deleted
from the chroots after "abuild undeps" removes the postmarketos-keys
package. This happens if e.g. building two device packages in a row (as
they depend on postmarketos-base, which depends on postmarketos-keys in
current pmaports.git master).
I've also considered installing the postmarketos-keys next to
alpine-base in new chroots. But this would introduce a bootstrap
problem, since you can't install the postmarketos-keys package unless
it already exists in the repository. We'd run into that when building
the next release.
Before this commit, package folders were copied into the chroot one by
one in order to run apkbuild-lint on them. This logic is replaced by
mounting pmaports.git into the chroot and using a single apkbuild-lint
invocation to lint the supplied packages.
Both of these changes result in a performance improvement, especially
when linting multiple packages at once.
Before this change:
$ time ./pmbootstrap.py -q lint $(cd ../pmaports/cross; echo *) \
> /dev/null
real 0m5,261s
user 0m7,046s
sys 0m1,842s
Using the pmaports.git mount but calling apkbuild-lint in a loop:
$ time ./pmbootstrap.py -q lint $(cd ../pmaports/cross; echo *) \
> /dev/null
real 0m4,089s
user 0m6,418s
sys 0m1,219s
After this change:
$ time ./pmbootstrap.py -q lint $(cd ../pmaports/cross; echo *) \
> /dev/null
real 0m3,518s
user 0m5,968s
sys 0m0,959s
Additionally, running apkbuild-lint from the pmaports.git mount point
has the benefit that every printed violation contains a nice source
identifier à la "./cross/grub-x86/APKBUILD". This makes it possible to
differentiate between different packages even though only a single
apkbuild-lint invocation is used.
Relates: postmarketOS/pmaports#564
Let warnings like the following not get displayed in the regular
pmbootstrap output anymore, only in 'pmbootstrap log' if -v / --verbose
was used. This message informs the user that a package's dependencies
are newer than the package itself. But the WARNING makes it sound like
this is something to be concerned about, whereas in reality this is
fine. In this example, postmarketos-mkinitfs has gotten a new feature /
fix after postmarketos-base and there's no need to rebuild
postmarketos-base.
[18:02:59] WARNING: postmarketos-base depends on rebuilt package(s)
postmarketos-mkinitfs (use 'pmbootstrap build postmarketos-base --force'
if necessary!)
[skip ci] already built in CI, change is trivial
The sideload command runs the supplied names through the pmbootstrap
buildsystem to make sure they're up-to-date, then uses scp from the host
to copy the built apks to /tmp on the phone and installs them through
ssh.
If the --install-key option is set then it will also copy over the apk
key that's used for signing the packages built by pmbootstrap in case
the postmarketOS install on the device isn't build by the same machine
as you're sideloading from.
Migrate to workdir version 5 and move already built packages into the edge
channel subdir, for example:
$WORK/packages/x86_64/hello-world-1-r5.apk
to:
$WORK/packages/edge/x86_64/hello-world-1-r5.apk
The build.postmarketos.org code has already been adjusted to find built
packages in either directory structure.
The --no-depends option is supposed to stop pmbootstrap if it was
instructed to build a package, but a dependency must be built first. So
far, this only covers the case if there is no binary package for a dependency.
Make it stop if the binary package exists, but is outdated, too.
Fixes: #1895
While at it, also remove unnecessary "#!/usr/bin/env python3" in files
that only get imported, and adjust other empty/comment lines in the
beginnings of the files for consistency.
This makes files easier to read, and makes the pmbootstrap codebase more
consistent with the build.postmarketos.org codebase.
At the moment we have a simple subpkgdesc() function that can only
parse "pkgdesc" from subpackages, without support for any variables.
But we have a quite nice variable parser now that can be extended
to work for subpackages.
Simply put this works by:
- Finding the lines that belong to the subpackage function
- Stripping indentation (tab)
- Parsing relevant attributes similar to the apkbuild() function
The "subpackages" in the parsed APKBUILD are replaced by a dict
of subpkgname: {"pkgdesc": "...", "depends": "..."} which are
parsed from the subpackage function (if found).
This makes it possible to get the "depends" of a subpackage.
Install rust (rustc) in native chroot, if it is in the APKBUILD's
build dependencies. Add a test to verify that crossdirect + rust works
as expected.
Closes: #1861
Make /home/pmos/build/.git point to the .git dir from pmaports.git, with
a symlink so abuild does not fail.
abuild expects the current working directory to be a subdirectory of a
cloned git repository (e.g. main/openrc from aports.git). If git is
installed, it will try to get the last git commit from that repository,
and place it in the resulting apk (.PKGINFO) as well as use the date
from that commit as SOURCE_DATE_EPOCH (for reproducible builds).
With that symlink, we actually make it use the last git commit from
pmaports.git for SOURCE_DATE_EPOCH and have that in the resulting apk's
.PKGINFO.
Fixes: #1841
When 'pmbootstrap build' is started with '--no-depends', we make sure
that binary packages of all dependencies exist before proceeding with a
build. Make sure that we have downloaded the APKINDEX for the given arch
first, otherwise this may complain that a binary package is missing
although it does exist.
This happened when using the "native" cross compile method, e.g. when
building kernels for different architectures. It would complain that
there is no binary package for "bash" (which is in Alpine obviously):
https://builds.sr.ht/~postmarketos/job/103882
Packages can add pmb:strict to their options to enable the --strict
mode. This can be used if packages need to get build in a clean chroot
or their make dependencies need to get removed.
Depend on new pmaports version, where crossdirect uses the native ccache
binary instead of going through the foreign arch ccache first and then
going through crossdirect.
Old:
ccache (foreign) -> crossdirect (native) -> gcc (native)
New:
crossdirect (native) -> ccache (native) -> gcc (native)
Set the PATH to the crossdirect binaries, and don't set CCACHE_PATH or
CCACHE_COMPILERCHECK from pmbootstrap anymore. crossdirect sets the
CCACHE_PATH to /native/usr/bin now, along with all other required
environment variables. CCACHE_COMPILERCHECK isn't necessary anymore,
because ccache will call gcc directly and therefore be able to use the
file's timestamp and size directly. Also passing that would not work
with the current crossdirect package.
Launch native cross compilers inside foreign chroot. Enable by default,
but allow disabling with --no-crossdirect for now. This option and the
distcc-sshd related code will be removed in the future.
Aborts the build if any dependencies would have to be build first. This
is useful for build.postmarketos.org, because we want to build exactly
one package in one build job. If dependencies would need to be built, we
made a mistake earlier, and not aborting the build makes it harder to
find that orginal mistake.
When a pmaport can not be built for the desired architecture, fall back
to the binary package (from postmarketOS or Alpine) if it exists. This
allows us to provide an updated version of mesa for arm arches, but
using Alpine's mesa package on x86* arches.
