This might have saved me a couple of minutes of debugging, at least.
While working with the aportgen code I had two distinct issues:
1. APKINDEX was outdated, so I got an error about not being able to
download the supposedly latest version of BusyBox as it no longer
existed on Alpine's mirror.
2. APKINDEX had not been created by any other command, so aportgen
would tell me that it couldn't find the "busybox" package.
The changes in this commit resolves both of these issues from my
testing.
These were almost identical across the Busybox, GRUB EFI, and musl
generators. Unify them by creating functions they all (and any future
aportgen generators) can call instead of having to maintain three
separate copies of this code.
This can otherwise fail if the package was only built for some
architectures so far, but not for the architecture that is being generated.
E.g. right now musl-1.2.5-r6 was built already for x86_64 but not aarch64,
which causes errors like the following when running "pmbootstrap aportgen
musl-x86_64" on aarch64:
>>> musl-x86_64: Fetching musl-1.2.5-r5-x86_64-edge.apk::http://dl-4.alpinelinux.org/alpine//edge/main/x86_64/musl-1.2.5-r5.apk
Connecting to dl-4.alpinelinux.org (147.75.40.42:80)
Connecting to dl-4.alpinelinux.org (147.75.40.42:443)
wget: server returned error: HTTP/1.1 404 Not Found
>>> ERROR: musl-x86_64: checksum failed
Cease merging pmbootstrap.cfg into args, implement a Context type to let
us pull globals out of thin air (as an intermediate workaround) and rip
args out of a lot of the codebase.
This is just a first pass, after this we can split all the state that
leaked over into Context into types with narrower scopes (like a
BuildContext(), etc).
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
With the new chroot type, we can now write fancy paths in the pythonic
way. Convert most of the codebase over, as well as adding various other
type hints.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
We use a custom verbose log level in pmbootstrap, unfortunately it isn't
possible to correctly type this due to some limitations in the logging
library [1], [2].
Given that our usecase is fairly simple, we can just wrap the module
with our own so we only have to tell mypy to ignore the error once
instead of at every callsite.
[1]: https://github.com/cryptax/droidlysis/issues/15
[2]: https://github.com/python/typing/discussions/980
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Introduce a new module: pmb.core to contain explicitly typed pmbootstrap
API. The first component being Suffix and SuffixType. This explicitly
defines what suffixes are possible, future changes should aim to further
constrain this API (e.g. by validating against available device
codenames or architectures for buildroot suffixes).
Additionally, migrate the entire codebase over to using pathlib.Path.
This is a relatively new part of the Python standard library that uses a
more object oriented model for path handling. It also uses strong type
hinting and has other features that make it much cleaner and easier to
work with than pure f-strings. The Chroot class overloads the "/"
operator the same way the Path object does, allowing one to write paths
relative to a given chroot as:
builddir = chroot / "home/pmos/build"
The Chroot class also has a string representation ("native", or
"rootfs_valve-jupiter"), and a .path property for directly accessing the
absolute path (as a Path object).
The general idea here is to encapsulate common patterns into type hinted
code, and gradually reduce the amount of assumptions made around the
codebase so that future changes are easier to implement.
As the chroot suffixes are now part of the Chroot class, we also
implement validation for them, this encodes the rules on suffix naming
and will cause a runtime exception if a suffix doesn't follow the rules.
Often I find myself confused and subsequently frustrated by the
behaviour of --fork-alpine as usually I apply whatever changes I want to
the aports package before forking. This is due to that in the proper
aports repo I can leverage the full power of git and apply patches or
revert commits, whereas after the package has been forked to pmaports
the context git needs to do this is lost.
To fix that, this commit introduces --fork-alpine-retain-branch which
works the same way as --fork-alpine except it doesn't change the aports
branch to match the current channel.
Sometimes I want to build an older version of a package from Alpine, and
since package upgrades can involve things like patches and other
externalities just changing the pkgver and running checksum in pmaports
may not be enough. As such, it tends to be easier to revert the change
in the local aports repo and then fork than forking and then trying to
manually revert the changes yourself (since you can't have git do that
for you given that they are distinct repositories).
Prior to this patch, that was not possible since pmbootstrap would
assume older aport version equals outdated aports in general and as such
cancel the whole operation. Instead, just print a warning and helpful
information to make this workflow possible while also warning users that
they may want to update their local aports.
Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20231209111813.37756-1-newbyte@postmarketos.org%3E
The typical workflow for upgrading cross/gcc-* is:
$ cd pmaports/cross
$ pmbootstrap aportgen gcc-*
Currently this is failing because the APKBUILD has been updated in
Alpine for gcc, but gcc has not been built for all arches yet. This
shouldn't prevent us from generating the proper updated APKBUILDs in
pmaports so just print a note and don't fail here.
I'm committing this directly to master as this currently breaks
test/test_aportgen.py::test_aportgen.
Previously it always used native (x84_64) arch for apkindex
version check.
As pmb.aportgen.core.alpine_apkindex_path() already has arch
parameter, use it to look up correct APKINDEX.
Also fixup test.
Checkout the aports.git branch for the current channel (e.g.
3.12-stable), before trying to find the APKBUILD.
I had tried to auto-unshallow the git repository earlier, but then the
tags were missing. I decided that it's not worth to provide a migration
path: "pmbootstrap aportgen" is only used by few advanced users (to
maintain the pmaports repo).
While at it, also remove unnecessary "#!/usr/bin/env python3" in files
that only get imported, and adjust other empty/comment lines in the
beginnings of the files for consistency.
This makes files easier to read, and makes the pmbootstrap codebase more
consistent with the build.postmarketos.org codebase.
Do not install git in the native chroot and use it from there. Remove the
chown_to_user argument from pmb.helpers.git.clone(), the resulting dir
is now always owned by the user. While at it, refactor the function and
display the clone URL.
Previously we had cloned aports_upstream (from Alpine) with
chown_to_user=False (legacy) and pmaports with chown_to_user=True.
pmb.helpers.git.rev_parse() would only work after chown_to_user=True.
Check if git is installed in "pmbootstrap init", and remove the same
check from rev_parse(). Add a new work dir version, that checks for git
and changes ownership of already checked out aports_upstream to the
host system's user.
When creating a new work dir, create cache_git instead of cache_http.
cache_http is created on demand already, with proper permissions. But
cache_git must be created, otherwise pmb.helpers.mount.bind will create it
as root.
This is in preparation for the "pmbootstrap pull" feature, as it allows
using the host system's git in all new code paths. We will be able to
handle repositories even if they were cloned outside of the work dir
(which we do in a few CI scripts for example).
Related: #1858
This MR add the --fork-alpine argument to the pmboostrap aportgen
command, which downloads the APKBUILD and related files and copies them
into the pmaports/temp folder.
APKBUILD should be indented with tabs not spaces.
This also resolves a bug which the replace_functions feature was always
trimming off the beginning of a line.
For example:
replace_functions = {"build": "return 0"}
would be formatted as "rn 0".
Due to changes in abuild, our `gcc-armhf` etc. packages did not build
when using strict mode (i.e. `pmbootstrap build --strict gcc-armhf`)
anymore.
Changes:
* Set `CBUILDDIR=/`, so apk can read a valid package index from there
* Directly set `_cross_configure`, so it does not use CBUILDDIR anymore
* Set `BOOTSTRAP="nobuildbase"` to prevent apk from installing
`build-base-armhf` etc. (these don't exist in pmOS)
* Remove legacy code for lazy reproducible builds that wrapped
`package()`
In order to get cross-compilers, we generate a few aports (e.g.
binutils-armhf, gcc-armhf) automatically from Alpine's aports.
pmbootstrap was already able to perform a git checkout of Alpine's
aports repository. But it needed to be manually updated. Otherwise
the `pmbootstrap aportgen` command could actually downgrade the aport
instead of updating it to the current version.
After thinking about adding a dedicated pmbootstrap command for
updating git repositories, I thought it would be better to not open
that can of worms (pmbootstrap as general git wrapper? no thanks).
The solution implemented here compares the upstream aport version of
the git checkout of a certain package (e.g. gcc for gcc-armhf) with the
version in Alpine's binary package APKINDEX. When the aport version is
lower than the binary package version, it shows the user how to update
the git repository with just one command:
pmbootstrap chroot --add=git --user -- \
git -C /mnt/pmbootstrap-git/aports_upstream pull
Changes:
* `pmb.aportgen.core.get_upstream_aport()`: new function, that returns
the absolute path to the upstream aport on disk, after checking the
version of the aport against the binary package.
* Use that new function in pmb.aportgen.gcc and pmb.aportgen.binutils
* New function `pmb.helpers.repo.alpine_apkindex_path()`: updates the
APKINDEX if necessary and returns the absolute path to the APKINDEX.
This code was basically present already, but not as function, so now
we have a bit less overhead there.
* `pmbootstrap chroot`: new `--user` argument
* `pmb.parse.apkbuild`: make pkgname check optional, as it fails with
the official gcc APKBUILD before we modify it (the current APKBUILD
parser is not meant to be perfect, as this would require a full shell
parsing implementation).
* Extended `test_aportgen.py` and enabled it by default in
`testcases_fast.sh`. Previously it was disabled due to traffic
concerns (cloning the aports repo, but then again we do a full KDE
plasma mobile installation in Travis now, so that shouldn't matter
too much).
* `testcases_fast.sh`: With "test_aport_in_sync_with_git" removed
from the disabled-by-default list (left over from timestamp based
rebuilds), there were no more test cases disabled by default. I've
changed it, so now the qemu_running_processes test case is disabled,
and added an `--all` parameter to the script to disable no test
cases. Travis runs with the `--all` parameter while it's useful to
do a quick local test without `--all` in roughly 2 minutes instead of
10.
* `aports/cross/binutils-*`: Fix `_mirror` variable to point to current
default Alpine mirror (so the aportgen testcase runs through).
