If the user tries to build a package, but it can't be found: check if
systemd is disabled and the package is in extra-repos/systemd, display a
hint about it.
$ pmbootstrap build systemd
[17:33:53] NOTE: The package 'systemd' exists in extra-repos/systemd, but systemd is currently disabled
[17:33:53] ERROR: Could not find aport for package: systemd
Fixes: issue 2398
Replace the boolean skip_extra_repos with a new with_extra_repos
argument that can be default, enabled or disabled.
This will be used to explicitly enable extra repos in a package search
even if systemd is currently disabled, so we can display a hint when a
package was not found because systemd is disabled in a follow-up patch.
There is a lot of context and state management needed when using
pmbootstrap, sometimes it can be a lot to keep in your head. Let's print
the output of "pmbootstrap status" when stuff goes wrong, this might
help remind people if e.g. their pmaports checkout is on the wrong
branch, or they're building for the wrong device.
Additionally, don't print the "run pmbootstrap log for details" message
if pmbootstrap was called with --details-to-stdout
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
There was an oversight when this API was originally created and it
implicitly assumed that the pmaports repository was always named
"pmaports". This unfortunately broke some peoples workflows.
Introduce a new "pkgrepo_name()" function and adjust the codebase to use
it, as well as adjusting pkgrepo internally to special case the
"pmaports" repo so that it's always named pmaports no matter what the
directory itself is named.
This is probably more complexity than we should be dealing with here, we
should probably create a new type to encode this behaviour.
Fixes: #2412
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
This was always a hack, and it looks like there (hopefully) aren't any
placs where we still need to handle this.
Possibly expecting regressions... But then we have something to write a
test for.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
When running "apk add" we must always have --cache-dir set! We also
expect --no-interactive to be set.
Add some asserts so we don't regress here.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Since parse.deviceinfo() returns a custom class now, it can't be
trivially serialized as JSON.
This has been broken for months now so it's clear nobody is using this.
Let's take the chance to get rid of it.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
In 0b4fb9119f (chroot: always run apk static v2 (MR 2423)) we adjusted
install_run_apk() to run apk static on the host and pass in the local
binary repo with "--repository". This function can call apk in two ways,
either with the progress bar handling or without, the second case was
never updated and still ran apk inside the chroot incorrectly and with
an incorrect --repository flag.
Let's finish the job by refactoring helpers/apk.py to support all our
usecases and pointing everything to it, removing the last few situations
where we call "pmb.chroot.root(["apk", ...]).
The apk_with_progress() function is replaced by a generic "run()"
function which takes a boolean to indicate if we should render apk
progress.
Additionally, a new cache_clean() function is added so that "pmbootstrap
zap --pkgs-online-mismatch" can FINALLY be refactored to not rely on a
chroot existing. This requires some hacks but nothing serious, see the
comments in the function for details.
The chroot.init() code is now simplified since handling the --root,
--arch, --cache-dir, and --repository flags is now all done by
apk._prepare_cmd() as and when appropriate.
Lastly, this fixes a (previously unnoticed) bug where apk.static was
actually using /var/cache/apk on your host machine for its cache... This
is definitely not good behaviour....
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
This function better belongs here, especially as it will be used outside
of the context of a chroot() soon.
Additionally, it's adjusted to take the rootfs path as its first
argument rather than a chroot, since it could operate on any rootfs.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Lazy load pmb.config.styles and move the pmb.__version__ print elsewhere
so the logging module is (closer to) a standalone entity. This is
necessary to be able to import it in pmb/helpers/apk.py otherwise we get
a cyclical dependency.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
I think the "these types make no sense" comment were about
output_return_buffer, which defaulted to False but was to be set to
list[bytes] if used. I changed it so that it defaults to None, which
probably is more conventional. Other than that I didn't notice anything
weird about the types here.
Also check for None as necessary to appease mypy.
Flush the progress bar displayed by the function before logging a
warning message for 404 not found responses from the server. Without
flushing, this message gets lost.
In addition to the warning, display a NOTE and ERROR at the end to
explain what went wrong:
[20:16:50] Update package index for x86_64 (4 file(s))
[20:16:51] WARNING: file not found: http://mirror.postmarketos.org/postmarketos_get_404_test/edge/main/x86_64/APKINDEX.tar.gz
[20:16:51] NOTE: check the [mirrors] section in 'pmbootstrap config'
[20:16:51] ERROR: getting APKINDEX from binary package mirror failed!
Without handling this properly, pmbootstrap would fail slightly later
with a much less useful error:
[15:14:28] ERROR: expected str, bytes or os.PathLike object, not NoneType
Fixes: pmbootstrap issue 2424
Don't update the git URL again when coming from 7-2.x where this step
was already done. This didn't fail, but was potentially confusing for
the user since the git URLs were already upgraded from 7 -> 7-2x.
All call sites that use this have been removed. This was only introduced
as a stopgap until we could refactor more code to not use args
everywhere, and that has now been achived. As such, remove this so it
doesn't get used in more locations.
For the gitlab.com -> gitlab.postmarketos.org migration we needed to
also migrate the workdir in pmb v2 since pmb v3 is not ready yet.
Therefore we need to handle that "7-2.x" workdir version and migrate
correctly when switching to pmbootstrap v3.
See https://gitlab.postmarketos.org/postmarketOS/pmbootstrap/-/merge_requests/2445
* Make sure all Python modules are documented by adding a bit of shell
to .ci/docs.sh
* Remove non-existing module references from .rst
* Fix various warnings from sphinx by adjusting Python docstrings
* Add class member docs to ApkindexBlock
I'm not sure why this was annotated as a Sequence rather than list. The
function clearly returns a list, not a Sequence, and the commit that
originally did it[1] provides no justification. This causes problems
with mypy checks in other places, so let's just annotate this as
returning a list.
[1]: 198f302a36
Now that we have moved the kconfigcheck configuration into pmaports
branches via kconfigcheck.toml, it is time to get rid of the hardcoded
list of valid "pmb:kconfigcheck-…" options for APKBUILDs. Generate it
from the kconfigcheck.toml of the current branch, too.
These options are passed from "pmbootstrap lint" to "apkbuild-lint",
which we run in pmaports CI.
The output of apkbuild-lint is hard to spot among other log messages
from pmbootstrap, because it is not colorized.
Add "*** apkbuild-lint output ***"" before and after the output of
apkbuild-lint, so we get a standing out message in green around the
apkbuild-lint output.
It is not needed to run the linter as root, so don't do it. This is
probably a leftover from early days pmbootstrap, where pmaports.git
wasn't chowned by the user running pmbootstrap.
When running some commands like "pmbootstrap chroot" or even
"pmbootstrap work_migrate", pmbootstrap will attempt to read the
pmaports.cfg from origin/master *before* doing the work dir migration.
In order to do this, the "get_upstream_remote" function tries to find
the upstream remote by URL. Let it search with the outdated URLs too, so
this doesn't fail right before migrating to the new URLs.
This could be re-purposed in the future in case we migrate URLs again.
Co-authored-by: Caleb Connolly <caleb@postmarketos.org>
Tweaked-by: Oliver Smith <ollieparanoid@postmarketos.org>
Now that we don't need weird apk-tools hacks for systemd, we can
re-implement this optimisation and always run apk static rather than
running apk through the chroot.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
For packages selected by the user (see last commit), set the
provider_priority to 999999 instead of 999. The previous number seems
not high enough, we might set this in the APKBUILDs.
This function is used by "pmbootstrap repo_missing", which is only used
by bpo. In order to support building the split repository in bpo, this
patch is required.
Return "systemd" for the packages that are in extra-repos/systemd, and
None for all other packages (indicating that they don't get split into a
separate repository).
Checking the parent-parent dir of the APKBUILD is a bit fragile
(assuming we don't have subdirs in the systemd repository), but that
assumption was made with the previous implementation as well (would only
return "systemd" in that case). I don't see a better solution with
reasonable effort right now, given that we also support multiple
pmaports dirs. We can improve this in the future, meanwhile let's just
not use subdirs in the systemd repository.
Don't attempt to find packages of other arches in repo_missing code.
This leads to unexpected results in general (we want to check if a
binary package exists for only one specific architecture!) - and in case
of master_staging_systemd, it current causes errors because pmbootstrap
tries to fetch armhf, riscv etc. APKINDEXes for which we don't build the
staging repositories.
As chroot.glob() now returns a generator, convert it to a list first to
check if it is empty. This may not be the most efficient method, but
since we expect it to be empty here it is fine.
Without this patch, it always shows the error even if there are no
locally built packages:
ERROR: --no-local-pkgs specified, but locally built packages found. Consider 'pmbootstrap zap -p' to delete them.
When a dependency cannot be found in source or binary packages of the
selected architecture, then don't look for binary packages in other
architectures package indexes.
This leads to a confusing error down the line when building packages for
extra_repos_systemd (and e.g. stable branches), because we don't have
binary packages for all arches that pmbootstrap supports, and so it
tries to fetch a non-existing APKINDEX.
Instead, print a nice error about a non-existing dependency.
