Add module that provides utilities and information related to localization.
Initial implementation includes code to generate keyboard configs
for locale set by user.
Do not abort if one or more APKINDEX files cannot be downloaded, if
PMB_APK_FORCE_MISSING_REPOSITORIES is set. This is needed when
bootstrapping new stable branches. The same environment variable is
already used in pmbootstrap code to pass --force-missing-repositories to
apk in pmb.chroot.apk.install_run_apk().
Previously, it would only fall back to parsing all APKBUILDs if a main
package was guessed for this potential subpackage. We do however want to
scan all APKBUILDs in case the package is provided by one of the
packages in pmaports.
Signed-off-by: Jens Reidel <adrian@travitia.xyz>
g++ is a subpackage of gcc, but cannot be detected as such easily by
pmbootstrap. This is because g++ is not gcc-g++ or any other variant of
such package names. Similarly, the detection for musl-dev-ppc64le and
other architectures is broken, since the -dev suffix detection does not
work if there is an architecture suffix.
To fix it, add special case handling for cross toolchain and packages
and have that fixup the -dev cases and hardcode g++ as a subpackage of
gcc.
To reproduce:
- Generate ppc64le cross packages:
pmbootstrap aportgen gcc-ppc64le musl-ppc64le
- Build a package (to trigger building cross compilers):
pmbootstrap build hello-world --arch=ppc64le
Signed-off-by: Jens Reidel <adrian@travitia.xyz>
After extending CI to always run "pmbootstrap zap -a" it revealed
another bug: we did not ensure at this point that apk.static is
available. Rung apk_static.init() to ensure it gets downloaded and
extracted at this point if it is missing.
The tmp dir is owned by root, when it gets created earlier by apk
running as root to store the apk progress fifo. Use another directory to
work around this for now (getting close to the 3.0.0 release), we can
rework the apk progress fifo later on if we want.
Fixes: issue 2491
Replace aports -> pmaports in these messages.
$ pmbootstrap build systemd
[17:37:02] NOTE: The package 'systemd' exists in extra-repos/systemd, but systemd is currently disabled
[17:37:02] ERROR: Could not find package 'systemd' in pmaports
If the user tries to build a package, but it can't be found: check if
systemd is disabled and the package is in extra-repos/systemd, display a
hint about it.
$ pmbootstrap build systemd
[17:33:53] NOTE: The package 'systemd' exists in extra-repos/systemd, but systemd is currently disabled
[17:33:53] ERROR: Could not find aport for package: systemd
Fixes: issue 2398
Replace the boolean skip_extra_repos with a new with_extra_repos
argument that can be default, enabled or disabled.
This will be used to explicitly enable extra repos in a package search
even if systemd is currently disabled, so we can display a hint when a
package was not found because systemd is disabled in a follow-up patch.
There is a lot of context and state management needed when using
pmbootstrap, sometimes it can be a lot to keep in your head. Let's print
the output of "pmbootstrap status" when stuff goes wrong, this might
help remind people if e.g. their pmaports checkout is on the wrong
branch, or they're building for the wrong device.
Additionally, don't print the "run pmbootstrap log for details" message
if pmbootstrap was called with --details-to-stdout
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
There was an oversight when this API was originally created and it
implicitly assumed that the pmaports repository was always named
"pmaports". This unfortunately broke some peoples workflows.
Introduce a new "pkgrepo_name()" function and adjust the codebase to use
it, as well as adjusting pkgrepo internally to special case the
"pmaports" repo so that it's always named pmaports no matter what the
directory itself is named.
This is probably more complexity than we should be dealing with here, we
should probably create a new type to encode this behaviour.
Fixes: #2412
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
This was always a hack, and it looks like there (hopefully) aren't any
placs where we still need to handle this.
Possibly expecting regressions... But then we have something to write a
test for.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
When running "apk add" we must always have --cache-dir set! We also
expect --no-interactive to be set.
Add some asserts so we don't regress here.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Since parse.deviceinfo() returns a custom class now, it can't be
trivially serialized as JSON.
This has been broken for months now so it's clear nobody is using this.
Let's take the chance to get rid of it.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
In 0b4fb9119f (chroot: always run apk static v2 (MR 2423)) we adjusted
install_run_apk() to run apk static on the host and pass in the local
binary repo with "--repository". This function can call apk in two ways,
either with the progress bar handling or without, the second case was
never updated and still ran apk inside the chroot incorrectly and with
an incorrect --repository flag.
Let's finish the job by refactoring helpers/apk.py to support all our
usecases and pointing everything to it, removing the last few situations
where we call "pmb.chroot.root(["apk", ...]).
The apk_with_progress() function is replaced by a generic "run()"
function which takes a boolean to indicate if we should render apk
progress.
Additionally, a new cache_clean() function is added so that "pmbootstrap
zap --pkgs-online-mismatch" can FINALLY be refactored to not rely on a
chroot existing. This requires some hacks but nothing serious, see the
comments in the function for details.
The chroot.init() code is now simplified since handling the --root,
--arch, --cache-dir, and --repository flags is now all done by
apk._prepare_cmd() as and when appropriate.
Lastly, this fixes a (previously unnoticed) bug where apk.static was
actually using /var/cache/apk on your host machine for its cache... This
is definitely not good behaviour....
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
This function better belongs here, especially as it will be used outside
of the context of a chroot() soon.
Additionally, it's adjusted to take the rootfs path as its first
argument rather than a chroot, since it could operate on any rootfs.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Lazy load pmb.config.styles and move the pmb.__version__ print elsewhere
so the logging module is (closer to) a standalone entity. This is
necessary to be able to import it in pmb/helpers/apk.py otherwise we get
a cyclical dependency.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
I think the "these types make no sense" comment were about
output_return_buffer, which defaulted to False but was to be set to
list[bytes] if used. I changed it so that it defaults to None, which
probably is more conventional. Other than that I didn't notice anything
weird about the types here.
Also check for None as necessary to appease mypy.
Flush the progress bar displayed by the function before logging a
warning message for 404 not found responses from the server. Without
flushing, this message gets lost.
In addition to the warning, display a NOTE and ERROR at the end to
explain what went wrong:
[20:16:50] Update package index for x86_64 (4 file(s))
[20:16:51] WARNING: file not found: http://mirror.postmarketos.org/postmarketos_get_404_test/edge/main/x86_64/APKINDEX.tar.gz
[20:16:51] NOTE: check the [mirrors] section in 'pmbootstrap config'
[20:16:51] ERROR: getting APKINDEX from binary package mirror failed!
Without handling this properly, pmbootstrap would fail slightly later
with a much less useful error:
[15:14:28] ERROR: expected str, bytes or os.PathLike object, not NoneType
Fixes: pmbootstrap issue 2424
Don't update the git URL again when coming from 7-2.x where this step
was already done. This didn't fail, but was potentially confusing for
the user since the git URLs were already upgraded from 7 -> 7-2x.
All call sites that use this have been removed. This was only introduced
as a stopgap until we could refactor more code to not use args
everywhere, and that has now been achived. As such, remove this so it
doesn't get used in more locations.
For the gitlab.com -> gitlab.postmarketos.org migration we needed to
also migrate the workdir in pmb v2 since pmb v3 is not ready yet.
Therefore we need to handle that "7-2.x" workdir version and migrate
correctly when switching to pmbootstrap v3.
See https://gitlab.postmarketos.org/postmarketOS/pmbootstrap/-/merge_requests/2445
* Make sure all Python modules are documented by adding a bit of shell
to .ci/docs.sh
* Remove non-existing module references from .rst
* Fix various warnings from sphinx by adjusting Python docstrings
* Add class member docs to ApkindexBlock
I'm not sure why this was annotated as a Sequence rather than list. The
function clearly returns a list, not a Sequence, and the commit that
originally did it[1] provides no justification. This causes problems
with mypy checks in other places, so let's just annotate this as
returning a list.
[1]: 198f302a36
Now that we have moved the kconfigcheck configuration into pmaports
branches via kconfigcheck.toml, it is time to get rid of the hardcoded
list of valid "pmb:kconfigcheck-…" options for APKBUILDs. Generate it
from the kconfigcheck.toml of the current branch, too.
These options are passed from "pmbootstrap lint" to "apkbuild-lint",
which we run in pmaports CI.
The output of apkbuild-lint is hard to spot among other log messages
from pmbootstrap, because it is not colorized.
Add "*** apkbuild-lint output ***"" before and after the output of
apkbuild-lint, so we get a standing out message in green around the
apkbuild-lint output.
It is not needed to run the linter as root, so don't do it. This is
probably a leftover from early days pmbootstrap, where pmaports.git
wasn't chowned by the user running pmbootstrap.
When running some commands like "pmbootstrap chroot" or even
"pmbootstrap work_migrate", pmbootstrap will attempt to read the
pmaports.cfg from origin/master *before* doing the work dir migration.
In order to do this, the "get_upstream_remote" function tries to find
the upstream remote by URL. Let it search with the outdated URLs too, so
this doesn't fail right before migrating to the new URLs.
This could be re-purposed in the future in case we migrate URLs again.
Co-authored-by: Caleb Connolly <caleb@postmarketos.org>
Tweaked-by: Oliver Smith <ollieparanoid@postmarketos.org>
Now that we don't need weird apk-tools hacks for systemd, we can
re-implement this optimisation and always run apk static rather than
running apk through the chroot.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
For packages selected by the user (see last commit), set the
provider_priority to 999999 instead of 999. The previous number seems
not high enough, we might set this in the APKBUILDs.
