Lazy load pmb.config.styles and move the pmb.__version__ print elsewhere
so the logging module is (closer to) a standalone entity. This is
necessary to be able to import it in pmb/helpers/apk.py otherwise we get
a cyclical dependency.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
I think the "these types make no sense" comment were about
output_return_buffer, which defaulted to False but was to be set to
list[bytes] if used. I changed it so that it defaults to None, which
probably is more conventional. Other than that I didn't notice anything
weird about the types here.
Also check for None as necessary to appease mypy.
Flush the progress bar displayed by the function before logging a
warning message for 404 not found responses from the server. Without
flushing, this message gets lost.
In addition to the warning, display a NOTE and ERROR at the end to
explain what went wrong:
[20:16:50] Update package index for x86_64 (4 file(s))
[20:16:51] WARNING: file not found: http://mirror.postmarketos.org/postmarketos_get_404_test/edge/main/x86_64/APKINDEX.tar.gz
[20:16:51] NOTE: check the [mirrors] section in 'pmbootstrap config'
[20:16:51] ERROR: getting APKINDEX from binary package mirror failed!
Without handling this properly, pmbootstrap would fail slightly later
with a much less useful error:
[15:14:28] ERROR: expected str, bytes or os.PathLike object, not NoneType
Fixes: pmbootstrap issue 2424
Don't update the git URL again when coming from 7-2.x where this step
was already done. This didn't fail, but was potentially confusing for
the user since the git URLs were already upgraded from 7 -> 7-2x.
All call sites that use this have been removed. This was only introduced
as a stopgap until we could refactor more code to not use args
everywhere, and that has now been achived. As such, remove this so it
doesn't get used in more locations.
For the gitlab.com -> gitlab.postmarketos.org migration we needed to
also migrate the workdir in pmb v2 since pmb v3 is not ready yet.
Therefore we need to handle that "7-2.x" workdir version and migrate
correctly when switching to pmbootstrap v3.
See https://gitlab.postmarketos.org/postmarketOS/pmbootstrap/-/merge_requests/2445
* Make sure all Python modules are documented by adding a bit of shell
to .ci/docs.sh
* Remove non-existing module references from .rst
* Fix various warnings from sphinx by adjusting Python docstrings
* Add class member docs to ApkindexBlock
I'm not sure why this was annotated as a Sequence rather than list. The
function clearly returns a list, not a Sequence, and the commit that
originally did it[1] provides no justification. This causes problems
with mypy checks in other places, so let's just annotate this as
returning a list.
[1]: 198f302a36
Now that we have moved the kconfigcheck configuration into pmaports
branches via kconfigcheck.toml, it is time to get rid of the hardcoded
list of valid "pmb:kconfigcheck-…" options for APKBUILDs. Generate it
from the kconfigcheck.toml of the current branch, too.
These options are passed from "pmbootstrap lint" to "apkbuild-lint",
which we run in pmaports CI.
The output of apkbuild-lint is hard to spot among other log messages
from pmbootstrap, because it is not colorized.
Add "*** apkbuild-lint output ***"" before and after the output of
apkbuild-lint, so we get a standing out message in green around the
apkbuild-lint output.
It is not needed to run the linter as root, so don't do it. This is
probably a leftover from early days pmbootstrap, where pmaports.git
wasn't chowned by the user running pmbootstrap.
When running some commands like "pmbootstrap chroot" or even
"pmbootstrap work_migrate", pmbootstrap will attempt to read the
pmaports.cfg from origin/master *before* doing the work dir migration.
In order to do this, the "get_upstream_remote" function tries to find
the upstream remote by URL. Let it search with the outdated URLs too, so
this doesn't fail right before migrating to the new URLs.
This could be re-purposed in the future in case we migrate URLs again.
Co-authored-by: Caleb Connolly <caleb@postmarketos.org>
Tweaked-by: Oliver Smith <ollieparanoid@postmarketos.org>
Now that we don't need weird apk-tools hacks for systemd, we can
re-implement this optimisation and always run apk static rather than
running apk through the chroot.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
For packages selected by the user (see last commit), set the
provider_priority to 999999 instead of 999. The previous number seems
not high enough, we might set this in the APKBUILDs.
This function is used by "pmbootstrap repo_missing", which is only used
by bpo. In order to support building the split repository in bpo, this
patch is required.
Return "systemd" for the packages that are in extra-repos/systemd, and
None for all other packages (indicating that they don't get split into a
separate repository).
Checking the parent-parent dir of the APKBUILD is a bit fragile
(assuming we don't have subdirs in the systemd repository), but that
assumption was made with the previous implementation as well (would only
return "systemd" in that case). I don't see a better solution with
reasonable effort right now, given that we also support multiple
pmaports dirs. We can improve this in the future, meanwhile let's just
not use subdirs in the systemd repository.
Don't attempt to find packages of other arches in repo_missing code.
This leads to unexpected results in general (we want to check if a
binary package exists for only one specific architecture!) - and in case
of master_staging_systemd, it current causes errors because pmbootstrap
tries to fetch armhf, riscv etc. APKINDEXes for which we don't build the
staging repositories.
As chroot.glob() now returns a generator, convert it to a list first to
check if it is empty. This may not be the most efficient method, but
since we expect it to be empty here it is fine.
Without this patch, it always shows the error even if there are no
locally built packages:
ERROR: --no-local-pkgs specified, but locally built packages found. Consider 'pmbootstrap zap -p' to delete them.
When a dependency cannot be found in source or binary packages of the
selected architecture, then don't look for binary packages in other
architectures package indexes.
This leads to a confusing error down the line when building packages for
extra_repos_systemd (and e.g. stable branches), because we don't have
binary packages for all arches that pmbootstrap supports, and so it
tries to fetch a non-existing APKINDEX.
Instead, print a nice error about a non-existing dependency.
Get rid of 3 years and older migration code, so we don't need to
maintain it anymore. If the user should have run "pmbootstrap init" 3
years ago and suddenly upgrades to pmbootstrap v3, then they will just
need to delete their work dir (as pmbootstrap instructs to do) and start
over. This is fine, the work dir is just a bunch of caches and locally
built packages.
repo is a Path obj now, and its name is used as a key in the dest_paths dict.
Fixes this failure:
working_dir=dest_paths[repo],
~~~~~~~~~~^^^^^^
KeyError: PosixPath('/home/clayton/src/pmaports')
Remove arguments to set postmarketOS and Alpine mirrors.
The current implementation is broken with the way the mirrors are now
configured: The current implementation fills a list, but the mirror
configuration code expects a string. It is not longer possible to set
an arbitrary amount of mirrors for the postmarketOS mirror. The benefit
is that we don't have comma separated values in the config file anymore
but instead just simple strings. Now it is possible to have one proper
mirror for aports, pmaports and pmaports_systemd, and one _custom one
to override each of these.
We already have too many options in "pmbootstrap -h", and changing a
mirror is more of a base configuration that you do once and maybe change
a few times, but don't need to be able to set it with each pmbootstrap
run. Users are not going to write this out manually on their
command-lines, it only makes sense in scripts and wrappers for
pmbootstrap, and there you could as well use another method (as now
described in docs/mirrors.md) to set the mirrors.
Less important, but another reason is that using "-mp" (two letters as
short argument) isn't really elegant.
Fix writing config options that are stored as CSV-list. Without this
patch:
$ pmbootstrap config aports /tmp/pmaports
[23:43:58] Config changed: aports='/tmp/pmaports'
$ pmbootstrap config aports
[PosixPath('/'), PosixPath('t'), PosixPath('m'), PosixPath('p'), PosixPath('/'), PosixPath('p'), PosixPath('m'), PosixPath('a'), PosixPath('p'), PosixPath('o'), PosixPath('r'), PosixPath('t'), PosixPath('s')]
Storing strings as CSV list isn't great and we probably want to
implement this more elegantly, see the related issue. But let's fix this
first.
Related: pmbootstrap issue 2412
Allow setting _custom mirrors in the config:
* alpine_custom
* pmaports_custom
* systemd_custom
When these are set, they are added to /etc/apk/repositories before real
repositories. This is used by bpo to build packages with a WIP
repository enabled, in addition to the final repository.
All mirrors can also be set to "none" to be disabled. This is important
for bootstrapping from pure Alpine without any binary repository, and
the bpo testsuite also uses this.
I've discussed with Caleb whether to name it _wip instead of _custom,
but the latter is more generic and people may also use this for other
use cases than the bpo wip repository thing.
