Introduce a new "cache" subdirectory in the pmbootstrap workdir, all the
cache and config bits go in here, anything that needs to be accessible
from inside a chroot. The whole dir is then bind-mounted into the chroot
as /cache with appropriate symlinks.
This dir is in the config as config.cache.
In addition, all the cache_* and other config dirs are renamed to
be closer to the names of the equivalent dirs in the chroot (e.g.
abuild-config) and to avoid redundant naming since they are now under a
"cache" dir.
Signed-off-by: Casey Connolly <kcxt@postmarketos.org>
The parameters of this function were needlessly complicated. While I
understand the intent of trying to replicate the API of retrieve()
without duplicating the parameter list, in practice this feels like
reducing duplication to a fault to me. This is due to that the third
parameter, <allow_404>, doesn't make sense in the context of
retrieve_json() as it would raise an exception[1] if retrieve() returns
None (which is the effect of <allow_404> when a 404 occurs), thus merely
changing the exception that gets raised to a less descriptive one
instead of actually behaving like retrieve() would with that parameter
set to True.
With <allow_404> eliminated, there's just two parameters left, and I
don't think duplicating those is a big deal. So make the function easier
to read and annotate by getting rid of the args/kwargs logic.
[1]: json.loads() raises an exception if its first argument is None
Flush the progress bar displayed by the function before logging a
warning message for 404 not found responses from the server. Without
flushing, this message gets lost.
In addition to the warning, display a NOTE and ERROR at the end to
explain what went wrong:
[20:16:50] Update package index for x86_64 (4 file(s))
[20:16:51] WARNING: file not found: http://mirror.postmarketos.org/postmarketos_get_404_test/edge/main/x86_64/APKINDEX.tar.gz
[20:16:51] NOTE: check the [mirrors] section in 'pmbootstrap config'
[20:16:51] ERROR: getting APKINDEX from binary package mirror failed!
Without handling this properly, pmbootstrap would fail slightly later
with a much less useful error:
[15:14:28] ERROR: expected str, bytes or os.PathLike object, not NoneType
Fixes: pmbootstrap issue 2424
Cease merging pmbootstrap.cfg into args, implement a Context type to let
us pull globals out of thin air (as an intermediate workaround) and rip
args out of a lot of the codebase.
This is just a first pass, after this we can split all the state that
leaked over into Context into types with narrower scopes (like a
BuildContext(), etc).
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
With the new chroot type, we can now write fancy paths in the pythonic
way. Convert most of the codebase over, as well as adding various other
type hints.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
We use a custom verbose log level in pmbootstrap, unfortunately it isn't
possible to correctly type this due to some limitations in the logging
library [1], [2].
Given that our usecase is fairly simple, we can just wrap the module
with our own so we only have to tell mypy to ignore the error once
instead of at every callsite.
[1]: https://github.com/cryptax/droidlysis/issues/15
[2]: https://github.com/python/typing/discussions/980
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Introduce a new module: pmb.core to contain explicitly typed pmbootstrap
API. The first component being Suffix and SuffixType. This explicitly
defines what suffixes are possible, future changes should aim to further
constrain this API (e.g. by validating against available device
codenames or architectures for buildroot suffixes).
Additionally, migrate the entire codebase over to using pathlib.Path.
This is a relatively new part of the Python standard library that uses a
more object oriented model for path handling. It also uses strong type
hinting and has other features that make it much cleaner and easier to
work with than pure f-strings. The Chroot class overloads the "/"
operator the same way the Path object does, allowing one to write paths
relative to a given chroot as:
builddir = chroot / "home/pmos/build"
The Chroot class also has a string representation ("native", or
"rootfs_valve-jupiter"), and a .path property for directly accessing the
absolute path (as a Path object).
The general idea here is to encapsulate common patterns into type hinted
code, and gradually reduce the amount of assumptions made around the
codebase so that future changes are easier to implement.
As the chroot suffixes are now part of the Chroot class, we also
implement validation for them, this encodes the rules on suffix naming
and will cause a runtime exception if a suffix doesn't follow the rules.
While at it, also remove unnecessary "#!/usr/bin/env python3" in files
that only get imported, and adjust other empty/comment lines in the
beginnings of the files for consistency.
This makes files easier to read, and makes the pmbootstrap codebase more
consistent with the build.postmarketos.org codebase.
Do not fail when an APKINDEX can not be downloaded, print a WARNING
instead. This matches, what apk does. Add a new allow_404 parameter to
pmb.helpers.http.download(), and use it in pmb.helpers.repo.update()
for downloading APKINDEX files. Cache the APKINDEX URLs that gave a 404
for the session, so we do not attempt to download these again.
This is needed for the new binary repository: the initial build is done
without existing APKINDEX files, so we must not fail in that case.
* add my own build key
* enable the repo in the config
* update the README file
* Adjust testcase, that validates the keys and enable it in testcases_fast.sh
* Only save/load keys to/from the config file, which we ask for during
'pmbootstrap init', so the binary repo gets used even if a config file
already exists (this also removes a workaround, that deletes the work
folder path from the config dictionary before writing it)
* Download missing APKINDEX.tar.gz files with Python code, before
attempting to build packages (so we know which ones aleady exist in
the binary packages repository)
* Consider APKINDEX files older than 4 hours as outdated and download
them again (also in Python code)
* Provide 'pmbootstrap update' to force-update the APKINDEX files
* Travis: more logging output on failure
* Only allow keys from config_keys to be used by "pmbootstrap config"
