Introduce a new "cache" subdirectory in the pmbootstrap workdir, all the
cache and config bits go in here, anything that needs to be accessible
from inside a chroot. The whole dir is then bind-mounted into the chroot
as /cache with appropriate symlinks.
This dir is in the config as config.cache.
In addition, all the cache_* and other config dirs are renamed to
be closer to the names of the equivalent dirs in the chroot (e.g.
abuild-config) and to avoid redundant naming since they are now under a
"cache" dir.
Signed-off-by: Casey Connolly <kcxt@postmarketos.org>
This patch made the assumption that if the Alpine APKINDEX for the
"main" repository exist, all other Alpine indexes ("community",
"testing") as well as the postmarketOS indexes (normal one, systemd)
exist as well.
This is not always the case, e.g. when the internet connection dropped
while downloading indexes and so only "main" was downloaded. While this
is unlikely to happen, it is currently also breaking BPO where the
postmarketOS repository does not get set during "pmbootstrap init" on
purpose as it causes problems when bringing up new releases.
This reverts commit 752a3a98f5.
I'll make a different fix in the next patch for the problem that this
patch was made for, having a broken "pmbootstrap zap -a" with apkv3.
Related: BPO issue 160
Part-of: https://gitlab.postmarketos.org/postmarketOS/pmbootstrap/-/merge_requests/2636
When using `pmb zap --pkgs-online-mismatch`, the local index is
downloaded if missing and `update()` is cached, however pmb zap removes
the index and subsequent calls to `update()` don't actually do anything
(i.e. redownload the index) so later things that expect the indext to
exist (like finding providers) blow up.
Wrap update() with a new update() method that checks if the APKINDEX
exists, and always do a "real" update if it doesn't. Also rename
update() to _update().
Co-Developed-by: Clayton Craft <clayton@craftyguy.net>
Part-of: https://gitlab.postmarketos.org/postmarketOS/pmbootstrap/-/merge_requests/2635
The .supported() method errs on the side of exposing too many
architectures to make porting for new or less-common platforms easier.
It therefore isn't suitable for using as a list of supported
architectures we can probe the binary repository for.
Introduce a .supported_binary() method to export only the architectures
where we have a binary repository.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Do not abort if one or more APKINDEX files cannot be downloaded, if
PMB_APK_FORCE_MISSING_REPOSITORIES is set. This is needed when
bootstrapping new stable branches. The same environment variable is
already used in pmbootstrap code to pass --force-missing-repositories to
apk in pmb.chroot.apk.install_run_apk().
In 0b4fb9119f (chroot: always run apk static v2 (MR 2423)) we adjusted
install_run_apk() to run apk static on the host and pass in the local
binary repo with "--repository". This function can call apk in two ways,
either with the progress bar handling or without, the second case was
never updated and still ran apk inside the chroot incorrectly and with
an incorrect --repository flag.
Let's finish the job by refactoring helpers/apk.py to support all our
usecases and pointing everything to it, removing the last few situations
where we call "pmb.chroot.root(["apk", ...]).
The apk_with_progress() function is replaced by a generic "run()"
function which takes a boolean to indicate if we should render apk
progress.
Additionally, a new cache_clean() function is added so that "pmbootstrap
zap --pkgs-online-mismatch" can FINALLY be refactored to not rely on a
chroot existing. This requires some hacks but nothing serious, see the
comments in the function for details.
The chroot.init() code is now simplified since handling the --root,
--arch, --cache-dir, and --repository flags is now all done by
apk._prepare_cmd() as and when appropriate.
Lastly, this fixes a (previously unnoticed) bug where apk.static was
actually using /var/cache/apk on your host machine for its cache... This
is definitely not good behaviour....
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Flush the progress bar displayed by the function before logging a
warning message for 404 not found responses from the server. Without
flushing, this message gets lost.
In addition to the warning, display a NOTE and ERROR at the end to
explain what went wrong:
[20:16:50] Update package index for x86_64 (4 file(s))
[20:16:51] WARNING: file not found: http://mirror.postmarketos.org/postmarketos_get_404_test/edge/main/x86_64/APKINDEX.tar.gz
[20:16:51] NOTE: check the [mirrors] section in 'pmbootstrap config'
[20:16:51] ERROR: getting APKINDEX from binary package mirror failed!
Without handling this properly, pmbootstrap would fail slightly later
with a much less useful error:
[15:14:28] ERROR: expected str, bytes or os.PathLike object, not NoneType
Fixes: pmbootstrap issue 2424
Allow setting _custom mirrors in the config:
* alpine_custom
* pmaports_custom
* systemd_custom
When these are set, they are added to /etc/apk/repositories before real
repositories. This is used by bpo to build packages with a WIP
repository enabled, in addition to the final repository.
All mirrors can also be set to "none" to be disabled. This is important
for bootstrapping from pure Alpine without any binary repository, and
the bpo testsuite also uses this.
I've discussed with Caleb whether to name it _wip instead of _custom,
but the latter is more generic and people may also use this for other
use cases than the bpo wip repository thing.
Now that we have target-version = "py310" in [tool.ruff] in
pyproject.toml, ruff check complains about using typing.Optional and
typing.Union instead of newer syntax. Run the tool to fix it.
Python 3.10.12 fails without this patch:
File "/media/gompa/73d88639-a730-456c-a428-6d500b4020b7/pmbootstrap/pmb/parse/arguments.py", line 844 help="Alpine Linux mirror, default: " f"{default_config.mirrors["alpine"]}",
SyntaxError: f-string: unmatched '['
Related: https://peps.python.org/pep-0701/
(Commit message written by Oliver)
It turns out that apk with the --root argument will still treat local
repositories relative to the host. This makes sense considering the
intended usecase, however as a result the recent change to use
apk.static for all apk commands inside the target root will break the
use of local packages.
To fix this, adjust how we populate /etc/apk/repositories, changing the
default to no longer include the local user repo. Instead we pass
--repository to apk.static with the host path to the repo.
The only time we want to add the user repo to /etc/apk/repositories
inside the chroot is when the user enters the chroot, so a special case
is added to helpers/frontend.py for the chroot command.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Generalise pmb.helpers.other.cache with a more python decorator.
The Cache decorator takes a list of function arguments to use as cache
keys, keyword args can be used to restrict caching so that it is skipped
entirely unless the attribute has a specific value.
For example, pmb.helpers.pmaports.get() has the decorator:
@Cache("pkgname", subpackages=True)
This means the return value will be cached only when subpackages is
True, otherwise it will always miss.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Use the new config.mirrors section to handle repository URLs instead of
the old mirror_alpine / mirrors_postmarketos options. This let's us
add the systemd staging repo automatically when on the systemd staging
branch / systemd is enabled.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Move pmb/parse/arch.py over to core and refactor it as an Arch type,
similar to how Chroot was done. Fix all the uses (that I can find) of
arch in the codebase that need adjusting.
The new Arch type is an Enum, making it clear what architectures can be
represented and making it much easier to reason about. Since we support
~5 (kinda) different representations of an Architecture (Alpine, Kernel,
target triple, platform, and QEMU), we now formalise that the Alpine
format is what we represent internally, with methods to convert to any
of the others as-needed.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Cease merging pmbootstrap.cfg into args, implement a Context type to let
us pull globals out of thin air (as an intermediate workaround) and rip
args out of a lot of the codebase.
This is just a first pass, after this we can split all the state that
leaked over into Context into types with narrower scopes (like a
BuildContext(), etc).
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Testing by building postmarketos-initramfs (which installs >100 packages
but is very fast to build, so a worst-case scenario) this results in a
~15-20% speedup (which everything cached and doing multiple back to back
runs). From 32 seconds down to 25.
Doing a full install with --no-image, this takes us from 70 seconds on
my laptop down to 40s!
This also lets us drastically simplify pmb/helpers/apk.py!
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
With the new chroot type, we can now write fancy paths in the pythonic
way. Convert most of the codebase over, as well as adding various other
type hints.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
We use a custom verbose log level in pmbootstrap, unfortunately it isn't
possible to correctly type this due to some limitations in the logging
library [1], [2].
Given that our usecase is fairly simple, we can just wrap the module
with our own so we only have to tell mypy to ignore the error once
instead of at every callsite.
[1]: https://github.com/cryptax/droidlysis/issues/15
[2]: https://github.com/python/typing/discussions/980
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Introduce a new module: pmb.core to contain explicitly typed pmbootstrap
API. The first component being Suffix and SuffixType. This explicitly
defines what suffixes are possible, future changes should aim to further
constrain this API (e.g. by validating against available device
codenames or architectures for buildroot suffixes).
Additionally, migrate the entire codebase over to using pathlib.Path.
This is a relatively new part of the Python standard library that uses a
more object oriented model for path handling. It also uses strong type
hinting and has other features that make it much cleaner and easier to
work with than pure f-strings. The Chroot class overloads the "/"
operator the same way the Path object does, allowing one to write paths
relative to a given chroot as:
builddir = chroot / "home/pmos/build"
The Chroot class also has a string representation ("native", or
"rootfs_valve-jupiter"), and a .path property for directly accessing the
absolute path (as a Path object).
The general idea here is to encapsulate common patterns into type hinted
code, and gradually reduce the amount of assumptions made around the
codebase so that future changes are easier to implement.
As the chroot suffixes are now part of the Chroot class, we also
implement validation for them, this encodes the rules on suffix naming
and will cause a runtime exception if a suffix doesn't follow the rules.
Replace "args.cache" with a global variable in order to
avoid passing "args" to all functions. This is a step to get rid of this
args-passed-to-all-functions pattern in pmbootstrap.
Replace "args.arch_native" with the direct function call in order to
avoid passing "args" to all functions. This is a step to get rid of this
args-passed-to-all-functions pattern in pmbootstrap.
Hide progress bars if --details-to-stdout is used, which redirects all
output that would land in the pmbootstrap log to stdout. This caused the
progress bar output to get mixed with the apk output. A new progress bar
would get drawn whenever a new package was installed, without removing
the previous progress bar.
Migrate to workdir version 5 and move already built packages into the edge
channel subdir, for example:
$WORK/packages/x86_64/hello-world-1-r5.apk
to:
$WORK/packages/edge/x86_64/hello-world-1-r5.apk
The build.postmarketos.org code has already been adjusted to find built
packages in either directory structure.
Replace the call to this function with the almost identical
pmb.helpers.repo.apkindex_files(), so release channel related changes
only need to be done in one place.
Use mirrordir_pmos and mirrordir_alpine from channels.cfg to generate
the mirror URLs for postmarketOS and Alpine, which get written to
/etc/apk/repositories and which postmarketOS uses to download the
APKINDEX files.
Remove hardcoded "master" at the end of the postmarketOS mirror and use
mirrordir_pmos instead (which is "master" for the edge channel). Let the
postmarketOS mirror end in a '/' for consistency with the Alpine mirror
in pmb/config/__init__.py.
Remove obsolete --alpine-version. To experiment with a different Alpine
version, one should pass a custom --config-channels from now on.
Don't generate the postmarketOS mirror URLs here, let the urls()
function do it. A follow-up commit will touch this code, hence it's
important to have it de-duplicated.
