The repo bootstrap logic has worked well enough in the initial systemd
bringup. But we have decided to replace the approach of building
packages multiple times during the repo_bootstrap with building each
package only once and adding -stage0 packages where necessary (currently
only a systemd-stage0 package).
Advantages:
* Replace the often breaking repo_bootstrap logic with something less
fragile (repo_bootstrap is currently broken again).
* Can get rid of the whole bootstrap logic in pmaports.cfg, pmbootstrap,
build.postmarketos.org. This will make pmbootstrap and bpo easier to
maintain.
* Fix problems we have seen when upgrading two or more of (systemd,
dbus, linux-pam-pmos) at once: it doesn't pass in CI and it doesn't
pass in BPO
* You don't need to do a whole bootstrap at once, it is again broken
down by package. This means if building one package fails (locally or
in bpo), we don't need to do the whole thing again, just start at the
package that failed. This also means it is much easier to optimize
e.g. the stage0 packages to build faster.
* Fixes some specific bugs we currently have (abuild twice in pmaports,
bpo tests disabled because of that, ...)
Part-of: https://gitlab.postmarketos.org/postmarketOS/pmbootstrap/-/merge_requests/2588
With recent rework in postmarketos-initramfs, we no longer need to mount
the /boot partition in the initramfs (assuming initramfs-extra is not
used). On devices that boot without accessing the boot file system (e.g.
Android boot images, fastboot, ...), that makes it possible to install
postmarketOS on a single (potentially encrypted) partition that contains
both root (/) and /boot files.
This avoids the extra complexity of the subpartition setup we usually use
on such devices, and also avoids having to flash two partitions (when using
--split to avoid the subpartitions).
Add a --single-partition option to pmbootstrap install that allows
installing postmarketOS in this mode. For now this is just an option that
must be selected explicitly, in the future we could choose to make this the
default for Android-based devices with a large enough boot partition.
Part-of: https://gitlab.postmarketos.org/postmarketOS/pmbootstrap/-/merge_requests/2512
Add a flag to bind-mount in the necessary sysfs directories for USB
devices to be accessed inside the chroot. This is the same as how we do
it for "pmbootstrap flasher" but allows running arbitrary commands in
the chroot.
This is useful to allow using pmbootstrap chroot's as a sandbox for
custom flashing procedures.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
The "pmbootstrap repo_missing" action is used exclusively by bpo. It
calls it only with these arguments:
pmbootstrap repo_missing --built --arch "$ARCH"
A blocker for merging systemd into pmaports master is, that the
current repo_missing code cannot display packages that are both in
extra-repos/systemd and in another path. I have considered just not
supporting this and discussed doing that with Caleb and Clayton, but we
figured it would be a major obstacle in the future to not be able to
easily override packages with systemd specific versions (currently we
need this for 3 packages).
Integrating this into the existing repo_missing code would be hacks upon
hacks. Also the scope of the current repo_missing code has many extra
features that are not used and would be extra effort to carry along:
* Allow specifying a pkgname
* Running without --built
* --overview
So I decided to replace the repo_missing code with a much simpler, more
modern implementation, that does exactly what is needed:
* Duplicate packages in systemd and non-systemd dirs are displayed
* The output always include all packages, no matter if they are already
built or not (same behavior as with --built)
* Removed --overview and selecting specific packages too
* The code for filling "repo" (either "systemd" or None) is more
resilient now, as it can use proper relative paths to the root of
pmaports. Unlike the previous implementation, it will not fail if
subdirs are added to the systemd dir.
I have made sure that the output is exactly the same as before on
current pmaports master.
Related: bpo issue 144
Related: bpo issue 140
Presumably, args and kwargs are just here to retain compatibility with
add_argument() from argparse.ArgumentParser, but they aren't actually
used and consequently only give the illusion of compatibility. Just
remove them to simplify the function signature.
We shouldn't overwrite config.aports if the user specified a custom
--aports directory. We only want to do this to update the default if
they're using a custom workdir.
Make the code here less aggressive, and correctly use the last entry in
config.aports as "pmaports" instead of the first.
Fixes: #2386
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
This can be used when building images for generic device targets that
support devices with different sector size requirements.
For example, trailblazer prebuilts are currently expected to be flashed
to a USB drive where a 4096 sector size would be unsuitable since the
bootloader wouldn't detect it. But when building for a Qualcomm phone,
one would use --split and --sector-size to build the root and boot
partitions with a 4k sector size which is appropriate to the UFS
storage.
This flag could also be used by BPO to build both variants.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Abort the "pmbootstrap kconfig check" on the first error, unless the
--keep-going argument was passed. This makes it easier to go through the
failed kernels one by one until they are all fixed.
Remove arguments to set postmarketOS and Alpine mirrors.
The current implementation is broken with the way the mirrors are now
configured: The current implementation fills a list, but the mirror
configuration code expects a string. It is not longer possible to set
an arbitrary amount of mirrors for the postmarketOS mirror. The benefit
is that we don't have comma separated values in the config file anymore
but instead just simple strings. Now it is possible to have one proper
mirror for aports, pmaports and pmaports_systemd, and one _custom one
to override each of these.
We already have too many options in "pmbootstrap -h", and changing a
mirror is more of a base configuration that you do once and maybe change
a few times, but don't need to be able to set it with each pmbootstrap
run. Users are not going to write this out manually on their
command-lines, it only makes sense in scripts and wrappers for
pmbootstrap, and there you could as well use another method (as now
described in docs/mirrors.md) to set the mirrors.
Less important, but another reason is that using "-mp" (two letters as
short argument) isn't really elegant.
- Change some arguments' types to bool to avoid type mismatch errors
after refactoring in the future
- Add more arguments from parse/arguments with corresponding types
- Change type of pmbootstrap log --lines argument to int
Using pmbootstrap v3 with the old config can cause problems, for example
when having $WORK in the pmaports dir instead of the actual work path.
This is not supported anymore by v3 to reduce complexity. The format of
how mirrors are stored in the config also has changed.
Use a separate config file, so users can go back from v3 to 2.3.x if
they need to (for figuring out a regression) and so users won't run into
bugs when moving from 2.3.x to v3.
Add the missing runtime conversion when --config is given on the
cmdline.
This expects a pathlib.Path object as the argument, but args.config is
only a string.
Remove a bunch of commented out code:
* debug leftovers
* update_work(): a function that used to replace $WORK in all kinds of
arguments to pmbootstrap, with the actual work dir. This seemed like a
good idea when I made this initially, but in hindsight it's clear that
this made args just much more complex. I'm glad that Caleb removed
this. Now $WORK only gets removed in one place, for mount points,
which makes much more sense.
* args.from_argparse and related comment: pmbootstrap used to make a
full copy of its args, so in some cases it could go back before $WORK
was replaced IIRC. This isn't great, but even worse is that this
caused each stack trace to say something like infinite recursion on
args. I'm also glad that Caleb got rid of this :)
Prepare to move the kconfigcheck config into pmaports, so it can be
different per branch and so we can adjust it at the same time as
adjusting the kernel configs.
Once it is moved, pmbootstrap will not know the cateogry names in
advance anymore, so remove the logic for having category-specific
argument names.
In a future patch, we could add an option to manually override the
categories to be tested with one specific category (taking a string
argument), if there is demand for that.
Python 3.10.12 fails without this patch:
File "/media/gompa/73d88639-a730-456c-a428-6d500b4020b7/pmbootstrap/pmb/parse/arguments.py", line 844 help="Alpine Linux mirror, default: " f"{default_config.mirrors["alpine"]}",
SyntaxError: f-string: unmatched '['
Related: https://peps.python.org/pep-0701/
(Commit message written by Oliver)
Add a new flag --image which can be used to mount the rootfs generated
with "pmbootstrap install".
For now this is quite limited in scope. But it's enough to allow for
building a package, updating it in the QEMU image, and then booting it.
The major "gotcha" with this is that the QEMU uses the kernel and
initramfs from the device chroot unless you run it with --efi.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
This is to serve as a place to dump useful internal tests, starting with
one that simply parses all available APKINDEX files.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Get rid of config.mirror_alpine and mirrors_postmarketos and make sure
they get migrated over for existing users.
mirrors_postmarketos being a list was always a bit off, but now we have
per-aports mirrors which make a lot more sense for what we're trying to
do with systemd.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Add a new config section "mirrors", to replace the mirrors_alpine and
mirrors_postmarketos options. This will allow for more flexibility since
we can then handle the systemd staging repo (and others like plasma
nightly) with relative ease.
The loading/saving is fixed and now properly avoids writing out default
values, this way if the defaults are changed the user won't be stuck
with old values in their pmbootstrap.cfg.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Move pmb/parse/arch.py over to core and refactor it as an Arch type,
similar to how Chroot was done. Fix all the uses (that I can find) of
arch in the codebase that need adjusting.
The new Arch type is an Enum, making it clear what architectures can be
represented and making it much easier to reason about. Since we support
~5 (kinda) different representations of an Architecture (Alpine, Kernel,
target triple, platform, and QEMU), we now formalise that the Alpine
format is what we represent internally, with methods to convert to any
of the others as-needed.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Keeping the Config class in types seemed kinda weird and was just done
as a workaround to some cyclical imports. But now things are more in
shape let's move it to core.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Use a different binary repo depending on the source repository for the
package. This makes it possible to split out systemd packages into their
own repository.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Cease merging pmbootstrap.cfg into args, implement a Context type to let
us pull globals out of thin air (as an intermediate workaround) and rip
args out of a lot of the codebase.
This is just a first pass, after this we can split all the state that
leaked over into Context into types with narrower scopes (like a
BuildContext(), etc).
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
With the new chroot type, we can now write fancy paths in the pythonic
way. Convert most of the codebase over, as well as adding various other
type hints.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
The init_cache() function just assigned some default constants, simplify
this by just declaring it that way to begin with, as well as adding type
hints.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
This is needed for sphinx autoprogram since that expects an
argparse.ArgumentParser, and arguments() returns some argparse
"Namespace" obj. Useful for sphinx/autoprogram and maybe other things
later that want to get at pmb's full args.
Reimplement "pmbootstrap status" to be just a simple and useful status
overview. The previous version ran a bunch of checks every time, and
would fail on these even if pmaports was used for normal development:
* "non-official" branch checked out in pmaports
* pmaports.git is not clean
The information about aports.git was also considered not so useful upon
revisiting this command, since it is only used for "pmbootstrap
aportgen". Most users don't need this, and if the user runs this
command, it will tell if aports.git is outdated.
All of the above made the previous version unpleasant to use and I
suspect most people stopped using the command after trying it out a few
times and seeing the irrelevant but loud NOK complaints.
New version:
$ pmbootstrap status
Channel: edge (pmaports: master_staging_systemd)
Device: qemu-amd64 (x86_64, kernel: virt)
UI: console
systemd: no (default for selected UI)
Old version (without --details it only shows NOK checks):
$ pmbootstrap status --details
[00:55:20] *** CONFIG ***
[00:55:20] Device: qemu-amd64 (x86_64, "QEMU amd64")
[00:55:20] Kernel: virt
[00:55:20] User Interface: console
[00:55:20]
[00:55:20] *** GIT REPOS ***
[00:55:20] Path: /home/user/.local/var/pmbootstrap/cache_git
[00:55:20] - aports_upstream (master)
[00:55:20] - pmaports (master)
[00:55:20]
[00:55:20] *** CHECKS ***
[00:55:20] [OK ] Chroots zapped recently (or non-existing)
[00:55:20] [OK ] aports_upstream: on official channel branch
[00:55:20] [OK ] aports_upstream: workdir is clean
[00:55:20] [OK ] aports_upstream: tracking proper remote branch 'origin/master'
[00:55:20] [OK ] aports_upstream: up to date with remote branch
[00:55:20] [OK ] aports_upstream: remote information updated recently (via git fetch/pull)
[00:55:20] [OK ] pmaports: on official channel branch
[00:55:20] [OK ] pmaports: workdir is clean
[00:55:20] [OK ] pmaports: tracking proper remote branch 'origin/master'
[00:55:20] [OK ] pmaports: up to date with remote branch
[00:55:20] [OK ] pmaports: remote information updated recently (via git fetch/pull)
[00:55:20]
[00:55:20] NOTE: chroot is still active (use 'pmbootstrap shutdown' as necessary)
[00:55:20] DONE!
Instead of assuming the architecture of the foreign device to be what
the user selected in pmbootstrap init, actually query the architecture
from the device and use that.
This does mean that one extra ssh connection is necessary, which does
slow down the procedure somewhat. However, I think that is worth the
user experience improvement this brings.
Also, the deduction process can be skipped by manually specifying
--arch in the sideload invocation should it fail, or if one really wants
to skip that extra ssh roundtrip.
Related: https://gitlab.com/postmarketOS/pmbootstrap/-/issues/2317
Often I find myself confused and subsequently frustrated by the
behaviour of --fork-alpine as usually I apply whatever changes I want to
the aports package before forking. This is due to that in the proper
aports repo I can leverage the full power of git and apply patches or
revert commits, whereas after the package has been forked to pmaports
the context git needs to do this is lost.
To fix that, this commit introduces --fork-alpine-retain-branch which
works the same way as --fork-alpine except it doesn't change the aports
branch to match the current channel.
Repurposing argcomplete as a boolean telling whether the module
argcomplete was found results in a variable with two distinct purposes
and types, which is bad practice and confuses Mypy. As such, don't
assign False to argcomplete if the module is not found and instead just
check for whether it exists in sys.modules to determine whether it was
imported successfully.
