The return value of this function is never used. Additionally, the type
hint for it was incorrect as it may return None. As such, don't bother
returning anything both to appease Mypy and also to remove unused
and consequently presumably untested functionality.
Why
Btrfs has some goodness (snapshots, switching between different rw snapshot)
which plays particularly well with certain "subvolume layouts".
What
This MR seeks to implement such a layout, namely a flat btrfs layout,
where the top level subvolume (i.e. the btrfs filesystem/partition itself)
remains unmounted in all situations,
except when making changes to direct child subvolumes of the filesystem.
- rename all subvols to follow the common @* btrfs subvol naming scheme.
- add subvol @root, because roots home directory shouldn't be rolled back.
- make subvol @var not Copy-on-Write (nodatacow) to avoid write
- multiplication on logs, VMs, databases, containers and flatpaks.
- add subvol @snapshots because that lets us change the root subvol to a
read-write snapshot of @ without affecting snapshots.
- add subvol @srv because it contains data for Web and FTP servers,
which shouldn't roll back together with root subvol.
- add subvol @tmp because we don't want to snapshot temporary files.
This subvol remains unmounted on the device,
unless conditions as laid out in pmaports!4737 are met.
- add check and error for btrfs when using rsync installation.
Fix "pmbootstrap chroot" and others not passing the proxy environment
variables correctly. Thanks to notfound405 for pointing this out!
Instead of only preserving proxy environment variables in
pmb.helpers.run_core, which should never be called directly, do it in
the calling functions:
* pmb.helpers.run.user
* pmb.helpers.run.root
* pmb.chroot.root
* pmb.chroot.user
This fixes that the environment variables were only really passed by
pmb.helpers.run.user, because the other functions would result in
something like:
HTTP_PROXY=mytestproxy sudo env -i /usr/bin/sh -c '…'
This is needed to either elevate to root, or to elevate to root first
and then enter the chroot as root or user. Due to the "env -i", the
environment intentionally gets cleaned, but unintentionally also removes
the proxy environment variables that were explicitly set.
By adjusting the functions, they now run a variant of:
sudo env -i /usr/bin/sh -c 'HTTP_PROXY=mytestproxy …'
The escaping is simplified in this example, run "pmbootstrap -v" to see
the not very readable, but proper escaping with shutil.quote().
Remove the previous test for preserving the environment variables in
pmb.helpers.run_core (as it should never be called directly), and test
instead the new behavior.
Fixes: issue 2299
Fixes: 13c4ac42 ("pmb.helpers.run_core: fix proxy env var logic")
Without this, pmbootstrap will give an error if you have a nonexistent
device selected when trying to checksum a package. This can be annoying
when switching between different branches where devices may or may not
exist, and I don't think these initialisation steps are necessary for
the checksum action anyway.
If we found an APKBUILD already, with the exact name of the package we
are looking for, then don't also guess what other package could contain
the package as subpackage.
The guess was thrown away with the previous logic, but it caused
misleading verbose logs like:
lomiri-history-service: guessed to be a subpackage of lomiri
Also this is slightly faster. While at it, tweak the related comments
to make this easier to understand.
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20231121222457.2871-1-ollieparanoid@postmarketos.org%3E
Fix that APKBUILDs mentioning other packages with an operator could not
be found. As we are building the initial branch of v23.12, this
currently happens with postmarketos-mkinitfs: it depends on
devicepkg-utils>=0.2.0 and currently pmbootstrap will only remove the
>=0.2.0 when looking for the pkgname in the APKINDEX of binary packages
(which is why it works on master). But it does not yet do that when
looking for the pkgname in pmaports.
Move the code for stripping the operator to a common place and use it
for getting packages from pmaports too.
Change the order of operators while at it, try to find <= before =, as
otherwise it would cut off example<=1.2.3 as "example<" instead of
"example".
Reviewed-by: Caleb Connolly <kc@postmarketos.org> (via chat)
It can be quite useful to check the configs for a list of packages, so
adjust the argument parsing code to support that.
Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Tested-by: Oliver Smith <ollieparanoid@postmarketos.org>
Reviewed-by: Clayton Craft <clayton@craftyguy.net>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20231028085901.35205-1-luca@z3ntu.xyz%3E
Fix that the list "cmd" was turned into a string if one of the proxy
vars was set in the environment. Add a test for this code path. Before
this patch:
$ FTP_PROXY=test pmbootstrap -v --details-to-stdout status
…
% cd /home/user/.local/var/pmbootstrap/cache_git/pmaports; git remote -v
run: FTP_PROXY=test ['git', 'remote', '-v']
ERROR: [Errno 2] No such file or directory: "FTP_PROXY=test ['git', 'remote', '-v']"
Fixes: 1a00c04f ("pmb.helpers.run_core: always configure proxy vars if set in environment")
Reviewed-by: Clayton Craft <clayton@craftyguy.net>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230713182337.6185-3-ollieparanoid@postmarketos.org%3E
With this code path, pmbootstrap would start a distccd + sshd in the
native chroot, and configure it so it runs the cross compiler. The
foreign arch chroots would then call this cross compiler from localhost
by calling the distcc client instead of gcc.
This code has been obsoleted by the much simpler crossdirect in 2019.
Let's finally remove it.
Fixes: issue 2179
Reviewed-by: Luca Weiss <luca@z3ntu.xyz>
Reviewed-by: Clayton Craft <clayton@craftyguy.net>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230613161437.570196-4-ollieparanoid@postmarketos.org%3E
pmb.config.sudo expects a list of commands + args, so when a string is
passed (e.g. "true"), it results in pmb trying to execute the equivalent
of "$ t r u e":
(8/119) Installing linux-purism-librem5 (6.3.4-r0)
doas: t: command not found
doas: t: command not found
doas: t: command not found
doas: t: command not found
Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230613141929.15718-1-clayton@craftyguy.net%3E
When looking at the table for possible output modes, it only makes sense
to pass stdin to "interactive" and "tui". The output mode "stdout" is
for non-interactive commands.
This fixes apk going interactive (asking for confirmation) when running
pmbootstrap with --details-to-stdout and building a package that depends
on postmarketos-base.
Fixes: issue 2208
Tested-by: Clayton Craft <clayton@craftyguy.net>
Reviewed-by: Clayton Craft <clayton@craftyguy.net>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230301204112.4351-1-ollieparanoid@postmarketos.org%3E
Instead of letting the mount -t binfmt_misc … command fail, and simply
telling the user that the command failed: rely on the following check
that prints a better error message on error. If it fails at this point,
the kernel option isn't enabled. Add two comments explaining why
check=False is used in this function while at it.
Reviewed-by: Luca Weiss <luca@z3ntu.xyz>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230123064516.1607-1-ollieparanoid@postmarketos.org%3E
You can either say you want all scripts, or give a list of script names,
not both. Add it this way and not with an add_mutually_exclusive_group,
as I'll add a add_mutually_exclusive_group in the next patch to only
specify --all or --fast, but having --fast with script names is fine.
Reviewed-by: Luca Weiss <luca@z3ntu.xyz>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20221111072354.3431-2-ollieparanoid@postmarketos.org%3E
Add a new command that makes running CI scripts easy. The user goes to
the git repository of choice, which has CI scripts written in a certain
format, and then runs 'pmbootstrap ci' to get an interactive selection
of which of the available scripts to run (or "all"). Specifying one or
multiple scripts on the command-line is also possible, e.g.
$ pmbootstrap ci flake8
$ pmbootstrap ci shellcheck flake8 pytest
$ pmbootstrap ci --all
pmbootstrap then either runs the selected scripts in a chroot (and
installs dependencies as defined at the beginning of the CI scripts), or
natively (with checks inside the scripts for having dependencies
installed). Running natively is needed for .ci/pytest.sh in this
pmbootstrap.git repository, as pmbootstrap can't run inside pmbootstrap.
Running natively or in chroot is defined in an "# Options: " comment
inside the script file.
Documentation for this command and how script files look like:
https://postmarketos.org/pmb-ci
When running the testsuite, most logging gets written to a separate
log_testsuite.txt file. Check if it exists, and if so, instruct tail to
print its output as well. This allows immediatelly figuring out what the
testsuite is doing without manually attaching to log_testsuite.txt
(which I often did while running the testsuite).
Add the new option that will be mandatory for all devices in
community/main category. This is just a combination of anbox + iwd +
nftables + containers + zram + netboot.
While the existing options could be removed we're keeping it for now
given that also some devices with downstream kernel might find some
options useful.
Don't pass stdin to commands that aren't supposed to be used
interactively (output: log, background, pipe).
This fixes an inconsistency between building packages in CI on gitlab
and building them via bpo on sourcehut or locally. In gitlab, apparently
there is no stdin for the entire build job and so unanswered kernel
config prompts will just use the default. In local builds and on
sourcehut stdin is available and so it just hangs at the prompt until
pmbootstrap kills the build job due to no output being written.
I considered adding an additional check to pmaports to ensure that there
are no unanswered kernel config prompts just in case users run abuild
manually on the kernel APKBUILD with stdin available. But I think
forcing the users to answer all the prompts even if it's not really
needed just creates additional work / makes the workflow worse without
real benefit.
Related: https://builds.sr.ht/~postmarketos/job/824373#task-pmbootstrap_build-432
Fixes: pmaports issue 1225
Stock bootloader on these devices boots kernel (it is intended to boot
kernel, but i place secondary bootloader there) from special ChromeOS
kernel partition on special GPT which is created with cgpt utility.
This MR adds initial support for it introducing new deviceinfo options:
- cgpt_kpart - path to file to be flashed to ChromeOS partition;
- cgpt_kpart_start - offset from the start in sectors;
- cgpt_kpart_size - partition size in sectors.
For example:
deviceinfo_cgpt_kpart="/usr/share/u-boot/google-peach-pit/u-boot-dtb.img.kpart"
deviceinfo_cgpt_kpart_start="8192"
deviceinfo_cgpt_kpart_size="32768"
cgpt requires start and size values of partition, so these values
are calculated for each partition.
Reserved size and on-device installer are not yet supported.
Reference: https://archlinuxarm.org/platforms/armv7/samsung/samsung-chromebook
pmbootstrap netboot command exposes the generated vendor-codename.img
rootfs through nbd interface so that device can mount it and boot
postmarketOS without having any storage medium at all.
Co-authored-by: Luca Weiss <luca@z3ntu.xyz>
