Drop the weird flag file stuff for state management, and just always
mount the source code in, and always unmount it on exit - including in
the error path.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
When running build --force, if a later package is a dependency of an
earlier package, it will be "visited" by the dependency resolver,
erroneously causing it to be skipped when later in the session it's
visited to force build. This is because previously visited packages are
marked as such in a hashmap, but skip_already_built() assumes that a
package which has been visited would have already been built.
Fix this by overriding skip_already_built() if doing a force build. This
works because package dependencies are only built if the APKBUILD
version is newer than the binary repo, even when --force is specified.
So there is no risk of an infinite loop here.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Some packages like kernels are cross compiled from the native chroot.
the apk_wrapper won't work here! So only enable it for buildroot chroots
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Why
Btrfs has some goodness (snapshots, switching between different rw snapshot)
which plays particularly well with certain "subvolume layouts".
What
This MR seeks to implement such a layout, namely a flat btrfs layout,
where the top level subvolume (i.e. the btrfs filesystem/partition itself)
remains unmounted in all situations,
except when making changes to direct child subvolumes of the filesystem.
- rename all subvols to follow the common @* btrfs subvol naming scheme.
- add subvol @root, because roots home directory shouldn't be rolled back.
- make subvol @var not Copy-on-Write (nodatacow) to avoid write
- multiplication on logs, VMs, databases, containers and flatpaks.
- add subvol @snapshots because that lets us change the root subvol to a
read-write snapshot of @ without affecting snapshots.
- add subvol @srv because it contains data for Web and FTP servers,
which shouldn't roll back together with root subvol.
- add subvol @tmp because we don't want to snapshot temporary files.
This subvol remains unmounted on the device,
unless conditions as laid out in pmaports!4737 are met.
- add check and error for btrfs when using rsync installation.
Speed up using abuild to install build dependencies by introducing a
wrapper which invokes abuild-apk in the native chroot with
LD_PRELOAD_PATH
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Fix "pmbootstrap chroot" and others not passing the proxy environment
variables correctly. Thanks to notfound405 for pointing this out!
Instead of only preserving proxy environment variables in
pmb.helpers.run_core, which should never be called directly, do it in
the calling functions:
* pmb.helpers.run.user
* pmb.helpers.run.root
* pmb.chroot.root
* pmb.chroot.user
This fixes that the environment variables were only really passed by
pmb.helpers.run.user, because the other functions would result in
something like:
HTTP_PROXY=mytestproxy sudo env -i /usr/bin/sh -c '…'
This is needed to either elevate to root, or to elevate to root first
and then enter the chroot as root or user. Due to the "env -i", the
environment intentionally gets cleaned, but unintentionally also removes
the proxy environment variables that were explicitly set.
By adjusting the functions, they now run a variant of:
sudo env -i /usr/bin/sh -c 'HTTP_PROXY=mytestproxy …'
The escaping is simplified in this example, run "pmbootstrap -v" to see
the not very readable, but proper escaping with shutil.quote().
Remove the previous test for preserving the environment variables in
pmb.helpers.run_core (as it should never be called directly), and test
instead the new behavior.
Fixes: issue 2299
Fixes: 13c4ac42 ("pmb.helpers.run_core: fix proxy env var logic")
find_kbuild_output_dir() searches for certain path patterns, which
won't be in the APKBUILD if it uses downstreamkernel_package. Guess
the default kbuild out dir in that case.
Theoretically this might break if some APKBUILD passes weird paths to
downstreamkernel_package, but no kernel package in pmaports today does
that.
Without this, pmbootstrap will give an error if you have a nonexistent
device selected when trying to checksum a package. This can be annoying
when switching between different branches where devices may or may not
exist, and I don't think these initialisation steps are necessary for
the checksum action anyway.
When running pmbootstrap install --android-recovery-zip it tries to
access pmb.flasher.variables but args.no_reboot and args.resume are
only defined when running pmbootstap flasher.
I zap chroots a lot, since I've found that it often "fixes" a lot of
weird issues that come about if you have stale chroots laying around.
So a common pattern I do is "pmb zap && pmb install ...". Having an
option to pmb install let's me simplify this.
Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20231214234051.4673-1-clayton@craftyguy.net%3E
Make sure the user has at least 256 MiB set as their installation size,
refuse to start the installation otherwise. The default was changed in
2021, 03e9fb05 ("pmb.config.init.boot_size: set to 256 MiB (MR 2037)").
If the user ran "pmbootstrap init" before that commit, the pmbootstrap
config will have the old default set. It is very annoying when you do an
installation with it and only realize it when you run into errors, e.g.
while upgrading. I had that when testing the upgrade to the v23.12
release and also adjusted postmarketos-release-upgrade to warn if the
boot partition is smaller than expected.
Reviewed-by: Luca Weiss <luca@z3ntu.xyz>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20231208222714.4601-1-ollieparanoid@postmarketos.org%3E
Sometimes I want to build an older version of a package from Alpine, and
since package upgrades can involve things like patches and other
externalities just changing the pkgver and running checksum in pmaports
may not be enough. As such, it tends to be easier to revert the change
in the local aports repo and then fork than forking and then trying to
manually revert the changes yourself (since you can't have git do that
for you given that they are distinct repositories).
Prior to this patch, that was not possible since pmbootstrap would
assume older aport version equals outdated aports in general and as such
cancel the whole operation. Instead, just print a warning and helpful
information to make this workflow possible while also warning users that
they may want to update their local aports.
Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20231209111813.37756-1-newbyte@postmarketos.org%3E
Having AppStream data is great for installation that make use of GNOME
Software or KDE Discover, however it's not a must. The generation of
AppStream data in alpine (which we maintain) is still certainly
improvable, and we have bumped into problems, unreliabilities and such
from time to time. Installations without AppStream data are
totally functional, even if the first experience on those apps is not
the best. Still, users are one refresh away on Software or Discover
from getting such data downloaded. So failing to generate installations
for this reason is unnecessarily breaking.
Signed-off-by: Pablo Correa Gómez <ablocorrea@hotmail.com>
Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20231205160205.13916-1-pabloyoyoista@postmarketos.org%3E
Instead of building cross packages for the native architecture, add a
new get_cross_packages_arches() function that returns the desired
architectures no matter what the current native architecture is.
So gcc-x86_64 will always have arch="aarch64", and gcc-aarch64 will
always have arch="x86_64".
We can tweak the function in the future if e.g. it becomes viable to
cross compile from riscv64. Or even better, remove this and the cross
compile packages upstreamed into alpine (see pmb#551).
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20231126104823.3723-1-ollieparanoid@postmarketos.org%3E
If we found an APKBUILD already, with the exact name of the package we
are looking for, then don't also guess what other package could contain
the package as subpackage.
The guess was thrown away with the previous logic, but it caused
misleading verbose logs like:
lomiri-history-service: guessed to be a subpackage of lomiri
Also this is slightly faster. While at it, tweak the related comments
to make this easier to understand.
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20231121222457.2871-1-ollieparanoid@postmarketos.org%3E
Fix that APKBUILDs mentioning other packages with an operator could not
be found. As we are building the initial branch of v23.12, this
currently happens with postmarketos-mkinitfs: it depends on
devicepkg-utils>=0.2.0 and currently pmbootstrap will only remove the
>=0.2.0 when looking for the pkgname in the APKINDEX of binary packages
(which is why it works on master). But it does not yet do that when
looking for the pkgname in pmaports.
Move the code for stripping the operator to a common place and use it
for getting packages from pmaports too.
Change the order of operators while at it, try to find <= before =, as
otherwise it would cut off example<=1.2.3 as "example<" instead of
"example".
Reviewed-by: Caleb Connolly <kc@postmarketos.org> (via chat)
If a package had _pmb_select set and an appropriate provider configured
in pmb's cfg, it was not being used when providers were being resolved.
With this change, those are now being considered. The order is also
important, we want a selected provider to be chosen before it tries to
make a decision based on provider_priority later in step #6.
Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20231122081214.11872-2-clayton@craftyguy.net%3E
The sdcard option can not only be used to write to SD cards, but also
for writing to USB sticks. Or even to write to internal storage if
pmbootstrap runs on a system booted from a live disk / seconds storage.
Drop the check. Thanks to Raymond Hackley for pointing out that this
check is not useful.
Reviewed-by: Anton Bambura <jenneron@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20231101193702.2223-1-ollieparanoid@postmarketos.org%3E
This fixes an issue when pmb is run on an Alpine/pmOS host and apk.static sees
that /etc/apk/interactive is set on the host. It's really annoying to have the
build pause so apk can prompt when building chroots, so this uses a flag to apk
to disable interactive mode. I can't think of any situations where we would
actually want to prompt users when building chroots, by then all user
configuration should have been completed via pmb init/config.
Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Co-developed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20231110185320.8358-1-clayton@craftyguy.net%3E
Currently, pmbootstrap checks if either the kernel or the ramdisk in a boot.img
contains the MediaTek header, and if one does, it assumes both do. It hardcodes
the label KERNEL for the kernel and ROOTFS for the ramdisk.
My Amazon Echo Dot (gen 2) has a boot.img where only the kernel has the header,
but not the ramdisk. These changes (as well as those in my new boot-deploy MR)
account for that situation (and any possible label an image has) by splitting
bootimg_mtk_mkimage into two variables for the kernel and the ramdisk labels.
Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/patches/46351
It can be quite useful to check the configs for a list of packages, so
adjust the argument parsing code to support that.
Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Tested-by: Oliver Smith <ollieparanoid@postmarketos.org>
Reviewed-by: Clayton Craft <clayton@craftyguy.net>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20231028085901.35205-1-luca@z3ntu.xyz%3E
This is required for being able to boot compressed kernels via EFI stub
on aarch64.
As far as I can tell, EFI_ZBOOT was added in a050910972bb2, and
available in 6.1.0 or later. I'm not sure if it was backported to other
older LTS versions or not...
Reviewed-by: Anton Bambura <jenneron@postmarketos.org>
Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Tested-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20231011212433.17131-1-clayton@craftyguy.net%3E
Cache the compiler output of rust code with sccache, like we use ccache
for c code.
I've considered using sccache to completely replace ccache since it can
cache output of C/C++ code too. But let's not do it for now since ccache
doesn't need to run a daemon in the background that needs to be stopped
when shutting down / zapping. Also it would need more refactoring.
Reviewed-by: Luca Weiss <luca@z3ntu.xyz>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230806184729.4891-5-ollieparanoid@postmarketos.org%3E
