Add a new environment variable that disables apk's caching feature for
space constrained environments such as the bpo image build jobs.
Currently we have a workaround in place in bpo: the apk cache is moved
to a tmpfs. But this is fragile and just disabling the apk cache is a
more elegant solution.
I've decided to make this an env var instead of a full pmbootstrap
option since this option is not relevant for normal users, only for CI
jobs in space constrained environments. Also we already have another
`PMB_APK_` env var.
Related: bpo issue 136
Part-of: https://gitlab.postmarketos.org/postmarketOS/pmbootstrap/-/merge_requests/2553
* Make CrossCompile a proper enum type rather than a string literal,
* Introduce methods to get the correct host/build chroots depending on the
cross compile type and target architecture.
* Remove autodetect.chroot() since it doesn't do what we expect, adjust
all users to use cross.build_chroot() instead.
* Refactor package building to correctly use cross.host_chroot() and
cross.build_chroot().
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Part-of: https://gitlab.postmarketos.org/postmarketOS/pmbootstrap/-/merge_requests/2568
Replace None with the three things it is actually used for to make the
code easier to follow:
* "autodetect"
* "unnecessary"
* "qemu-only"
I've used the term "unnecessary" instead of "native", because "native"
was previously used for "cross-native", then "cross-native2" and it
still sounds similar to these.
Part-of: https://gitlab.postmarketos.org/postmarketOS/pmbootstrap/-/merge_requests/2568
This adds support for using an environment variable to set the FDE
passphrase, allowing us to automate image creation when using FDE.
The method used here was borrowed from how we set the password with when
using the --password arg: write to a temp file, call something in the
chroot to read/use it, then remove it.
Part-of: https://gitlab.postmarketos.org/postmarketOS/pmbootstrap/-/merge_requests/2538
Currently packages that are not in the systemd channel cannot be
sideloaded when using the systemd channel. This is because apk sideload
uses the global aports channel and not the channel of the package. Fix
it with this patch.
We fully expect to reach this code path within normal use of
pmbootstrap. It's not a bug if we get here. As such, use NonBugError to
indicate that and clean up the output.
We fully expect to reach this code path within normal use of
pmbootstrap. It's not a bug if we get here. As such, use NonBugError to
indicate that and clean up the output.
We fully expect to reach this code path within normal use of
pmbootstrap. It's not a bug if we get here. As such, use NonBugError to
indicate that and clean up the output.
The install action will initialise the native chroot, so this way we
don't have to initialise the chroot in this code path. It also gives
users feedback quicker if they haven't already initialised the rootfs
since this way they don't have to wait around for nbd to be installed
before being told that they need to generate the rootfs first.
Closes https://gitlab.postmarketos.org/postmarketOS/pmbootstrap/-/issues/2549
Cross compiling go programs with cross-native2 fails for armv7, riscv64
and armhf unless we use CGO_ENABLED=1 in the APKBUILD (see related
aports issue).
Add --sysroot to CGO_CFLAGS, so cgo can find the includes and doesn't
fail with:
go build -buildmode=pie -modcacherw -trimpath -buildvcs=false -ldflags "-s -w -X main.Version=2.6.1" -o mkinitfs ./cmd/mkinitfs
# runtime/cgo
_cgo_export.c:3:10: fatal error: stdlib.h: No such file or directory
Add it to LDFLAGS too for linking against libraries from the sysroot.
Related: https://gitlab.alpinelinux.org/alpine/aports/-/issues/15809
Related: https://stackoverflow.com/a/38835938
Fixes a bug where checkdepends are not being installed, and packages
that run tests that depend on something in checkdepends fail because
it's missing. For example, the `plasma-workspace` forked pkg in pmaports
tells abuild to run `xwfb-run` in `check()`, this binary comes from
`xwayland-run`, which is listed in the `checkdepends`. However this
isn't being installed and so `pmb build` ultimately fails to build the
package.
Set things up so that we can run abuild on the native chroot and use
it's cross compilation features rather than running it and the build
system through QEMU. This massively speeds up building when it works.
cross-native used to be quite limited in functionality and didn't
integrate into abuild itself, this commit fixes that.
Packages can opt-in to this by adding pmb:cross-native2 to their options
and configuring makedepends_host and makedepends_build.
This also speeds up building packages like postmarketos-initramfs since
it entirely avoids running commands through QEMU (usually abuild itself
would be run through QEMU).
Lastly, we preserve the old pmb:cross-kernel options, this can be used to
enable the old cross compiler behaviour which is used for cross
compiling kernels in pmaports. This allows them to keep being supporting
while we adapt them to the new cross-native2.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
The .supported() method errs on the side of exposing too many
architectures to make porting for new or less-common platforms easier.
It therefore isn't suitable for using as a list of supported
architectures we can probe the binary repository for.
Introduce a .supported_binary() method to export only the architectures
where we have a binary repository.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Instead of automatic python package discovery that for some reason still
picks up the test module even though it should be excluded, specify the
"pmb" module manually to be installed.
Follow-up to MR 2551.
Fixes: issue 2541
I didn't really understand mypy's overloading well when I wrote this. It
makes sense to have a third overload here with bool as type for
multiple_providers used in circumstances where a literal isn't provided
for that parameter.
According to Python documentation, p.match checks
if characters at the beginning of string match the regex
This commit changes validation to full length of string
- Properly run commands at host to mount rootfs
- Always refuse to zap chroot_image, since it is never added to workdir.cfg
Co-authored-by: Newbyte <newbyte@postmarketos.org>
After setting "-T big", we still saw some images failing with "out of
space" errors in bpo, such as "v24.12:samsung-espresso10:xfce4".
I've looked at /etc/mke2fs.conf to figure out what exactly gets
configured with "-T big". What I did not realize is that the inode
ratio gets bigger by going from small -> big, big -> huge, but with a
larger ratio FEWER inodes will be created (see man mkfs.ext4, -i
bytes-per-inode).
In /etc/mke2fs.conf in Alpine edge:
[defaults]
…
blocksize = 4096
inode_size = 256
inode_ratio = 16384
[fs_types]
…
small = {
blocksize = 1024
inode_ratio = 4096
}
floppy = {
blocksize = 1024
inode_ratio = 8192
}
big = {
inode_ratio = 32768
}
huge = {
inode_ratio = 65536
}
news = {
inode_ratio = 4096
}
…
So I've tried out half the ratio value we get with "big" now, 16384.
With that we finally seem to have:
* Enough inodes for small UIs (console, none) as well as big UIs with
lots of files (xfce4, plasma-desktop, ...)
* We don't directly specify an inode count (-N) anymore as we did
earlier, which had problems with small images as the inode count would
be too large for the given image size (pmb#2572).
Test builds with various images:
Filesystem Inodes Used Available Use% Mounted on
master:postmarketos-trailblazer:none:
/dev/installp2 35440 5111 30329 14% /mnt/install
master:postmarketos-trailblazer:console:
/dev/installp2 47232 8713 38519 18% /mnt/install
master:pine64-pinebookpro:plasma-desktop:
/dev/installp2 276352 100607 175745 36% /mnt/install
v24.12:samsung-espresso10:xfce4:
/dev/installp2 80960 51150 29810 63% /mnt/install
So the most Use% I could get is 63% with this, leaving a nice margin for
the future.
This name is what's used at the one call site the codebase currently
has, and it means we can have an explicit re-export which Ruff won't
complain about.
Using "-N" to set a specific inode count caused errors:
* With 100K, we got "out of space" errors when creating images for
bigger UIs with many small files like:
pine64-pinebookpro:plasma-desktop
* With 1M (current value), we got "out of space" for small images like:
postmarketos-trailblazer:console
Use "-T big" instead to set the usage-type that is closest to how we
expect the filesystem to be used, once it is flashed to the user's
device and resized. This will indirectly set the bytes-per-inode. I've
verified that building both the big
pine64-pinebookpro:plasma-desktop and small
postmarketos-trailblazer:console (and even with UI=none) works with
this change.
I've verified that building both the small image works with this.
From the mkfs.ext4 man page:
-T usage-type[,...]
Specify how the file system is going to be used, so that mke2fs
can choose optimal file system parameters for that use. The
usage types that are supported are defined in the configuration
file /etc/mke2fs.conf. The user may specify one or more usage
types using a comma separated list.
If this option is is not specified, mke2fs will pick a single
default usage type based on the size of the file system to be
created. If the file system size is less than 3 megabytes,
mke2fs will use the file system type floppy. If the file system
size is greater than or equal to 3 but less than 512 megabytes,
mke2fs(8) will use the file system type small. If the file
system size is greater than or equal to 4 terabytes but less
than 16 terabytes, mke2fs(8) will use the file system type big.
If the file system size is greater than or equal to 16
terabytes, mke2fs(8) will use the file system type huge.
Otherwise, mke2fs(8) will use the default file system type
default.
Fixes: d0d7c2d4 ("pmb.install.format: set ext4 usage-type to big")
Fixes: issue 2572
Don't try to modify a variable in the while loop, as it runs in a
subshell because of the pipe. Fail on the first error instead, like we
have it in .ci/shellcheck.sh.
This can be optimized later on (for both scripts) if we really want to.
Fix for:
In ./.ci/markdown.sh line 17:
markdownlint-cli "$file" || MDL_FAILED=1
^--------^ SC2030 (info): Modification of MDL_FAILED is local (to subshell caused by pipeline).
In ./.ci/markdown.sh line 20:
if [ "$MDL_FAILED" = "1" ]; then
^---------^ SC2031 (info): MDL_FAILED was modified in a subshell. That change might be lost.
Move the "npm install" command to the section that runs as regular user
instead of root. Instead of creating a /usr/bin/markdownlint-cli
symlink, run the JS file from the path where npm installs it. Try to
find a markdownlint-cli executable and use it instead if it exists, in
case users run this script outside of "pmbootstrap chroot" and have the
linter installed under that name.
Add a pre-commit hook that runs ruff.sh and tells you how to auto-fix
issues, add a section to CONTRIBUTING.md to document it.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Add a flag to bind-mount in the necessary sysfs directories for USB
devices to be accessed inside the chroot. This is the same as how we do
it for "pmbootstrap flasher" but allows running arbitrary commands in
the chroot.
This is useful to allow using pmbootstrap chroot's as a sandbox for
custom flashing procedures.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
_type(value) in super().__setattr__(key, _type(value)) always returns
True for a valid string, even if the string is "False".
This code also didn't fail when any other value besides True or False
was passed, though it must fail.
Fix#2554
Make 10x the amount of inodes available for ext4 partitions when using
"pmbootstrap install" without the "--disk" argument so we won't run out
for sure. The previous value was not enough in some edge cases. We do
need to set this as the default is not enough either.
Fixes: issue 2568
