Commit graph

2115 commits

Author SHA1 Message Date
Oliver Smith
d90a037fd6
pmb.install.format: set bytes-per-inode to 16384 (MR 2558)
After setting "-T big", we still saw some images failing with "out of
space" errors in bpo, such as "v24.12:samsung-espresso10:xfce4".

I've looked at /etc/mke2fs.conf to figure out what exactly gets
configured with "-T big". What I did not realize is that the inode
ratio gets bigger by going from small -> big, big -> huge, but with a
larger ratio FEWER inodes will be created (see man mkfs.ext4, -i
bytes-per-inode).

In /etc/mke2fs.conf in Alpine edge:

	[defaults]
	…
		blocksize = 4096
		inode_size = 256
		inode_ratio = 16384

	[fs_types]
	…
		small = {
			blocksize = 1024
			inode_ratio = 4096
		}
		floppy = {
			blocksize = 1024
			inode_ratio = 8192
		}
		big = {
			inode_ratio = 32768
		}
		huge = {
			inode_ratio = 65536
		}
		news = {
			inode_ratio = 4096
		}
	…

So I've tried out half the ratio value we get with "big" now, 16384.
With that we finally seem to have:

* Enough inodes for small UIs (console, none) as well as big UIs with
  lots of files (xfce4, plasma-desktop, ...)

* We don't directly specify an inode count (-N) anymore as we did
  earlier, which had problems with small images as the inode count would
  be too large for the given image size (pmb#2572).

Test builds with various images:

Filesystem              Inodes      Used Available Use% Mounted on
master:postmarketos-trailblazer:none:
/dev/installp2           35440      5111     30329  14% /mnt/install
master:postmarketos-trailblazer:console:
/dev/installp2           47232      8713     38519  18% /mnt/install
master:pine64-pinebookpro:plasma-desktop:
/dev/installp2          276352    100607    175745  36% /mnt/install
v24.12:samsung-espresso10:xfce4:
/dev/installp2           80960     51150     29810  63% /mnt/install

So the most Use% I could get is 63% with this, leaving a nice margin for
the future.
2025-02-23 13:45:36 +01:00
Oliver Smith
06957b2173
install: log space/inodes used in image (MR 2558)
Log how well we have used the space and available inodes in the
resulting images, to show how well the mkfs parameters we have decided
on are working.
2025-02-23 13:45:32 +01:00
Stefan Hansson
ebc5ab9ecd
pmb.chroot: Rename exists() to user_exists() (MR 2545)
This name is what's used at the one call site the codebase currently
has, and it means we can have an explicit re-export which Ruff won't
complain about.
2025-02-23 11:40:44 +01:00
Stefan Hansson
bdcc188f05
pmb: Use explicit re-exports (MR 2545)
See https://docs.astral.sh/ruff/rules/unused-import
2025-02-23 11:40:44 +01:00
Stefan Hansson
0620232819
pmb: Remove unused imports (MR 2545) 2025-02-23 11:40:40 +01:00
Oliver Smith
63fce4fcb5
pmb.install.format: set ext4 usage-type to big (MR 2555)
Using "-N" to set a specific inode count caused errors:
* With 100K, we got "out of space" errors when creating images for
  bigger UIs with many small files like:
  pine64-pinebookpro:plasma-desktop
* With 1M (current value), we got "out of space" for small images like:
  postmarketos-trailblazer:console

Use "-T big" instead to set the usage-type that is closest to how we
expect the filesystem to be used, once it is flashed to the user's
device and resized. This will indirectly set the bytes-per-inode. I've
verified that building both the big
pine64-pinebookpro:plasma-desktop and small
postmarketos-trailblazer:console (and even with UI=none) works with
this change.

I've verified that building both the small image works with this.

From the mkfs.ext4 man page:

       -T usage-type[,...]
              Specify how the file system is going to be used, so that mke2fs
              can choose optimal file system parameters for that use.  The
              usage types that are supported are defined in the configuration
              file /etc/mke2fs.conf.  The user may specify one or more usage
              types using a comma separated list.

              If this option is is not specified, mke2fs will pick a single
              default usage type based on the size of the file system to be
              created.  If the file system size is less than 3 megabytes,
              mke2fs will use the file system type floppy.  If the file system
              size is greater than or equal to 3 but less than 512 megabytes,
              mke2fs(8) will use the file system type small.  If the file
              system size is greater than or equal to 4 terabytes but less
              than 16 terabytes, mke2fs(8) will use the file system type big.
              If the file system size is greater than or equal to 16
              terabytes, mke2fs(8) will use the file system type huge.
              Otherwise, mke2fs(8) will use the default file system type
              default.

Fixes: d0d7c2d4 ("pmb.install.format: set ext4 usage-type to big")
Fixes: issue 2572
2025-02-19 19:49:17 +01:00
Caleb Connolly
67773c4293
chroot: add --usb flag to make usb devices available (MR 2554)
Add a flag to bind-mount in the necessary sysfs directories for USB
devices to be accessed inside the chroot. This is the same as how we do
it for "pmbootstrap flasher" but allows running arbitrary commands in
the chroot.

This is useful to allow using pmbootstrap chroot's as a sandbox for
custom flashing procedures.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2025-02-16 22:16:11 +01:00
Anri Dellal
192a29fb1b
pmbootstrap config: Fix setting attribute value to False (MR 2546)
_type(value) in super().__setattr__(key, _type(value)) always returns
True for a valid string, even if the string is "False".

This code also didn't fail when any other value besides True or False
was passed, though it must fail.

Fix #2554
2025-02-16 20:41:18 +01:00
Oliver Smith
ffa58f402c
pmb.install.format: set ext4 inodes to 1M (MR 2552)
Make 10x the amount of inodes available for ext4 partitions when using
"pmbootstrap install" without the "--disk" argument so we won't run out
for sure. The previous value was not enough in some edge cases. We do
need to set this as the default is not enough either.

Fixes: issue 2568
2025-02-16 18:08:03 +01:00
Hugo Osvaldo Barrera
5a152aebae
test: move all tests into separate module (MR 2551)
When installing pmboostrap, all tests are installed alongside it. There
doesn't seem to be any way to selectively exclude some files inside
python modules, so move all test into a separate module instead.

This is the typical convention in python projects.

See: https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/79862
Fixes: https://gitlab.postmarketos.org/postmarketOS/pmbootstrap/-/issues/2541
2025-02-13 01:28:20 +01:00
Hugo Osvaldo Barrera
c2bcc81589
pmb.config.init: prompt for pmaports path (MR 2502)
Allows easily using an existing checkout. The default path remains the
same.

Fixes: https://gitlab.postmarketos.org/postmarketOS/pmbootstrap/-/issues/2506
[ci:skip-build]: already built successfully in CI
2025-02-12 13:50:13 +01:00
Hugo Osvaldo Barrera
aed6c7f61e
pmb.config.init: resolve double-negation (MR 2502) 2025-02-12 13:50:13 +01:00
Newbyte
cda5d61593
pmb.config.init: Reformat some strings (MR 2535)
We extended the maximum line length at some point, so let's make use of
it.

[ci:skip-build]: already built successfully in CI
2025-02-10 13:22:07 +00:00
Newbyte
86aec42713
pmb.config.init: Improve message when "Archived:" header is missing (MR 2535)
This changes the output from e.g.

    WARNING: xiaomi-markw is archived: None

to

    WARNING: xiaomi-markw is archived: No reason given (this is a bug)

which I think is better and makes it clear that this is not behaviour
one should rely upon. All archived ports should have a reason given, but
this has not been (and still isn't as I'm writing this) enforced in
pmaports, so we can't rely on there being one.

Related: https://gitlab.postmarketos.org/postmarketOS/pmaports/-/merge_requests/6085
2025-02-10 13:22:07 +00:00
Newbyte
1965818fae
pmb.helpers.logging: Use PEP 8-compliant naming (MR 2537)
See https://docs.astral.sh/ruff/rules/invalid-class-name
2025-02-02 22:48:51 +01:00
Newbyte
97da7ebe4e
pmb.conftest: Use PEP 8-compliant naming (MR 2537)
See https://docs.astral.sh/ruff/rules/non-lowercase-variable-in-function
2025-02-02 22:48:51 +01:00
Newbyte
70411ce3c4
pmb.install.losetup: Remove unnecessary range start (MR 2537)
See https://docs.astral.sh/ruff/rules/unnecessary-range-start
2025-02-02 22:48:50 +01:00
Newbyte
6415aea8af
pmb: Remove unnecessary pass statements (MR 2537)
See https://docs.astral.sh/ruff/rules/unnecessary-placeholder
2025-02-02 22:48:50 +01:00
Newbyte
59227a333e
pmb.aportgen.core: Combine startswith(...) calls into one (MR 2537)
This is equivalent and simpler.

See https://docs.astral.sh/ruff/rules/multiple-starts-ends-with
2025-02-02 22:48:50 +01:00
knuxify
00c670ffda
pmb.flasher: Fix passing command line flags (MR 2543)
Commit 37ec73c dropped the use of args to access command line arguments,
but did not add another way to pass command line arguments; thus,
optional flags to flash_kernel, flash_rootfs, flash_lk2nd, sideload and
list_devices were ignored.

Fix this by making sure the relevant arguments are passed for all
frontend methods.

Fixes #2522.
2025-02-01 22:38:55 +01:00
SzczurekYT
b5f14e2b94
install: create_device_rootfs: check if ui extras package exists (MR 2534)
Don't try installing the ui -extras subpackage if the currently selected ui doesn't have one.
2025-01-28 11:41:33 +01:00
Oliver Smith
afecbb151d
Prepare 3.2.0 release 2025-01-19 14:08:06 +01:00
Newbyte
016dbe42fe
pmb.chroot.binfmt: Initialise chroot before installing packages in register() (MR 2526)
Closes https://gitlab.postmarketos.org/postmarketOS/pmbootstrap/-/issues/2453
2025-01-19 13:54:31 +01:00
Newbyte
a01578d897
pmb.parse.bootimg: Initialise chroot before installing packages in bootimg() (MR 2526)
Closes https://gitlab.postmarketos.org/postmarketOS/pmbootstrap/-/issues/2535
2025-01-19 13:54:31 +01:00
Newbyte
9a31fe43b3
pmb.config.other: Don't require selected UI package to exist (MR 2530)
Without this, `$ pmbootstrap init` crashes if a nonexistent UI package
is selected. This can happen if one previously selected a UI package in
a former checkout of pmaports, but then switched away to a different one
that doesn't have the UI package. For example, if someone is working on
a new UI in a feature branch and then switches away from it, or if a UI
is removed from the canonical pmaports repository altogether.
2025-01-16 12:25:02 +01:00
Newbyte
5a2eafd3af
pmb.helpers.ui: Add must_exist argument to check_option() (MR 2530)
Will be used in the next commit.
2025-01-16 12:22:28 +01:00
Newbyte
01faedcf37
pmb.types: Fix type for PmbArgs.output (MR 2522)
And remove hack needed to accomodate this wrong type.
2025-01-15 23:25:57 +01:00
Newbyte
8e09be579a
pmb.build.autodetect: Remove FIXME about type hinting arch (MR 2522)
This has been done now.
2025-01-15 23:25:57 +01:00
Newbyte
78cddc90f1
pmb.qemu.run: Remove FIXME and type ignore stuff from command_qemu() (MR 2522)
This was probably fixed by 225d8b30a0 as
it added an Env type hint to the env variable. Either way, the mypy
check passes evenn with this removed now.
2025-01-15 23:25:57 +01:00
Eisenbahnfan
a84b3eb4d2
Fix pmbootstrap install --android-recovery-zip (MR 2527)
pmbootstrap install --android-recovery-zip always failed with:
ERROR: The recovery zip has not been generated yet, please run
'pmbootstrap install' with the '--android-recovery-zip' parameter first!

I debugged it with a simple print statement of the full path and saw
a repetition of the first part of the path. I discovered that the
mountpoint is "/mnt/" + chroot, so adding chroot again leads to a
repetition of the first part of the path and to a non-existent location.

Closes: issue 2441
2025-01-15 23:20:19 +01:00
Oliver Smith
6a776fc67c
helpers.run.core.add_proxy_env_vars: fix variables (MR 2525)
Looks like there was a missing comma when this was added initially.
Caught this while reviewing the formatting patch.
2025-01-15 22:55:27 +01:00
Newbyte
3061e702ab
pmb: Reformat with Ruff 0.9 (MR 2525)
See https://astral.sh/blog/ruff-v0.9.0
2025-01-15 22:35:02 +01:00
Newbyte
2be49f8caf
pmb.install: Remove unnecessary calls to round() (MR 2525)
These values are already integers, so calling round() on them is
effectively a no-op.
2025-01-15 22:35:02 +01:00
Newbyte
01abf537bb
pmb.config.init: Simplify removal of select from providers_cfg (MR 2525)
See https://docs.astral.sh/ruff/rules/if-key-in-dict-del
2025-01-15 22:35:02 +01:00
Newbyte
7037d9ab46
pmb.helpers.aportupgrade: Remove unnecessary regex (MR 2525)
See https://docs.astral.sh/ruff/rules/unnecessary-regular-expression
2025-01-15 22:35:02 +01:00
Newbyte
d6f0c80431
pmb: Use raw strings for regexes (MR 2525)
See https://docs.astral.sh/ruff/rules/unraw-re-pattern
2025-01-15 22:35:02 +01:00
Newbyte
077efe3623
pmb.chroot.apk: Use unpacking operator to create command in install_run_apk() (MR 2525)
The unpacking operator can unpack any iterable of any type, unlike the +
operator which only works on some sequences and requires them to be of
the same type usually. Using it allows us to get rid of the line that
disables the "operator" error code for this function.
2025-01-15 22:35:02 +01:00
Newbyte
c797b30dfe
pmb: Use unpacking operator to concatenate collections (MR 2525)
See https://docs.astral.sh/ruff/rules/collection-literal-concatenation

This is also slightly faster according to a microbenchmark that
shows it taking around 19% less time to run:
https://github.com/astral-sh/ruff/pull/1957#issue-1538092351
2025-01-15 22:35:02 +01:00
Newbyte
4868cc9548
pmb.core.config: Add note about aports variable being class variable (MR 2525)
See https://docs.astral.sh/ruff/rules/mutable-class-default
2025-01-15 22:35:02 +01:00
Newbyte
e36801cc81
pmb.core.config: Mark mirrors and providers as class variables (MR 2525)
While it probably wasn't the intent, this is what they are in practice.
Mark them as such to help mypy.

See https://docs.astral.sh/ruff/rules/mutable-class-default
2025-01-15 22:35:02 +01:00
Newbyte
38b333372d
pmb.helpers.locale: Use frozenset for primary_layouts (MR 2525)
This shouldn't change at runtime, so avoid any potential issues with it
being a mutable class variable by making it immutable.
2025-01-15 22:35:02 +01:00
Newbyte
678ce1db52
pmb.build._package: Replace sum() with functools.reduce() (MR 2525)
See https://docs.astral.sh/ruff/rules/quadratic-list-summation
2025-01-15 22:35:02 +01:00
Newbyte
f43af8a115
pmb.commands.repo_bootstrap: Remove unnecessary noqa comment (MR 2525)
Not sure why this was added, but Ruff doesn't complain if I remove it.
2025-01-15 22:35:02 +01:00
Newbyte
5d707e154c
pmb.helpers.run_core: Remove unnecessary str(...) (MR 2525)
Integers automatically get converted to strings when used in f-strings
like this, so just get rid of this explicit conversion.
2025-01-15 22:35:02 +01:00
Newbyte
c2787aa76f
pmb: Replace list(...)[0] with next(iter(...)) (MR 2525)
See https://docs.astral.sh/ruff/rules/unnecessary-iterable-allocation-for-first-element
2025-01-15 22:35:02 +01:00
Newbyte
752c1df039
pmb.build.other: Fix broken f-string (MR 2525)
I'm guessing this was meant to be an f-string given that it uses the
formatting syntax for those.
2025-01-15 22:34:57 +01:00
Oliver Smith
3f9f1c96c1
install: support systemd for disabling services (MR 2529)
Write a preset file and run "systemctl preset" afterwards:

    % sh -c echo 'disable sshd.service' > /usr/lib/systemd/system-preset/80-pmbootstrap-install-disable-sshd.preset
    % systemctl preset sshd.service
    % systemctl is-enabled sshd.service
    disabled
2025-01-12 15:33:02 +01:00
Oliver Smith
786c62767e
install: merge disable_sshd(), disable_firewall() (MR 2529)
These functions are identical, except for the service name. Merge them
in preparation to add systemd support.
2025-01-12 15:33:02 +01:00
Robert Eckelmann
a2f177433f
Cosmetic: pmb/install/_install.py: tweak strings (MR 2529)
Merge strings that have a break in the same line. These were generated
when running "ruff format" initially.

Old:
  logging.info("Firewall is not supported in checked out pmaports" " branch.")
New:
  logging.info("Firewall is not supported in checked out pmaports branch.")

Signed-off-by: Robert Eckelmann <longnoserob@postmarketos.org>
Tweaked-by: Oliver Smith <ollieparanoid@postmarketos.org>
2025-01-12 15:32:57 +01:00
Oliver Smith
eb34843f4b
Update systemd repo URL
Now that systemd was merged into master, it is extra-repos/systemd
instead of staging/systemd.

As of writing, the repository is not published yet but this will happen
soon.

Related: https://postmarketos.org/edge/2025/01/09/systemd-soon/
2025-01-10 19:59:50 +01:00