Commit graph

462 commits

Author SHA1 Message Date
Caleb Connolly
d4d7ba0368
test: parse: apkindex: check that cache works (MR 2455)
Add a basic test to ensure that the cache does get hit as we expect.
This doesn't test for cache conflicts.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-10-27 16:42:54 +01:00
Caleb Connolly
12f9e43a08
parse: apkindex: don't require python 3.12 (MR 2455)
Use a different way of keying the apkindex path cache.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-10-27 16:42:49 +01:00
Caleb Connolly
8627ee7e73
test: parse: add test_apkindex (MR 2448)
Add a test to parse some APKINDEX data.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-10-23 23:22:04 +02:00
Caleb Connolly
a874025bbe
parse: apkindex: fix provider_priority type (MR 2448)
should be an int, but was always a string...

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-10-23 23:17:03 +02:00
Clayton Craft
2305cc5db6
pmb.parse.apkindex: fix crash when origin or timestamp are unset (MR 2449)
Sometimes these fields are not set, e.g.:

pdb> p ret
{'depends': ['build-base', 'bash', 'bison', 'findutils', 'flex', 'musl-dev', 'openssl-dev', 'perl', 'python3'], 'arch': 'noarch', 'version': '20241022.022230', 'pkgname': '.makedepends-linux-lenovo-21bx', 'provides': []}

I hit this crash when I installed the build deps for the
linux-lenovo-21bx package in the native chroot, and then ran
`pmbootstrap stats`
[ci:skip-build]: already built successfully in CI
2024-10-22 11:11:28 +02:00
Newbyte
11c7d23575
pmb.parse.arguments: Make -x and -n mutually exclusive in kconfig edit (MR 2346)
It doesn't make sense to both request the nconfig UI and the xconfig UI
at the same time. Make them mutually exclusive.
2024-10-22 00:04:33 +02:00
Luca Weiss
35cd64086f
Improvements to documentation (MR 2440)
* Make sure all Python modules are documented by adding a bit of shell
  to .ci/docs.sh
* Remove non-existing module references from .rst
* Fix various warnings from sphinx by adjusting Python docstrings
* Add class member docs to ApkindexBlock
2024-10-21 21:46:26 +02:00
Luca Weiss
327010cff1
pmb.parse.apkindex: Add missing type hints in package() (MR 2438)
Make sure to annotate the parameters.
2024-10-21 21:05:18 +02:00
Clayton Craft
47dc493701
pmb.parse.depends.package_provider: add type hinting and fix references (MR 2439)
pmb.parse.apkindex.providers now returns a dict of string:ApkindexBlock,
probably from 71772b9b6

This adds type hinting to package_provider to reflect this, and fixes
code that calls it to properly deal with the possible return types.
Without this, building FDE images is broken.
2024-10-18 13:45:57 +02:00
Stefan Hansson
71772b9b6b
pmb.parse.apkindex: Introduce proper typing (MR 2425)
And adjust other code.

Closes https://gitlab.postmarketos.org/postmarketOS/pmbootstrap/-/issues/2455
2024-10-13 20:12:34 +02:00
Stefan Hansson
fb6073e835
pmb.parse.arguments: Remove unnecessary quotes (MR 2419)
This is probably a leftover from automatically reformatting this file.
2024-09-30 18:51:44 +02:00
Stefan Hansson
742300b090
pmb.commands: Add pkgver_bump (MR 2415)
Similar to pkgrel but for pkgver. Useful when dealing with e.g.
metapackages.

[ci:skip-build]: already built successfully in CI
2024-09-30 12:45:21 +02:00
Caleb Connolly
fbdc4013aa
parse: apkindex: add user_repository arg to package() (MR 2388)
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Tweaked-by: Oliver Smith <ollieparanoid@postmarketos.org>
2024-09-17 02:27:44 +02:00
Caleb Connolly
7f536b3057
parse: apkindex: mention that package() can't be cached (MR 2388)
tried this, it leads to hard-to-debug errors!

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-09-17 02:27:44 +02:00
Oliver Smith
479bad049f
kconfig check: don't print URL for each warning (MR 2399)
Instead of printing an URL with each warning, print one once at the
end. This gives less cluttered output, and we don't have information
about each and every option on that wiki page anyway (and given that we
now have so many options, I don't think it's good use of time to add it
there.)
2024-09-08 00:09:16 +02:00
fossdd
b65d677d34
pmb.{aportgen,parse}: deprecate deviceinfo_keyboard (MR 2396)
See https://gitlab.com/postmarketOS/pmaports/-/issues/3145
2024-09-07 18:04:10 +02:00
Caleb Connolly
f6f503035b
install: add a --sector-size flag (MR 2391)
This can be used when building images for generic device targets that
support devices with different sector size requirements.

For example, trailblazer prebuilts are currently expected to be flashed
to a USB drive where a 4096 sector size would be unsuitable since the
bootloader wouldn't detect it. But when building for a Qualcomm phone,
one would use --split and --sector-size to build the root and boot
partitions with a 4k sector size which is appropriate to the UFS
storage.

This flag could also be used by BPO to build both variants.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-08-25 22:06:17 +02:00
Oliver Smith
f1afdeaaa1
kconfig check: add --keep-going argument (MR 2384)
Abort the "pmbootstrap kconfig check" on the first error, unless the
--keep-going argument was passed. This makes it easier to go through the
failed kernels one by one until they are all fixed.
2024-08-12 19:32:10 +02:00
Anri Dellal
170829326c
kconfig: edit,migrate: use single package as argument (MR 2387) 2024-07-30 21:08:36 +03:00
Anri Dellal
08758bef90
parse: deviceinfo: fix "deviceinfo.dtb missing" error (MR 2381)
Fixes #2416
2024-07-23 11:37:53 +00:00
Oliver Smith
ebfda16d6d
args: remove -m and -mp (MR 2371)
Remove arguments to set postmarketOS and Alpine mirrors.

The current implementation is broken with the way the mirrors are now
configured: The current implementation fills a list, but the mirror
configuration code expects a string. It is not longer possible to set
an arbitrary amount of mirrors for the postmarketOS mirror. The benefit
is that we don't have comma separated values in the config file anymore
but instead just simple strings. Now it is possible to have one proper
mirror for aports, pmaports and pmaports_systemd, and one _custom one
to override each of these.

We already have too many options in "pmbootstrap -h", and changing a
mirror is more of a base configuration that you do once and maybe change
a few times, but don't need to be able to set it with each pmbootstrap
run. Users are not going to write this out manually on their
command-lines, it only makes sense in scripts and wrappers for
pmbootstrap, and there you could as well use another method (as now
described in docs/mirrors.md) to set the mirrors.

Less important, but another reason is that using "-mp" (two letters as
short argument) isn't really elegant.
2024-07-22 11:54:45 +02:00
Oliver Smith
36dd53f402
Run ruff check --fix (MR 2357)
Now that we have target-version = "py310" in [tool.ruff] in
pyproject.toml, ruff check complains about using typing.Optional and
typing.Union instead of newer syntax. Run the tool to fix it.
2024-07-16 00:26:35 +02:00
Anri Dellal
b8ca63dc2d
Fix types of pmb arguments (MR 2370)
- Change some arguments' types to bool to avoid type mismatch errors
  after refactoring in the future
- Add more arguments from parse/arguments with corresponding types
- Change type of pmbootstrap log --lines argument to int
2024-07-15 23:50:13 +02:00
Anri Dellal
30d8ea997f
Refactor bootimg_analyze (MR 2359) 2024-07-11 21:54:53 +02:00
Oliver Smith
9dfa89c58c
Use pmbootstrap_v3.cfg as config file (MR 2350)
Using pmbootstrap v3 with the old config can cause problems, for example
when having $WORK in the pmaports dir instead of the actual work path.
This is not supported anymore by v3 to reduce complexity. The format of
how mirrors are stored in the config also has changed.

Use a separate config file, so users can go back from v3 to 2.3.x if
they need to (for figuring out a regression) and so users won't run into
bugs when moving from 2.3.x to v3.
2024-07-10 20:06:25 +02:00
Clayton Craft
0a60750684
parse: arguments: fix type of --config (MR 2347)
Add the missing runtime conversion when --config is given on the
cmdline.

This expects a pathlib.Path object as the argument, but args.config is
only a string.
2024-07-08 16:30:58 +02:00
Oliver Smith
852b8d6c7d
args: remove dead code (MR 2349)
Remove a bunch of commented out code:
* debug leftovers
* update_work(): a function that used to replace $WORK in all kinds of
  arguments to pmbootstrap, with the actual work dir. This seemed like a
  good idea when I made this initially, but in hindsight it's clear that
  this made args just much more complex. I'm glad that Caleb removed
  this. Now $WORK only gets removed in one place, for mount points,
  which makes much more sense.
* args.from_argparse and related comment: pmbootstrap used to make a
  full copy of its args, so in some cases it could go back before $WORK
  was replaced IIRC. This isn't great, but even worse is that this
  caused each stack trace to say something like infinite recursion on
  args. I'm also glad that Caleb got rid of this :)
2024-07-08 16:22:38 +02:00
Caleb Connolly
0e62eaac67
ruff: format (MR 2344)
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-07-08 16:09:42 +02:00
Caleb Connolly
65d23905bc
treewide: another round of mypy fixes (MR 2344)
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-07-08 16:01:13 +02:00
Oliver Smith
1dfaafe577
kconfig check: move rules to kconfigcheck.toml (MR 2333)
Move all kconfig rules from pmb.config to a separate toml file
pmb/data/kconfigcheck.toml. This is a fallback, pmbootstrap now prefers
loading kconfigcheck.toml from the currently checked out pmaports branch
if it exists.

This finally allows having separate kconfig check rules per pmaports
branch and makes the workflow of adjusting these rules much more
pleasant as the rules and kernel configs can just be adjusted at the
same time in pmaports!

This patch also moves the definition of what rules should be checked for
community and main devices, those that have pmb:kconfigcheck-community
in their linux APKBUILD, to the new kconfigcheck.toml. This should make
it much more intuitive, previously one needed to find the place in the
pmbootstrap source and edit it there.

Furthermore the "enforce_check" logic is removed. Previously pmbootstrap
would print warnings for failed config checks in some cases, but not
exit with error which was very confusing. Now exit 0 means all checks
passed and exit 1 means, that there is at least one error.

Use toml for the file, as discussed in pmbootstrap issue 2165. Python
3.11 has a native toml reader, use tomli for previous Python versions
for compatibility.
2024-06-30 18:04:31 +02:00
Oliver Smith
6f6cf1e8da
kconfig check: remove --kconfig-check-{name} args (MR 2333)
Prepare to move the kconfigcheck config into pmaports, so it can be
different per branch and so we can adjust it at the same time as
adjusting the kernel configs.

Once it is moved, pmbootstrap will not know the cateogry names in
advance anymore, so remove the logic for having category-specific
argument names.

In a future patch, we could add an option to manually override the
categories to be tested with one specific category (taking a string
argument), if there is demand for that.
2024-06-30 18:04:31 +02:00
Anri Dellal
0c932c273d
Fix bootimg_analyze error (MR 2339)
Fix error:
rm: can't remove '/tmp/bootimg_parser': No such file or directory

Fixes #2379
2024-06-30 17:40:01 +02:00
jane400
3a9284da8a
pmb.parse.deviceinfo: add default for gpu_accelerated (MR 2340)
this fixes pmb.config.init: ask_for_ui.
2024-06-30 17:36:32 +02:00
gompa
8390a093d8
Fix quotation reuse (MR 2336)
Python 3.10.12 fails without this patch:

File "/media/gompa/73d88639-a730-456c-a428-6d500b4020b7/pmbootstrap/pmb/parse/arguments.py", line 844 help="Alpine Linux mirror, default: " f"{default_config.mirrors["alpine"]}",
SyntaxError: f-string: unmatched '['

Related: https://peps.python.org/pep-0701/
(Commit message written by Oliver)
2024-06-24 21:55:53 +02:00
Hugo Osvaldo Barrera
d6d088b68b
Remove needless lint exceptions (MR 2334)
- Ruff won't complain about long lines which end in a long URL.
- The typing exception is no longer required.
2024-06-24 13:14:40 +02:00
Oliver Smith
18fa4e58a3
Ruff: fix typing.Xxx is deprecated, use xxx instead (MR 2327) 2024-06-23 19:13:57 +02:00
Hugo Osvaldo Barrera
cebd784d84
Prefer f-strings (MR 2327) 2024-06-23 19:13:57 +02:00
Hugo Osvaldo Barrera
f3f392ef66
Use simplified Python 3.8 syntax (MR 2327)
This commit was generated with:

    ruff check --fix --extend-select=UP .
2024-06-23 19:13:52 +02:00
Hugo Osvaldo Barrera
e421bb2d41
Auto-format codebase with ruff (MR 2325)
See: https://gitlab.com/postmarketOS/pmbootstrap/-/issues/2324
Closes: https://gitlab.com/postmarketOS/pmbootstrap/-/merge_requests/2278
2024-06-23 15:40:13 +02:00
Caleb Connolly
5c652e0602
parse: apkindex: handle empty APKINDEX (MR 2252)
Sometimes the local binary repo will have no packages but still an
APKINDEX. It will be empty in this case. Handle this gracefully.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:42 +02:00
Caleb Connolly
f331b95824
chroot: allow mounting the device rootfs (MR 2252)
Add a new flag --image which can be used to mount the rootfs generated
with "pmbootstrap install".

For now this is quite limited in scope. But it's enough to allow for
building a package, updating it in the QEMU image, and then booting it.

The major "gotcha" with this is that the QEMU uses the kernel and
initramfs from the device chroot unless you run it with --efi.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:41 +02:00
Caleb Connolly
f2ca9c618e
ruff: run check --fix (MR 2252)
Get rid of unused imports and such

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:41 +02:00
Caleb Connolly
ca722b499e
tests: add tests for Arch and Chroot types (MR 2252)
Add some exhaustive unit testing to validate that these types behave as
expected.

And fix a few bugs uncovered by the tests.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:41 +02:00
Caleb Connolly
6087a9df8f
core: don't re-export get_context (MR 2252)
This makes testing a lot more annoying.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:40 +02:00
Caleb Connolly
f186ee8498
parse: apkindex: optimise parser (MR 2252)
Rewrite the parser to be more efficient:
* Avoid using str.startswith()
* Simplify logic by parsing backwards and using lines.pop() rather than
  passing around an index.

Testing with "pmbootstrap test apkindex_parse_all" on my x86 laptop:

Before: Parsed 78537 packages from 7 APKINDEX files in 1.131 seconds
After : Parsed 78537 packages from 7 APKINDEX files in 0.609 seconds

That makes for an ~86% improvement, almost twice as fast.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:40 +02:00
Caleb Connolly
62d700c3d8
commands: add pmbootstrap test (MR 2252)
This is to serve as a place to dump useful internal tests, starting with
one that simply parses all available APKINDEX files.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:40 +02:00
Caleb Connolly
fc010bc7c8
config: fixes + handle mirrors migration (MR 2252)
Get rid of config.mirror_alpine and mirrors_postmarketos and make sure
they get migrated over for existing users.

mirrors_postmarketos being a list was always a bit off, but now we have
per-aports mirrors which make a lot more sense for what we're trying to
do with systemd.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:40 +02:00
Caleb Connolly
62f3aa8cc4
parse: apkindex: don't set arch if None in providers() (MR 2252)
apkindex_files() handles arch=None just fine.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:40 +02:00
Caleb Connolly
1299c2ef6d
chroot: apk: don't recurse depends (MR 2252)
We used to recursively descend into the dependencies of the packages we
mark for install, this was needed so that we build all the packages we
want to.

However, the new package builder now does this for us. So simplify and
speed this up a bit by not doing it in apk.install().

Additionally, soft-remove support for passing in packages to delete by
having the package name start with a "!". This was useful at some point,
but we now handle this much better in pmaports.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:39 +02:00
Caleb Connolly
185d8bcef5
meta: introduce a Cache decorator (MR 2252)
Generalise pmb.helpers.other.cache with a more python decorator.

The Cache decorator takes a list of function arguments to use as cache
keys, keyword args can be used to restrict caching so that it is skipped
entirely unless the attribute has a specific value.

For example, pmb.helpers.pmaports.get() has the decorator:
@Cache("pkgname", subpackages=True)

This means the return value will be cached only when subpackages is
True, otherwise it will always miss.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:39 +02:00