Commit graph

254 commits

Author SHA1 Message Date
Oliver Smith
9dfa89c58c
Use pmbootstrap_v3.cfg as config file (MR 2350)
Using pmbootstrap v3 with the old config can cause problems, for example
when having $WORK in the pmaports dir instead of the actual work path.
This is not supported anymore by v3 to reduce complexity. The format of
how mirrors are stored in the config also has changed.

Use a separate config file, so users can go back from v3 to 2.3.x if
they need to (for figuring out a regression) and so users won't run into
bugs when moving from 2.3.x to v3.
2024-07-10 20:06:25 +02:00
Clayton Craft
0a60750684
parse: arguments: fix type of --config (MR 2347)
Add the missing runtime conversion when --config is given on the
cmdline.

This expects a pathlib.Path object as the argument, but args.config is
only a string.
2024-07-08 16:30:58 +02:00
Oliver Smith
852b8d6c7d
args: remove dead code (MR 2349)
Remove a bunch of commented out code:
* debug leftovers
* update_work(): a function that used to replace $WORK in all kinds of
  arguments to pmbootstrap, with the actual work dir. This seemed like a
  good idea when I made this initially, but in hindsight it's clear that
  this made args just much more complex. I'm glad that Caleb removed
  this. Now $WORK only gets removed in one place, for mount points,
  which makes much more sense.
* args.from_argparse and related comment: pmbootstrap used to make a
  full copy of its args, so in some cases it could go back before $WORK
  was replaced IIRC. This isn't great, but even worse is that this
  caused each stack trace to say something like infinite recursion on
  args. I'm also glad that Caleb got rid of this :)
2024-07-08 16:22:38 +02:00
Oliver Smith
6f6cf1e8da
kconfig check: remove --kconfig-check-{name} args (MR 2333)
Prepare to move the kconfigcheck config into pmaports, so it can be
different per branch and so we can adjust it at the same time as
adjusting the kernel configs.

Once it is moved, pmbootstrap will not know the cateogry names in
advance anymore, so remove the logic for having category-specific
argument names.

In a future patch, we could add an option to manually override the
categories to be tested with one specific category (taking a string
argument), if there is demand for that.
2024-06-30 18:04:31 +02:00
gompa
8390a093d8
Fix quotation reuse (MR 2336)
Python 3.10.12 fails without this patch:

File "/media/gompa/73d88639-a730-456c-a428-6d500b4020b7/pmbootstrap/pmb/parse/arguments.py", line 844 help="Alpine Linux mirror, default: " f"{default_config.mirrors["alpine"]}",
SyntaxError: f-string: unmatched '['

Related: https://peps.python.org/pep-0701/
(Commit message written by Oliver)
2024-06-24 21:55:53 +02:00
Hugo Osvaldo Barrera
d6d088b68b
Remove needless lint exceptions (MR 2334)
- Ruff won't complain about long lines which end in a long URL.
- The typing exception is no longer required.
2024-06-24 13:14:40 +02:00
Hugo Osvaldo Barrera
e421bb2d41
Auto-format codebase with ruff (MR 2325)
See: https://gitlab.com/postmarketOS/pmbootstrap/-/issues/2324
Closes: https://gitlab.com/postmarketOS/pmbootstrap/-/merge_requests/2278
2024-06-23 15:40:13 +02:00
Caleb Connolly
f331b95824
chroot: allow mounting the device rootfs (MR 2252)
Add a new flag --image which can be used to mount the rootfs generated
with "pmbootstrap install".

For now this is quite limited in scope. But it's enough to allow for
building a package, updating it in the QEMU image, and then booting it.

The major "gotcha" with this is that the QEMU uses the kernel and
initramfs from the device chroot unless you run it with --efi.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:41 +02:00
Caleb Connolly
62d700c3d8
commands: add pmbootstrap test (MR 2252)
This is to serve as a place to dump useful internal tests, starting with
one that simply parses all available APKINDEX files.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:40 +02:00
Caleb Connolly
fc010bc7c8
config: fixes + handle mirrors migration (MR 2252)
Get rid of config.mirror_alpine and mirrors_postmarketos and make sure
they get migrated over for existing users.

mirrors_postmarketos being a list was always a bit off, but now we have
per-aports mirrors which make a lot more sense for what we're trying to
do with systemd.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:40 +02:00
Caleb Connolly
084309e756
parse: arguments: typing improvements (MR 2252)
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:39 +02:00
Caleb Connolly
7a8deb0f5e
config: clean up parsing and add mirrors (MR 2252)
Add a new config section "mirrors", to replace the mirrors_alpine and
mirrors_postmarketos options. This will allow for more flexibility since
we can then handle the systemd staging repo (and others like plasma
nightly) with relative ease.

The loading/saving is fixed and now properly avoids writing out default
values, this way if the defaults are changed the user won't be stuck
with old values in their pmbootstrap.cfg.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:39 +02:00
Caleb Connolly
866e5bcfab
core: add an Arch type (MR 2252)
Move pmb/parse/arch.py over to core and refactor it as an Arch type,
similar to how Chroot was done. Fix all the uses (that I can find) of
arch in the codebase that need adjusting.

The new Arch type is an Enum, making it clear what architectures can be
represented and making it much easier to reason about. Since we support
~5 (kinda) different representations of an Architecture (Alpine, Kernel,
target triple, platform, and QEMU), we now formalise that the Alpine
format is what we represent internally, with methods to convert to any
of the others as-needed.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:39 +02:00
Caleb Connolly
450bc6873d
core: move Config type (MR 2252)
Keeping the Config class in types seemed kinda weird and was just done
as a workaround to some cyclical imports. But now things are more in
shape let's move it to core.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:39 +02:00
Caleb Connolly
b43724fee4
build: multiple binary repos (MR 2252)
Use a different binary repo depending on the source repository for the
package. This makes it possible to split out systemd packages into their
own repository.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:39 +02:00
Caleb Connolly
d9e1da98b2
more n more (MR 2252)
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:38 +02:00
Caleb Connolly
de4c912692
WIP: 2024-06-05: args hacking and more (MR 2252)
Continue removing args and do some other optimisations.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:38 +02:00
Caleb Connolly
34dd9d42ba
WIP: start ripping out args (MR 2252)
Cease merging pmbootstrap.cfg into args, implement a Context type to let
us pull globals out of thin air (as an intermediate workaround) and rip
args out of a lot of the codebase.

This is just a first pass, after this we can split all the state that
leaked over into Context into types with narrower scopes (like a
BuildContext(), etc).

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:38 +02:00
Caleb Connolly
52338c5e76
args: drop config_channels (MR 2252)
This doesn't seem to be ever used, drop it.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:38 +02:00
Caleb Connolly
31cc898dd5
treewide: adopt pathlib.Path and type hinting (MR 2252)
With the new chroot type, we can now write fancy paths in the pythonic
way. Convert most of the codebase over, as well as adding various other
type hints.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:37 +02:00
Caleb Connolly
7ed08e74d3
pmb.helpers.other.cache: simplify (MR 2252)
The init_cache() function just assigned some default constants, simplify
this by just declaring it that way to begin with, as well as adding type
hints.

Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
2024-06-23 12:38:37 +02:00
Robert Eckelmann
044d3b5a6a
pmb.*: various comment reformatting to assist with generating docs (MR 2266) 2024-05-14 14:36:22 +02:00
Clayton Craft
415e7364f4
pmb.parse: add get_parser for returning ArgumentParser obj (MR 2266)
This is needed for sphinx autoprogram since that expects an
argparse.ArgumentParser, and arguments() returns some argparse
"Namespace" obj. Useful for sphinx/autoprogram and maybe other things
later that want to get at pmb's full args.
2024-05-14 14:36:18 +02:00
Oliver Smith
0d320d0613
pmbootstrap status: rework (MR 2294)
Reimplement "pmbootstrap status" to be just a simple and useful status
overview. The previous version ran a bunch of checks every time, and
would fail on these even if pmaports was used for normal development:
  * "non-official" branch checked out in pmaports
  * pmaports.git is not clean

The information about aports.git was also considered not so useful upon
revisiting this command, since it is only used for "pmbootstrap
aportgen". Most users don't need this, and if the user runs this
command, it will tell if aports.git is outdated.

All of the above made the previous version unpleasant to use and I
suspect most people stopped using the command after trying it out a few
times and seeing the irrelevant but loud NOK complaints.

New version:

  $ pmbootstrap status
  Channel: edge (pmaports: master_staging_systemd)
  Device:  qemu-amd64 (x86_64, kernel: virt)
  UI:      console
  systemd: no (default for selected UI)

Old version (without --details it only shows NOK checks):

  $ pmbootstrap status --details
  [00:55:20] *** CONFIG ***
  [00:55:20] Device: qemu-amd64 (x86_64, "QEMU amd64")
  [00:55:20] Kernel: virt
  [00:55:20] User Interface: console
  [00:55:20]
  [00:55:20] *** GIT REPOS ***
  [00:55:20] Path: /home/user/.local/var/pmbootstrap/cache_git
  [00:55:20] - aports_upstream (master)
  [00:55:20] - pmaports (master)
  [00:55:20]
  [00:55:20] *** CHECKS ***
  [00:55:20] [OK ] Chroots zapped recently (or non-existing)
  [00:55:20] [OK ] aports_upstream: on official channel branch
  [00:55:20] [OK ] aports_upstream: workdir is clean
  [00:55:20] [OK ] aports_upstream: tracking proper remote branch 'origin/master'
  [00:55:20] [OK ] aports_upstream: up to date with remote branch
  [00:55:20] [OK ] aports_upstream: remote information updated recently (via git fetch/pull)
  [00:55:20] [OK ] pmaports: on official channel branch
  [00:55:20] [OK ] pmaports: workdir is clean
  [00:55:20] [OK ] pmaports: tracking proper remote branch 'origin/master'
  [00:55:20] [OK ] pmaports: up to date with remote branch
  [00:55:20] [OK ] pmaports: remote information updated recently (via git fetch/pull)
  [00:55:20]
  [00:55:20] NOTE: chroot is still active (use 'pmbootstrap shutdown' as necessary)
  [00:55:20] DONE!
2024-04-16 18:54:35 +00:00
Newbyte
c2a069b1f5
pmb.sideload: Query architecture from foreign device (MR 2282)
Instead of assuming the architecture of the foreign device to be what
the user selected in pmbootstrap init, actually query the architecture
from the device and use that.

This does mean that one extra ssh connection is necessary, which does
slow down the procedure somewhat. However, I think that is worth the
user experience improvement this brings.

Also, the deduction process can be skipped by manually specifying
--arch in the sideload invocation should it fail, or if one really wants
to skip that extra ssh roundtrip.

Related: https://gitlab.com/postmarketOS/pmbootstrap/-/issues/2317
2024-03-28 11:03:25 +01:00
Oliver Smith
416781d2c2
aportgen: shorten hasatrr checks with getattr (MR 2271) 2024-03-17 18:36:51 +00:00
Newbyte
883be0f119
aportgen: Add short forms of arguments (MR 2271)
With --fork-alpine-retain-branch the arguments are getting really long,
so let's add short forms for both to be consistent.
2024-03-17 18:36:45 +00:00
Newbyte
d63ea90f2b
aportgen: Allow retaining current branch when forking Alpine (MR 2271)
Often I find myself confused and subsequently frustrated by the
behaviour of --fork-alpine as usually I apply whatever changes I want to
the aports package before forking. This is due to that in the proper
aports repo I can leverage the full power of git and apply patches or
revert commits, whereas after the package has been forked to pmaports
the context git needs to do this is lost.

To fix that, this commit introduces --fork-alpine-retain-branch which
works the same way as --fork-alpine except it doesn't change the aports
branch to match the current channel.
2024-03-17 18:36:36 +00:00
Oliver Smith
fafd4e7304
Add "pmbootstrap repo_bootstrap" (MR 2273)
Related: https://postmarketos.org/pmaports.cfg bootstrap_1 etc.
2024-03-12 23:53:47 +00:00
Newbyte
3aad061c68
pmb.parse.arguments: Check for argcomplete in sys.modules instead (MR 2258)
Repurposing argcomplete as a boolean telling whether the module
argcomplete was found results in a variable with two distinct purposes
and types, which is bad practice and confuses Mypy. As such, don't
assign False to argcomplete if the module is not found and instead just
check for whether it exists in sys.modules to determine whether it was
imported successfully.
2024-02-16 10:04:24 +01:00
Andras Sebok
e0e3e213ba
flasher: heimdall-bootimg: add support for '--no-reboot' and '--resume'
Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C170534361606.26168.17672643433174186875-0@git.sr.ht%3E
2024-01-16 21:16:07 +00:00
Clayton Craft
1020ea1a8b
pmb.install: add --zap option
I zap chroots a lot, since I've found that it often "fixes" a lot of
weird issues that come about if you have stale chroots laying around.
So a common pattern I do is "pmb zap && pmb install ...". Having an
option to pmb install let's me simplify this.

Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20231214234051.4673-1-clayton@craftyguy.net%3E
2024-01-09 16:55:47 +01:00
Luca Weiss
1f1bb8e7e0
pmb.helpers.frontend: Convert kconfig check --file argument
Stop using --file as a boolean argument and just use the value behind
the argument as file to check. This also unbreaks this functionality
after the recent change now that args.package is a list since you can
pass multiple kernel packages as arguments.

Fixes: 41daa850 ("parse.arguments: Allow providing multiple packages for kconfig_check")
Closes: https://gitlab.com/postmarketOS/pmbootstrap/-/issues/2282
Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20231203094559.413672-1-luca@z3ntu.xyz%3E
2023-12-03 20:11:31 +01:00
Oliver Smith
ef047137d0
install: rename --sdcard arg to --disk
Rename the argument, because any block device can be passed to the
argument. Use "disk", because the other short word "device" usually
means the target device/phone to install.

Keep --sdcard as alias for compatibility with existing scripts and
muscle memory.

Reviewed-by: Clayton Craft <clayton@craftyguy.net>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20231119182302.2415-1-ollieparanoid@postmarketos.org%3E
2023-11-19 20:27:37 +01:00
Luca Weiss
41daa850d7
parse.arguments: Allow providing multiple packages for kconfig_check
It can be quite useful to check the configs for a list of packages, so
adjust the argument parsing code to support that.

Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Tested-by: Oliver Smith <ollieparanoid@postmarketos.org>
Reviewed-by: Clayton Craft <clayton@craftyguy.net>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20231028085901.35205-1-luca@z3ntu.xyz%3E
2023-11-01 20:42:56 +01:00
Anton Bambura
047df10d57
Support --no-cgpt
Some people may use U-Boot instead of depthcharge and they don't want cgpt

Reviewed-by: Luca Weiss <luca@z3ntu.xyz>
Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C168861285289.22824.11703157261469263638-0@git.sr.ht%3E
2023-07-13 10:07:47 +02:00
Oliver Smith
185973fd97
Remove distcc support
With this code path, pmbootstrap would start a distccd + sshd in the
native chroot, and configure it so it runs the cross compiler. The
foreign arch chroots would then call this cross compiler from localhost
by calling the distcc client instead of gcc.

This code has been obsoleted by the much simpler crossdirect in 2019.
Let's finally remove it.

Fixes: issue 2179
Reviewed-by: Luca Weiss <luca@z3ntu.xyz>
Reviewed-by: Clayton Craft <clayton@craftyguy.net>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230613161437.570196-4-ollieparanoid@postmarketos.org%3E
2023-06-15 08:31:03 +02:00
Oliver Smith
25b3530c4e
Remove --no-crossdirect argument
With an upcoming patch to remove the legacy distcc code, the
--no-crossdirect argument would do the same as --no-cross: perform the
build entirely in qemu. So remove the argument. I doubt anybody was
using it anyway.

Reviewed-by: Luca Weiss <luca@z3ntu.xyz>
Reviewed-by: Clayton Craft <clayton@craftyguy.net>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230613161437.570196-3-ollieparanoid@postmarketos.org%3E
2023-06-15 08:31:03 +02:00
Luca Weiss
97eb39f34b
flasher: remove long deprecated flash_system alias
A long long time ago we renamed flash_system to flash_rootfs, and now
it's time to finally remove this alias since nobody should be using it
anymore.

Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230605220329.14328-4-luca@z3ntu.xyz%3E
2023-06-12 08:46:22 +02:00
Luca Weiss
b8d5b9e4e5
treewide: rename _system to _rootfs in various places
A long time ago we renamed the flash_system action into flash_rootfs.
Since we still kept some variables around, it's finally time to clean
that up.

Keep backwards compatibility for now since we cannot update pmaports at
the same time since the new deviceinfo names won't be supported in older
pmbootstrap versions.

Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230605220329.14328-3-luca@z3ntu.xyz%3E
2023-06-12 08:46:22 +02:00
Oliver Smith
6352ab9c2d
Move version to pmb.__version__
Move the pmbootstrap version to __version__ in the pmb module, so it can
still be found automatically after setup.py is replaced.

Reviewed-by: Luca Weiss <luca@z3ntu.xyz>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230407233026.1712-4-ollieparanoid@postmarketos.org%3E
2023-04-15 01:44:37 +02:00
Oliver Smith
05675321d6
pmb.parse.arguments: don't use python 3.9 feature
Replace action=argparse.BooleanOptionalAction with action="store_true"
so it still works with Python 3.8 and 3.7. These aren't EOL yet, and
unless we have a good reason to drop support for them, we can still
support them.

The purpose of vermin ("pmbootstrap ci vermin") would be to warn about
this, filed a bug that it didn't catch it:
https://github.com/netromdk/vermin/issues/168

Reviewed-by: Luca Weiss <luca@z3ntu.xyz>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230326164931.2390-1-ollieparanoid@postmarketos.org%3E
2023-04-04 09:13:09 +02:00
Clayton Craft
6c66bfb8aa
parse/arguments: add qemu --efi option
This will be used to support EFI boot in qemu

Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Reviewed-by: Caleb Connolly <kc@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230316192838.21431-2-clayton@craftyguy.net%3E
2023-03-24 09:11:48 +01:00
Oliver Smith
97e21fa876
parse.arguments.add_kernel_arg: add nargs param
Prepare for a future patch that will set it to None to make sure we get
exactly one kernel argument.

Reviewed-by: Clayton Craft <clayton@craftyguy.net>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230312151325.1968-15-ollieparanoid@postmarketos.org%3E
2023-03-24 09:11:47 +01:00
Oliver Smith
dffb8614bc
kconfig check: add --no-details argument
Provide a fast way to go through the no-details mode for kconfig check,
which is otherwise only triggered after running "kconfig edit". This is
useful for development.

Reviewed-by: Clayton Craft <clayton@craftyguy.net>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230312151325.1968-9-ollieparanoid@postmarketos.org%3E
2023-03-24 09:11:47 +01:00
Oliver Smith
1a124ec2b6
kconfig check: remove redundant component lists
Components are the sets of kernel config options in
pmb/config/__init__.py, such as the ones for waydroid, nftables, etc.

Generate the list of components dynamically and refactor the code, so we
can add/remove components in just one place.

Reviewed-by: Clayton Craft <clayton@craftyguy.net>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230312151325.1968-4-ollieparanoid@postmarketos.org%3E
2023-03-24 09:11:46 +01:00
Oliver Smith
a0cea6255a
pmb.config: mount go caching directories
Always set GOCACHE during build and let it point to pmbootstrap's work
dir. This has a similar effect as using ccache for C/C++.

Set GOMODCACHE conditionally: this is for caching the go modules (git
repositories) that get cloned during a build if they are not bundled
with the source. Usually APKBUILDs should cache them, but when using
pmbootstrap build --src, they would get downloaded over and over again.
Set GOMODCACHE automatically for --src and allow enabling/disabling this
manually with new --go-mod-cache / --no-go-mod-cache.

This speeds up multiple iterations of building the same package
significantly. I'm using it for:

$ pmbootstrap build postmarketos-mkinitfs --src=...

I've verified that using the same GOCACHE dir for multiple architectures
doesn't result in build artifacts for the wrong arch getting used.

Reviewed-by: Clayton Craft <clayton@craftyguy.net>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230223064743.1773-1-ollieparanoid@postmarketos.org%3E
2023-02-26 19:37:30 +01:00
Oliver Smith
9975d373b0
Bump copyright to 2023 2023-01-22 19:18:06 +01:00
Oliver Smith
315621d5b8
pmb ci: add --fast argument
Make it easy to only run the fast tests.

Reviewed-by: Luca Weiss <luca@z3ntu.xyz>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20221111072354.3431-3-ollieparanoid@postmarketos.org%3E
2022-11-20 15:35:11 +01:00
Oliver Smith
9a84ad20b1
pmb ci: fix arg desc for --all
Reviewed-by: Luca Weiss <luca@z3ntu.xyz>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20221111072354.3431-1-ollieparanoid@postmarketos.org%3E
2022-11-20 15:35:05 +01:00