This allows us to get rid of some of the validation in sanity_checks()
as mypy handles this validation at "build time", and any typos in the
enum instantiation would be a runtime error rather than a silent
failure.
Additionally, it allows us to encode some of the behaviour of the
different output types into the type definition itself by using methods.
Part-of: https://gitlab.postmarketos.org/postmarketOS/pmbootstrap/-/merge_requests/2642
This was always a hack, and it looks like there (hopefully) aren't any
placs where we still need to handle this.
Possibly expecting regressions... But then we have something to write a
test for.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
I think the "these types make no sense" comment were about
output_return_buffer, which defaulted to False but was to be set to
list[bytes] if used. I changed it so that it defaults to None, which
probably is more conventional. Other than that I didn't notice anything
weird about the types here.
Also check for None as necessary to appease mypy.
Now that we have target-version = "py310" in [tool.ruff] in
pyproject.toml, ruff check complains about using typing.Optional and
typing.Union instead of newer syntax. Run the tool to fix it.
Move pmb/parse/arch.py over to core and refactor it as an Arch type,
similar to how Chroot was done. Fix all the uses (that I can find) of
arch in the codebase that need adjusting.
The new Arch type is an Enum, making it clear what architectures can be
represented and making it much easier to reason about. Since we support
~5 (kinda) different representations of an Architecture (Alpine, Kernel,
target triple, platform, and QEMU), we now formalise that the Alpine
format is what we represent internally, with methods to convert to any
of the others as-needed.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Building the command strings and entering the chroot is a
not-insubstantial amount of overhead. Implement support for running
multiple commands with a new pmb.chroot.rootm() function.
TODO: add alternative for chroot.user and run.root/user.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Cease merging pmbootstrap.cfg into args, implement a Context type to let
us pull globals out of thin air (as an intermediate workaround) and rip
args out of a lot of the codebase.
This is just a first pass, after this we can split all the state that
leaked over into Context into types with narrower scopes (like a
BuildContext(), etc).
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
We can't totally escape the need for some runtime state defined by args.
To make the migration easier, introduce a global "Context" class and
move some of the read-only global options there.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
With the new chroot type, we can now write fancy paths in the pythonic
way. Convert most of the codebase over, as well as adding various other
type hints.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
We use a custom verbose log level in pmbootstrap, unfortunately it isn't
possible to correctly type this due to some limitations in the logging
library [1], [2].
Given that our usecase is fairly simple, we can just wrap the module
with our own so we only have to tell mypy to ignore the error once
instead of at every callsite.
[1]: https://github.com/cryptax/droidlysis/issues/15
[2]: https://github.com/python/typing/discussions/980
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Introduce a new module: pmb.core to contain explicitly typed pmbootstrap
API. The first component being Suffix and SuffixType. This explicitly
defines what suffixes are possible, future changes should aim to further
constrain this API (e.g. by validating against available device
codenames or architectures for buildroot suffixes).
Additionally, migrate the entire codebase over to using pathlib.Path.
This is a relatively new part of the Python standard library that uses a
more object oriented model for path handling. It also uses strong type
hinting and has other features that make it much cleaner and easier to
work with than pure f-strings. The Chroot class overloads the "/"
operator the same way the Path object does, allowing one to write paths
relative to a given chroot as:
builddir = chroot / "home/pmos/build"
The Chroot class also has a string representation ("native", or
"rootfs_valve-jupiter"), and a .path property for directly accessing the
absolute path (as a Path object).
The general idea here is to encapsulate common patterns into type hinted
code, and gradually reduce the amount of assumptions made around the
codebase so that future changes are easier to implement.
As the chroot suffixes are now part of the Chroot class, we also
implement validation for them, this encodes the rules on suffix naming
and will cause a runtime exception if a suffix doesn't follow the rules.
Fix "pmbootstrap chroot" and others not passing the proxy environment
variables correctly. Thanks to notfound405 for pointing this out!
Instead of only preserving proxy environment variables in
pmb.helpers.run_core, which should never be called directly, do it in
the calling functions:
* pmb.helpers.run.user
* pmb.helpers.run.root
* pmb.chroot.root
* pmb.chroot.user
This fixes that the environment variables were only really passed by
pmb.helpers.run.user, because the other functions would result in
something like:
HTTP_PROXY=mytestproxy sudo env -i /usr/bin/sh -c '…'
This is needed to either elevate to root, or to elevate to root first
and then enter the chroot as root or user. Due to the "env -i", the
environment intentionally gets cleaned, but unintentionally also removes
the proxy environment variables that were explicitly set.
By adjusting the functions, they now run a variant of:
sudo env -i /usr/bin/sh -c 'HTTP_PROXY=mytestproxy …'
The escaping is simplified in this example, run "pmbootstrap -v" to see
the not very readable, but proper escaping with shutil.quote().
Remove the previous test for preserving the environment variables in
pmb.helpers.run_core (as it should never be called directly), and test
instead the new behavior.
Fixes: issue 2299
Fixes: 13c4ac42 ("pmb.helpers.run_core: fix proxy env var logic")
Fix that the list "cmd" was turned into a string if one of the proxy
vars was set in the environment. Add a test for this code path. Before
this patch:
$ FTP_PROXY=test pmbootstrap -v --details-to-stdout status
…
% cd /home/user/.local/var/pmbootstrap/cache_git/pmaports; git remote -v
run: FTP_PROXY=test ['git', 'remote', '-v']
ERROR: [Errno 2] No such file or directory: "FTP_PROXY=test ['git', 'remote', '-v']"
Fixes: 1a00c04f ("pmb.helpers.run_core: always configure proxy vars if set in environment")
Reviewed-by: Clayton Craft <clayton@craftyguy.net>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230713182337.6185-3-ollieparanoid@postmarketos.org%3E
pmb.config.sudo expects a list of commands + args, so when a string is
passed (e.g. "true"), it results in pmb trying to execute the equivalent
of "$ t r u e":
(8/119) Installing linux-purism-librem5 (6.3.4-r0)
doas: t: command not found
doas: t: command not found
doas: t: command not found
doas: t: command not found
Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230613141929.15718-1-clayton@craftyguy.net%3E
When looking at the table for possible output modes, it only makes sense
to pass stdin to "interactive" and "tui". The output mode "stdout" is
for non-interactive commands.
This fixes apk going interactive (asking for confirmation) when running
pmbootstrap with --details-to-stdout and building a package that depends
on postmarketos-base.
Fixes: issue 2208
Tested-by: Clayton Craft <clayton@craftyguy.net>
Reviewed-by: Clayton Craft <clayton@craftyguy.net>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230301204112.4351-1-ollieparanoid@postmarketos.org%3E
Don't pass stdin to commands that aren't supposed to be used
interactively (output: log, background, pipe).
This fixes an inconsistency between building packages in CI on gitlab
and building them via bpo on sourcehut or locally. In gitlab, apparently
there is no stdin for the entire build job and so unanswered kernel
config prompts will just use the default. In local builds and on
sourcehut stdin is available and so it just hangs at the prompt until
pmbootstrap kills the build job due to no output being written.
I considered adding an additional check to pmaports to ensure that there
are no unanswered kernel config prompts just in case users run abuild
manually on the kernel APKBUILD with stdin available. But I think
forcing the users to answer all the prompts even if it's not really
needed just creates additional work / makes the workflow worse without
real benefit.
Related: https://builds.sr.ht/~postmarketos/job/824373#task-pmbootstrap_build-432
Fixes: pmaports issue 1225
Replace "args.cache" with a global variable in order to
avoid passing "args" to all functions. This is a step to get rid of this
args-passed-to-all-functions pattern in pmbootstrap.
Replace "args.logfd" with "pmb.helpers.logging.logfd" in order to avoid
passing "args" to all functions that only use it to write to logfd. This
is the first step to get rid of this args-passed-to-all-functions
pattern in pmbootstrap.
Many of pmbootstrap's actions require root rights. When after requesting
sudo access pmbootstrap takes longer than the sudo timeout interval to finish
execution, the password will have to be entered again on the next sudo
action.
This change adds an opt-in feature to run sudo -v in a background loop
in order to prevent having to enter the password more than once for a single
pmbootstrap run. The loop runs as a daemon timer which automatically gets
canceled when pmbootstrap exits.
Closes: #1677
Replace the "kill_as_root" argument with a much simpler "sudo" argument
and remove the now obsolete check for the output mode of "kill_as_root".
"kill_as_root" would only get set to True if both conditions are met:
a) command is running with sudo
b) command is running with an output mode ("log" or "stdout") where
pmb.helpers.run_core would kill it if it does not output anything
before a timeout is reached
The new "sudo" argument just indicates if the command is running with
sudo (a), regardless of the output mode (b).
This adds a new output mode "pipe" that is identical to the existing
"background" mode except for that its stdout is redirected into a
pipe so that it can be retrieved.
