Move pmb/parse/arch.py over to core and refactor it as an Arch type,
similar to how Chroot was done. Fix all the uses (that I can find) of
arch in the codebase that need adjusting.
The new Arch type is an Enum, making it clear what architectures can be
represented and making it much easier to reason about. Since we support
~5 (kinda) different representations of an Architecture (Alpine, Kernel,
target triple, platform, and QEMU), we now formalise that the Alpine
format is what we represent internally, with methods to convert to any
of the others as-needed.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Cease merging pmbootstrap.cfg into args, implement a Context type to let
us pull globals out of thin air (as an intermediate workaround) and rip
args out of a lot of the codebase.
This is just a first pass, after this we can split all the state that
leaked over into Context into types with narrower scopes (like a
BuildContext(), etc).
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
With the new chroot type, we can now write fancy paths in the pythonic
way. Convert most of the codebase over, as well as adding various other
type hints.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
We use a custom verbose log level in pmbootstrap, unfortunately it isn't
possible to correctly type this due to some limitations in the logging
library [1], [2].
Given that our usecase is fairly simple, we can just wrap the module
with our own so we only have to tell mypy to ignore the error once
instead of at every callsite.
[1]: https://github.com/cryptax/droidlysis/issues/15
[2]: https://github.com/python/typing/discussions/980
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Introduce a new module: pmb.core to contain explicitly typed pmbootstrap
API. The first component being Suffix and SuffixType. This explicitly
defines what suffixes are possible, future changes should aim to further
constrain this API (e.g. by validating against available device
codenames or architectures for buildroot suffixes).
Additionally, migrate the entire codebase over to using pathlib.Path.
This is a relatively new part of the Python standard library that uses a
more object oriented model for path handling. It also uses strong type
hinting and has other features that make it much cleaner and easier to
work with than pure f-strings. The Chroot class overloads the "/"
operator the same way the Path object does, allowing one to write paths
relative to a given chroot as:
builddir = chroot / "home/pmos/build"
The Chroot class also has a string representation ("native", or
"rootfs_valve-jupiter"), and a .path property for directly accessing the
absolute path (as a Path object).
The general idea here is to encapsulate common patterns into type hinted
code, and gradually reduce the amount of assumptions made around the
codebase so that future changes are easier to implement.
As the chroot suffixes are now part of the Chroot class, we also
implement validation for them, this encodes the rules on suffix naming
and will cause a runtime exception if a suffix doesn't follow the rules.
* Replace aports -> pmaports
* Make the "binary repo has newer version" message much shorter, and
mention "pmbootstrap pull". If users didn't mess with their pmaports
repository (checkout a custom branch, make commits), this command
will update to the latest commit and resolve the warning. This also
ties into MR 2294 where I removed a feature from "pmbootstrap status"
that would complain about the last fetch of pmaports.git being too old
and was also recommending "pmbootstrap pull". I think having it here
in the warning makes more sense, as more people will see it.
* Remove obvious / not helpful line:
"# Get package name, version, define start of debug message"
* Remove "old" in "Get old version from APKINDEX" because it may be
newer than the version in pmaports
* Replace "Aports [folder]" with pmaports (in some parts of pmbootstrap
code, pmaports is still referred to as aports, this was one of them)
Give more meaningful names to the variables:
* version_new -> version_pmaports
* version_old -> version_binary
This makes the code less confusing for the case where version_binary is
actually newer than version_pmaports. This is a relict from the time
before there was a binary package repository, in that case the version
from pmaports would always be the newer one, built from source, compared
to the local binary package that was probably built before.
abuild.conf got moved to /usr/share/abuild/default.conf and
/etc/abuild.conf is now just for user-changes. Adjust
configure_abuild() to append to /etc/abuild.conf if there is no line
defining JOBS
This should be backward compatible
Signed-off-by: Caleb Connolly <kc@postmarketos.org>
Tested-by: Oliver Smith <ollieparanoid@postmarketos.org>
Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Running abuild on the host directly creates directories in the
aport where it gets built. Interrupting abuild results in those
working-dirs not getting deleted.
We don't want to copy those entries to our builder.
It's only really noticeable if pmbootstrap tries to copy a broken
symlink in src/ and thus fails.
Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20221207205201.22139-1-jane400@bingo-ev.de%3E
Migrate to workdir version 5 and move already built packages into the edge
channel subdir, for example:
$WORK/packages/x86_64/hello-world-1-r5.apk
to:
$WORK/packages/edge/x86_64/hello-world-1-r5.apk
The build.postmarketos.org code has already been adjusted to find built
packages in either directory structure.
No build is necessary if pmaport can't be built for given arch.
pmbootstrap must use Alpine's binary package in that case, even if the
pmaport version is higher than Alpine's binary package version.
Fixes: #1897
While at it, also remove unnecessary "#!/usr/bin/env python3" in files
that only get imported, and adjust other empty/comment lines in the
beginnings of the files for consistency.
This makes files easier to read, and makes the pmbootstrap codebase more
consistent with the build.postmarketos.org codebase.
This file is used by the package browser frontend build by the alpine
developers. It uses the contents of the DESCRIPTION file to validate the
cache in the database.
Move find_aport() and find_aport_guess_main() from pmb/build/other.py
to the new file pmb/helpers/pmaports.py.
Finding aports is not only needed when building packages, hence it
makes sense to move it out of pmb.build. The pmb/helpers/pmaports.py
file will have more pmaports related functions in a follow up commit.
Clearly state which version is being used, and also display the message
when using build --force.
- Old:
WARNING: Package 'ubuntu-app-launch' in your aports folder has version
0_git20180604-r0, but the binary package repositories already have version
0_p20181101174257-r0! See also: <https://postmarketos.org/warning-repo2>
- New:
WARNING: package hello-world: aport version 1-r4 is lower than 1-r5 from
the binary repository. 1-r5 will be used when installing hello-world.
See also: <https://postmarketos.org/warning-repo2>
Find the main package by assuming it is a prefix of the subpkgname. We
do that, because in some APKBUILDs the subpkgname="" variable gets
filled with a shell loop and the APKBUILD parser in pmbootstrap can't
parse this right. (Intentionally, we don't want to implement a full
shell parser.)
## Introduction
In #1302 we noticed that `pmb.chroot.user()` does not escape commands
properly: When passing one string with spaces, it would pass them as
two strings to the chroot. The use case is passing a description with
a space inside to `newapkbuild` with `pmboostrap newapkbuild`.
This is not a security issue, as we don't pass strings from untrusted
input to this function.
## Functions for running commands in pmbootstrap
To put the rest of the description in context: We have four high level
functions that run commands:
* `pmb.helpers.run.user()`
* `pmb.helpers.run.root()`
* `pmb.chroot.root()`
* `pmb.chroot.user()`
In addition, one low level function that the others invoke:
* `pmb.helpers.run.core()`
## Flawed test case
The issue described above did not get detected for so long, because we
have a test case in place since day one, which verifies that all of the
functions above escape everything properly:
* `test/test_shell_escape.py`
So the test case ran a given command through all these functions, and
compared the result each time. However, `pmb.chroot.root()`
modified the command variable (passed by reference) and did the
escaping already, which means `pmb.chroot.user()` running directly
afterwards only returns the right output when *not* doing any escaping.
Without questioning the accuracy of the test case, I've escaped
commands and environment variables with `shlex.quote()` *before*
passing them to `pmb.chroot.user()`. In retrospective this does not
make sense at all and is reverted with this commit.
## Environment variables
By coincidence, we have only passed custom environment variables to
`pmb.chroot.user()`, never to the other high level functions. This only
worked, because we did not do any escaping and the passed line gets
executed as shell command:
```
$ MYENV=test echo test2
test 2
```
If it was properly escaped as one shell command:
```
$ 'MYENV=test echo test2'
sh: MYENV=test echo test2: not found
```
So doing that clearly doesn't work anymore. I have added a new `env`
parameter to `pmb.chroot.user()` (and to all other high level functions
for consistency), where environment variables can be passed as a
dictionary. Then the function knows what to do and we end up with
properly escaped commands and environment variables.
## Details
* Add new `env` parameter to all high level command execution functions
* New `pmb.helpers.run.flat_cmd()` function, that takes a command as
list and environment variables as dict, and creates a properly escaped
flat string from the input.
* Use that function for proper escaping in all high level exec funcs
* Don't escape commands *before* passing them to `pmb.chroot.user()`
* Describe parameters of the command execution functions
* `pmbootstrap -v` writes the exact command to the log that was
executed (in addition to the simplified form we always write down for
readability)
* `test_shell_escape.py`: verify that the command passed by reference
has not been modified, add a new test for strings with spaces, add
tests for new function `pmb.helpers.run.flat_cmd()`
* Remove obsolete commend in `pmb.chroot.distccd` about environment
variables, because we don't use any there anymore
* Add `TERM=xterm` to default environment variables in the chroot,
so running ncurses applications like `menuconfig` and `nano` works out of
the box
Use case: `mkbootimg` provides the `unpackbootimg` package. When
running `pmb.chroot.apk.install(args,"unpackbootimg")`, it was not
able to properly build the package.
Reproducing the error:
```
sudo rm ~/.local/var/pmbootstrap/packages/x86_64/mkbootimg*
pmbootstrap index
pmbootstrap --mirror-pmOS="" chroot --add=unpackbootimg
```
Or alternatively (simpler but less illustrative):
```
pmbootstrap build unpackbootimg --force
```
* The APKINDEX parser used to return a dictionary with one package for
a given package name. This works for the installed packages database,
because there can only be one provider for a package. But when
parsing packages from binary repositories, we need to support
multiple providers for one package. It is now possible to get a
dictionary with either multiple providers, or just a single provider
for each package.
* Dependency parsing logic has been adjusted, to support multiple
providers. For multiple providers, the one with the same package
name as the package we are looking up is prefered. If there is none
(eg. "so:libEGL.so.1" is provided by "mesa-egl"), it prefers packages
that will be installed anyway, and after that packages that are
already installed. When all else fails, it just picks the first one
and prints a note in the "pmbootstrap log".
* Added testcases for all functions in pmb.parse.apkindex and
pmb.parse.depends
* pmbootstrap chroot has a new "--add" parameter to specify packages
that pmbootstrap should build if neccessary, and install in the
chroot. This can be used to quickly test the depencency resolution
of pmbootstrap without doing a full "pmbootstrap install".
Fixes#1122.
If you want to build a package without changing the version number,
please use `--force` from now on. For example:
pmbootstrap build --force hello-world
Prior to this commit, changes were detected automatically (timestamp
based rebuilds). However, that feature does not work as expected with
the binary package repository we have now, and depending on how you use
git, it has never worked. Close#1167, close#1156, close#1023 and
close#985. This commit also mentions --force when a package is up to date,
but the user requested to build it.
Small improvements:
* Allow to specify multiple packages to `pmbootstrap parse_apkbuild`
* Specifying no package will parse all packages (like kconfig_check)
(also `parse_apkbuild`)
* JSON output is sorted of `parse_apkbuild`
* Make pkgver check optional, so we can disable it in the device wizard test case
* Parse_apk* -> apk*_parse
* Don't let the user mess with globs (disallow '*' in pkgname)
* ccache: Fix for distcc cross-compiling / various improvements
* Make ccache work when cross-compiling with distcc (fix#716)
* Allow to configure the ccache size in "pmbootstrap init"
* Moved ccache stats code from pmb/build/other.py to
pmb/helpers/frontend.py
* Grouped job count, ccache size and timestamp based rebuilds
together to "build options" and allow to skip them
* Sorted config options that had to be modified anyway
alphabetically
* Improve comment in arch-bin-masquerade APKBUILD
* Rename pmb/build/package.py to pmb/build/_package.py, so we can
access the functions it contains in testcases, and still use
pmb.build.package()
* Refactor the entire file. Instead of one big function that does
too many things, we have many small ones now, that are tested
in the testsuite and easier to modify
* Whenever building a package, pmbootstrap does not only build and
install the "makedepends" (like we did before), now it does the
same for the "depends". That's required to be compatible with
abuild. The old behavior can still be used with 'pmbootstrap
build --ignore-depends'.
* Because of that change, noarch packages can no longer be built in
the native chroot if we need them for a foreign chroot. A device-
package depending on a kernel would pull in the same kernel for
the native architecture otherwise.
* Running 'pmbootstrap build device-...' without '--ignore-depends'
and without a matching '--arch' displays a note that explains
this change to the user and tells how to use it instead.
* Noarch packages no longer get symlinked. That was only
implemented for packages built in the native chroot, and now that
is not always the case anymore. Symlinking these packages creates
packages with broken dependencies anyway (e.g.
device-samsung-i9100 can't be installed in x86_64, because
linux-samsung-i9100 is armhf only).
* Rename "carch" to "arch" wherever used. Naming it "carch"
sometimes is confusing with no benefit.
* Add a testcase for the aarch64 qemu workaround (because it failed
first and I needed to know for sure if it is working again).
* Improved some verbose logging, which helped with development of
this feature.
* Removed the old "build" test case (which was disabled in
testcases_fast.sh) as the new "build_package" test case covers its
functionallity.
* Only build indexes if the packages folder exists for that arch (Travis
couldn't run a test case otherwise)
* Allow to specify a custom username in "pmbootstrap init"
* Build chroots have "pmos" instead of "user" as username now
* Installation user UID is 1000 now (as in all other Linux distributions)
* Adjust autologins
* postmarketos-base: enable wheel group for sudo, removed previous sudoers file
* Implement safe upgrade path:
We save the version of the work folder format now, in $WORK/version.
When this file does not exist, it defaults to 0.
In case it does not match the currently required version
(pmb.config.work_version), then ask the user if it should
automatically be upgraded.
* apkindex:
* Also parse the architecture field
* symlink_noarch_package:
* Renamed to symlink_noarch_packages
* Always work on all packages (so we don't need to guess which
subpackages have been generated after a certain build)
* Get invoked when running 'pmbootstrap index'
* Use 'apk index' to generate one index, where the architecture
does not get rewritten (abuild does that by default, due to
Alpine's repos not having a 'noarch' folder and diverging from
that doesn't make things easier for us). That goes super fast,
and then we know which packages are noarch packages and can
create the symlinks.
* Made output less verbose:
* Use -q for 'apk index' when calling it directly (when it gets
called by abuild we can't control that)
* Output that the APKINDEXes get reindexed only to the 'pmbootstrap
log'.
We check if origin/HEAD is present. In case that reference is
missing, we show a meaningful error message now, with an explanation
on how to add it. Also moved find_out_of_sync_files_tracked() to
pmb.helpers.git
* Don't specify pkgnames from "provides" as dependencies
Always use the regular pkgname. That way, we avoid listing all
kinds of so: files as dependencies (because Alpine automatically
adds them as depends= to the package database). This fixes building
weston, and reproducing the build with `pmbootstrap challenge`.
Additional changes.
* Clear the parsed APKINDEX cache for the current pmbootstrap
session after building a package
* Avoid rebuilding a package, in case it was already built due to
circular dependencies
I've replaced all instances in the code of `os.path.abspath`
with `os.path.realpath`, as this does the same as `abspath`
plus resolving symlinks.
See also: https://stackoverflow.com/a/40311142
