This allows us to get rid of some of the validation in sanity_checks()
as mypy handles this validation at "build time", and any typos in the
enum instantiation would be a runtime error rather than a silent
failure.
Additionally, it allows us to encode some of the behaviour of the
different output types into the type definition itself by using methods.
Part-of: https://gitlab.postmarketos.org/postmarketOS/pmbootstrap/-/merge_requests/2642
There is a lot of context and state management needed when using
pmbootstrap, sometimes it can be a lot to keep in your head. Let's print
the output of "pmbootstrap status" when stuff goes wrong, this might
help remind people if e.g. their pmaports checkout is on the wrong
branch, or they're building for the wrong device.
Additionally, don't print the "run pmbootstrap log for details" message
if pmbootstrap was called with --details-to-stdout
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
There was an oversight when this API was originally created and it
implicitly assumed that the pmaports repository was always named
"pmaports". This unfortunately broke some peoples workflows.
Introduce a new "pkgrepo_name()" function and adjust the codebase to use
it, as well as adjusting pkgrepo internally to special case the
"pmaports" repo so that it's always named pmaports no matter what the
directory itself is named.
This is probably more complexity than we should be dealing with here, we
should probably create a new type to encode this behaviour.
Fixes: #2412
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
When running some commands like "pmbootstrap chroot" or even
"pmbootstrap work_migrate", pmbootstrap will attempt to read the
pmaports.cfg from origin/master *before* doing the work dir migration.
In order to do this, the "get_upstream_remote" function tries to find
the upstream remote by URL. Let it search with the outdated URLs too, so
this doesn't fail right before migrating to the new URLs.
This could be re-purposed in the future in case we migrate URLs again.
Co-authored-by: Caleb Connolly <caleb@postmarketos.org>
Tweaked-by: Oliver Smith <ollieparanoid@postmarketos.org>
Fix for:
$ pmbootstrap pull
[21:09:43] aports_upstream (branch: 3.20-stable): not on one of the official branches (master), skipping pull!
Replace the get_branches_official() function that according to git log
was never able to tell if an aports branch was official. The new
function branch_looks_official() can do that by just checking if the
branch follows the typical naming pattern.
The comment I had put into get_branches_official earlier was not true
anymore, by now this did not only get used by "pmbootstrap status", but
also by "pmbootstrap pull".
Generalise pmb.helpers.other.cache with a more python decorator.
The Cache decorator takes a list of function arguments to use as cache
keys, keyword args can be used to restrict caching so that it is skipped
entirely unless the attribute has a specific value.
For example, pmb.helpers.pmaports.get() has the decorator:
@Cache("pkgname", subpackages=True)
This means the return value will be cached only when subpackages is
True, otherwise it will always miss.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Cease merging pmbootstrap.cfg into args, implement a Context type to let
us pull globals out of thin air (as an intermediate workaround) and rip
args out of a lot of the codebase.
This is just a first pass, after this we can split all the state that
leaked over into Context into types with narrower scopes (like a
BuildContext(), etc).
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
With the new chroot type, we can now write fancy paths in the pythonic
way. Convert most of the codebase over, as well as adding various other
type hints.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
We use a custom verbose log level in pmbootstrap, unfortunately it isn't
possible to correctly type this due to some limitations in the logging
library [1], [2].
Given that our usecase is fairly simple, we can just wrap the module
with our own so we only have to tell mypy to ignore the error once
instead of at every callsite.
[1]: https://github.com/cryptax/droidlysis/issues/15
[2]: https://github.com/python/typing/discussions/980
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Introduce a new module: pmb.core to contain explicitly typed pmbootstrap
API. The first component being Suffix and SuffixType. This explicitly
defines what suffixes are possible, future changes should aim to further
constrain this API (e.g. by validating against available device
codenames or architectures for buildroot suffixes).
Additionally, migrate the entire codebase over to using pathlib.Path.
This is a relatively new part of the Python standard library that uses a
more object oriented model for path handling. It also uses strong type
hinting and has other features that make it much cleaner and easier to
work with than pure f-strings. The Chroot class overloads the "/"
operator the same way the Path object does, allowing one to write paths
relative to a given chroot as:
builddir = chroot / "home/pmos/build"
The Chroot class also has a string representation ("native", or
"rootfs_valve-jupiter"), and a .path property for directly accessing the
absolute path (as a Path object).
The general idea here is to encapsulate common patterns into type hinted
code, and gradually reduce the amount of assumptions made around the
codebase so that future changes are easier to implement.
As the chroot suffixes are now part of the Chroot class, we also
implement validation for them, this encodes the rules on suffix naming
and will cause a runtime exception if a suffix doesn't follow the rules.
HTTP auth is heavily discouraged, and becoming annoying to set up and
use. Let folks reconfigure pmaports origin remote using the SSH URL.
Signed-off-by: Caleb Connolly <caleb@postmarketos.org>
Reimplement "pmbootstrap status" to be just a simple and useful status
overview. The previous version ran a bunch of checks every time, and
would fail on these even if pmaports was used for normal development:
* "non-official" branch checked out in pmaports
* pmaports.git is not clean
The information about aports.git was also considered not so useful upon
revisiting this command, since it is only used for "pmbootstrap
aportgen". Most users don't need this, and if the user runs this
command, it will tell if aports.git is outdated.
All of the above made the previous version unpleasant to use and I
suspect most people stopped using the command after trying it out a few
times and seeing the irrelevant but loud NOK complaints.
New version:
$ pmbootstrap status
Channel: edge (pmaports: master_staging_systemd)
Device: qemu-amd64 (x86_64, kernel: virt)
UI: console
systemd: no (default for selected UI)
Old version (without --details it only shows NOK checks):
$ pmbootstrap status --details
[00:55:20] *** CONFIG ***
[00:55:20] Device: qemu-amd64 (x86_64, "QEMU amd64")
[00:55:20] Kernel: virt
[00:55:20] User Interface: console
[00:55:20]
[00:55:20] *** GIT REPOS ***
[00:55:20] Path: /home/user/.local/var/pmbootstrap/cache_git
[00:55:20] - aports_upstream (master)
[00:55:20] - pmaports (master)
[00:55:20]
[00:55:20] *** CHECKS ***
[00:55:20] [OK ] Chroots zapped recently (or non-existing)
[00:55:20] [OK ] aports_upstream: on official channel branch
[00:55:20] [OK ] aports_upstream: workdir is clean
[00:55:20] [OK ] aports_upstream: tracking proper remote branch 'origin/master'
[00:55:20] [OK ] aports_upstream: up to date with remote branch
[00:55:20] [OK ] aports_upstream: remote information updated recently (via git fetch/pull)
[00:55:20] [OK ] pmaports: on official channel branch
[00:55:20] [OK ] pmaports: workdir is clean
[00:55:20] [OK ] pmaports: tracking proper remote branch 'origin/master'
[00:55:20] [OK ] pmaports: up to date with remote branch
[00:55:20] [OK ] pmaports: remote information updated recently (via git fetch/pull)
[00:55:20]
[00:55:20] NOTE: chroot is still active (use 'pmbootstrap shutdown' as necessary)
[00:55:20] DONE!
Add a new command that makes running CI scripts easy. The user goes to
the git repository of choice, which has CI scripts written in a certain
format, and then runs 'pmbootstrap ci' to get an interactive selection
of which of the available scripts to run (or "all"). Specifying one or
multiple scripts on the command-line is also possible, e.g.
$ pmbootstrap ci flake8
$ pmbootstrap ci shellcheck flake8 pytest
$ pmbootstrap ci --all
pmbootstrap then either runs the selected scripts in a chroot (and
installs dependencies as defined at the beginning of the CI scripts), or
natively (with checks inside the scripts for having dependencies
installed). Running natively is needed for .ci/pytest.sh in this
pmbootstrap.git repository, as pmbootstrap can't run inside pmbootstrap.
Running natively or in chroot is defined in an "# Options: " comment
inside the script file.
Documentation for this command and how script files look like:
https://postmarketos.org/pmb-ci
Replace "args.cache" with a global variable in order to
avoid passing "args" to all functions. This is a step to get rid of this
args-passed-to-all-functions pattern in pmbootstrap.
Translate the pmaports channels "stable" to "v20.05" and "stable-next"
to "v21.03", so these have the same channel name as the pmaports.git
branch name.
The original plan was to switch the "stable" channel from the "v20.05"
branch to the "v21.03" branch when the release is done. However, now
that we are close to that, I'm realizing that this would not be useful.
It would lead to conflicts in the dir with locally built packages
(default: ~/.local/var/pmbootstrap/packages/$CHANNEL). And it would make
it awkward to go back to a previous branch (we may name it old-stable
for the time being, but what after that, old-old-stable?).
This made sense for Alpine's aports.git repository as we were only using
the master branch. But now that we are using more branches, we need the
entire git repository with all its branches cloned.
Prepare to base postmarketOS on Alpine stable by parsing the new
channels.cfg file in pmaports.git, that describes which channel
needs which branches and mirror dirs from postmarketOS and Alpine.
Use the information in pmb.helpers.git.get_branches_official() first,
more is coming in follow-up commits.
Read the file from origin/master, so we get the latest fetched version
even if the last checked out master branch is not up-to-date (think of
currently checked out release branch instead of master, master will
never be updated to point to latest origin/master). Allow to override
the file with a new --config-channels parameter.
Related: https://postmarketos.org/channels.cfg
Use the timestamp of .git/FETCH_HEAD in each git repository, to
determine if too much time has passed since the last fetch/pull.
Modify pmb.helpers.git.clone, so FETCH_HEAD is always created if it does
not exist (because "git clone" would not create it).
Related: #1829
Add dummy function that only returns ["master"] for now, so we can use it
in the upcoming git checks for "pmbootstrap status". More sophisticated
logic to figure out the branches will be added soon, see project
direction 2020 issue.
Related: #1829
While at it, also remove unnecessary "#!/usr/bin/env python3" in files
that only get imported, and adjust other empty/comment lines in the
beginnings of the files for consistency.
This makes files easier to read, and makes the pmbootstrap codebase more
consistent with the build.postmarketos.org codebase.
