When generating an ssh key it's simple to change the name of the key to
something that's not e.g. id_rsa.pub or id_ed25519.pub, so let's try to
capture ssh keys for more users.
This is section is not meant to represent my opinions, rather I seek to
document the status quo. I base the support range on what's been
reasoned in prior merge requests such as [1] and [2], as well as [3]
which only fixed CI in the one active stable release and left old stable
releases unfixed.
While I don't necessary think this is a great situation given that old
releases may sometimes be useful, extending the time pmbootstrap has to
support old releases would incur significant additional maintenance
effort and consequently require more discussion than just documenting
what's already happening. As such, I think this is the most reasonable
thing to do for now.
[1]: https://gitlab.com/postmarketOS/pmbootstrap/-/merge_requests/2360#note_1999012908
[2]: https://gitlab.com/postmarketOS/pmbootstrap/-/merge_requests/2417#note_2135527528
[3]: https://gitlab.com/postmarketOS/pmaports/-/merge_requests/5676
Using pmbootstrap v3 with the old config can cause problems, for example
when having $WORK in the pmaports dir instead of the actual work path.
This is not supported anymore by v3 to reduce complexity. The format of
how mirrors are stored in the config also has changed.
Use a separate config file, so users can go back from v3 to 2.3.x if
they need to (for figuring out a regression) and so users won't run into
bugs when moving from 2.3.x to v3.
Move all kconfig rules from pmb.config to a separate toml file
pmb/data/kconfigcheck.toml. This is a fallback, pmbootstrap now prefers
loading kconfigcheck.toml from the currently checked out pmaports branch
if it exists.
This finally allows having separate kconfig check rules per pmaports
branch and makes the workflow of adjusting these rules much more
pleasant as the rules and kernel configs can just be adjusted at the
same time in pmaports!
This patch also moves the definition of what rules should be checked for
community and main devices, those that have pmb:kconfigcheck-community
in their linux APKBUILD, to the new kconfigcheck.toml. This should make
it much more intuitive, previously one needed to find the place in the
pmbootstrap source and edit it there.
Furthermore the "enforce_check" logic is removed. Previously pmbootstrap
would print warnings for failed config checks in some cases, but not
exit with error which was very confusing. Now exit 0 means all checks
passed and exit 1 means, that there is at least one error.
Use toml for the file, as discussed in pmbootstrap issue 2165. Python
3.11 has a native toml reader, use tomli for previous Python versions
for compatibility.
Python 3.9 was required in pmb/__init__.py already, but some references
still said 3.8 and 3.7.
Co-Developed-By: Oliver Smith <ollieparanoid@postmarketos.org>
Many have been unable to use pmbootstrap install on kernel versions
between 5.8 and 6.0 as parted was giving the following error:
Error: Partition(s) 1, <...>, 64 on /dev/install have been
written, but we have been unable to inform the kernel
of the change, probably because it/they are in use.
As a result, the old partition(s) will remain in use.
You should reboot now before making further changes.
And pmb couldn't find /dev/loop0p1:
ERROR: Unable to find the first partition of /dev/loop0,
expected it to be at /dev/loop0p1!
Related: pmbootstrap#2309
[ci:skip-build]: no code changes
Signed-off-by: Ferass El Hafidi <funderscore@postmarketos.org>
With this code path, pmbootstrap would start a distccd + sshd in the
native chroot, and configure it so it runs the cross compiler. The
foreign arch chroots would then call this cross compiler from localhost
by calling the distcc client instead of gcc.
This code has been obsoleted by the much simpler crossdirect in 2019.
Let's finally remove it.
Fixes: issue 2179
Reviewed-by: Luca Weiss <luca@z3ntu.xyz>
Reviewed-by: Clayton Craft <clayton@craftyguy.net>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230613161437.570196-4-ollieparanoid@postmarketos.org%3E
When invoking pmbootstrap build most packages default to the host arch,
however depending on your workflow it might be preferrable to default to
the device arch.
Add a new config option "build_default_device_arch" which when set will
make "pmbootstrap build" prioritise the device arch over the native
arch.
Default to False to preserve the old behaviour and don't ask during
pmbootstrap init as this may not be relevant for most folks.
Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Co-developed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230418-build-default-arch-v5-2-5223fab65867@postmarketos.org%3E
It's possible to set the default To: address and subject prefix in the
git config of the local checkout. This makes the workflow a bit easier
and makes sure the subject starts with [PATCH pmbootstrap] instead of
the regular [PATCH].
* Links in the top row point to some very old blog posts and to the
devices wiki page. Remove them, the first line has a link to
postmarketos.org where the user should be able to get all relevant
information about postmarketOS if they don't know this project.
* "Package build scripts live in the pmaports repository now.", this
was useful shortly after we made the change, but it has been like this
for years now. Again, the postmarketOS homepage lists where other
related source code is, in case the user doesn't know.
* 2 GB of RAM recommened for compiling: this was a rule of thumb for
when all postmarketOS packages had to be compiled from source. This is
not the case anymore, we have a binary repository. Depending on which
package you want to compile, you don't need as much RAM. And some
users who just want to run 'pmbootstrap install' don't need to build
anything at all. I think this is more confusing than helpful, so
remove it.
* Kernels based on grsec patchset are not supported. I don't think
anyone will try this, this was more relevant when the patchset was
still distributed freely and you could actually use it in Alpine
Linux (by default even?) and Arch Linux. An artefact of the past.
CI started failing with:
/builds/ollieparanoid/pmbootstrap/venv/bin/python3: No module named pytest
I've briefly tried to fix this with the existing scripts. However,
instead of investing more time into that, do the long overdue
refactoring of the scripts that involve dropping the venv logic and
support for a custon gitlab-ci-runner using some python docker image as
base. This configuration hasn't been used for a long time and is
probably broken anyway.
Refactor the logic to skip the qemu test case in gitlab CI by using
pytest markers. The new script is now similar to bpo's .ci/pytest.py.
Do not install git in the native chroot and use it from there. Remove the
chown_to_user argument from pmb.helpers.git.clone(), the resulting dir
is now always owned by the user. While at it, refactor the function and
display the clone URL.
Previously we had cloned aports_upstream (from Alpine) with
chown_to_user=False (legacy) and pmaports with chown_to_user=True.
pmb.helpers.git.rev_parse() would only work after chown_to_user=True.
Check if git is installed in "pmbootstrap init", and remove the same
check from rev_parse(). Add a new work dir version, that checks for git
and changes ownership of already checked out aports_upstream to the
host system's user.
When creating a new work dir, create cache_git instead of cache_http.
cache_http is created on demand already, with proper permissions. But
cache_git must be created, otherwise pmb.helpers.mount.bind will create it
as root.
This is in preparation for the "pmbootstrap pull" feature, as it allows
using the host system's git in all new code paths. We will be able to
handle repositories even if they were cloned outside of the work dir
(which we do in a few CI scripts for example).
Related: #1858
Add a new action that lists all aports, for which no binary packages
exist. Only list packages that can be built for the relevant arch
(specified with --arch). This works recursively: when a package can be
built for a certain arch, but one of its dependencies
(or their depends) can not be built for that arch, then don't list it.
This action will be used for the new sr.ht based build infrastructure,
to figure out which packages need to be built ahead of time (so we can
trigger each of them as single build job). Determining the order of the
packages to be built is not determined with pmbootstrap, the serverside
code of build.postmarketos.org takes care of that.
For testing purposes, a single package can also be specified and the
action will list if it can be built for that arch with its
dependencies, and what needs to be built exactly.
Add pmb/helpers/package.py to hold functions that work on both pmaports
and (binary package) repos - in contrary to the existing
pmb/helpers/pmaports.py (see previous commit) and pmb/helpers/repo.py,
which only work with one of those.
Refactoring:
* pmb/helpers/pmaports.py: add a get_list() function, which lists all
aports and use it instead of writing the same glob loop over and over
* add pmb.helpers.pmaports.get(), which finds an APKBUILD and parses it
in one step.
* rename pmb.build._package.check_arch to ...check_arch_abort to
distinguish it from the other check_arch function
When the timeout occurs it is important to ensure clean up of child
processes. Killing only the direct process created by a command can
leave child processes running.
For example a pmbootstrap.py install will run apk add. This run command
creates multiple processes as follows:
(cmd line arguments snipped for readability)
$ ps -e -o pid,ppid,pgid,cmd
PID PPID PGID CMD
31738 23247 31738 python3 ./pmbootstrap.py -t 15 install --no-fde
31746 31738 31738 sudo env -i /bin/sh -c ... ;apk --no-progress add
31747 31746 31738 /bin/sh -c ... ;apk --no-progress add
31748 31747 31738 apk --no-progress add
The root process of the run command is PID 31746. We want to kill
the child processes too. Otherwise only running kill -9 31746 will leave
the processes 31747 and 31748 running.
Follow-up to !1373, where `pmbootstrap flasher flash_system` was
replaced with `pmbootstrap flasher flash_rootfs`. We still had used
terms like "system partition" in a lot of places.
This commit replaces it everywhere, so it's clear that we're talking
about the pmOS rootfs (which may or may not be installed to Android's
system partition).
