pmbootstrap-meow/pmb/aportgen/core.py
Oliver Smith b8f35d45b8
aportgen: Gracefully handle old aports_upstream (#1291)
In order to get cross-compilers, we generate a few aports (e.g.
binutils-armhf, gcc-armhf) automatically from Alpine's aports.
pmbootstrap was already able to perform a git checkout of Alpine's
aports repository. But it needed to be manually updated. Otherwise
the `pmbootstrap aportgen` command could actually downgrade the aport
instead of updating it to the current version.

After thinking about adding a dedicated pmbootstrap command for
updating git repositories, I thought it would be better to not open
that can of worms (pmbootstrap as general git wrapper? no thanks).

The solution implemented here compares the upstream aport version of
the git checkout of a certain package (e.g. gcc for gcc-armhf) with the
version in Alpine's binary package APKINDEX. When the aport version is
lower than the binary package version, it shows the user how to update
the git repository with just one command:

    pmbootstrap chroot --add=git --user -- \
        git -C /mnt/pmbootstrap-git/aports_upstream pull

Changes:
* `pmb.aportgen.core.get_upstream_aport()`: new function, that returns
  the absolute path to the upstream aport on disk, after checking the
  version of the aport against the binary package.
* Use that new function in pmb.aportgen.gcc and pmb.aportgen.binutils
* New function `pmb.helpers.repo.alpine_apkindex_path()`: updates the
  APKINDEX if necessary and returns the absolute path to the APKINDEX.
  This code was basically present already, but not as function, so now
  we have a bit less overhead there.
* `pmbootstrap chroot`: new `--user` argument
* `pmb.parse.apkbuild`: make pkgname check optional, as it fails with
  the official gcc APKBUILD before we modify it (the current APKBUILD
  parser is not meant to be perfect, as this would require a full shell
  parsing implementation).
* Extended `test_aportgen.py` and enabled it by default in
  `testcases_fast.sh`.  Previously it was disabled due to traffic
  concerns (cloning the aports repo, but then again we do a full KDE
  plasma mobile installation in Travis now, so that shouldn't matter
  too much).
* `testcases_fast.sh`: With "test_aport_in_sync_with_git" removed
  from the disabled-by-default list (left over from timestamp based
  rebuilds), there were no more test cases disabled by default. I've
  changed it, so now the qemu_running_processes test case is disabled,
  and added an `--all` parameter to the script to disable no test
  cases. Travis runs with the `--all` parameter while it's useful to
  do a quick local test without `--all` in roughly 2 minutes instead of
  10.
* `aports/cross/binutils-*`: Fix `_mirror` variable to point to current
  default Alpine mirror (so the aportgen testcase runs through).
2018-03-11 14:18:21 +00:00

165 lines
6.1 KiB
Python

"""
Copyright 2018 Oliver Smith
This file is part of pmbootstrap.
pmbootstrap is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
pmbootstrap is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with pmbootstrap. If not, see <http://www.gnu.org/licenses/>.
"""
import fnmatch
import logging
import pmb.helpers.git
def format_function(name, body, remove_indent=4):
"""
Format the body of a shell function passed to rewrite() below, so it fits
the format of the original APKBUILD.
"""
ret = ""
lines = body.split("\n")
for i in range(len(lines)):
line = lines[i]
if not line.strip():
if not ret or i == len(lines) - 1:
continue
ret += line[remove_indent:] + "\n"
return name + "() {\n" + ret + "}\n"
def rewrite(args, pkgname, path_original, fields={}, replace_pkgname=None,
replace_functions={}, replace_simple={}, below_header=""):
"""
Append a header to $WORK/aportgen/APKBUILD, delete maintainer/contributor
lines (so they won't be bugged with issues regarding our generated aports),
and add reference to the original aport.
:param fields: key-value pairs of fields, that shall be changed in the
APKBUILD. For example: {"pkgdesc": "my new package", "subpkgs": ""}
:param replace_pkgname: When set, $pkgname gets replaced with that string in
every line.
:param replace_functions: Function names and new bodies, for example:
{"build": "return 0"}
The body can also be None (deletes the function)
:param replace_simple: Lines, that fnmatch the pattern, get
replaced/deleted. Example: {"*test*": "# test", "*mv test.bin*": None}
:param below_header: String, that gets directly placed below the header.
"""
# Header
lines_new = [
"# Automatically generated aport, do not edit!\n",
"# Generator: pmbootstrap aportgen " + pkgname + "\n",
"# Based on: " + path_original + "\n",
"\n",
]
for line in below_header.split("\n"):
if not line[:8].strip():
line = line[8:]
lines_new += line.rstrip() + "\n"
# Copy/modify lines, skip Maintainer/Contributor
path = args.work + "/aportgen/APKBUILD"
with open(path, "r+", encoding="utf-8") as handle:
skip_in_func = False
for line in handle.readlines():
# Skip maintainer/contributor
if line.startswith("# Maintainer") or line.startswith(
"# Contributor"):
continue
# Replace functions
if skip_in_func:
if line.startswith("}"):
skip_in_func = False
continue
else:
for func, body in replace_functions.items():
if line.startswith(func + "() {"):
skip_in_func = True
if body:
lines_new += format_function(func, body)
break
if skip_in_func:
continue
# Replace fields
for key, value in fields.items():
if line.startswith(key + "="):
line = key + "=\"" + value + "\"\n"
break
# Replace $pkgname
if replace_pkgname and "$pkgname" in line:
line = line.replace("$pkgname", replace_pkgname)
# Replace simple
for pattern, replacement in replace_simple.items():
if fnmatch.fnmatch(line, pattern):
line = replacement
if replacement:
line += "\n"
break
if line is None:
continue
lines_new.append(line)
# Write back
handle.seek(0)
handle.write("".join(lines_new))
handle.truncate()
def get_upstream_aport(args, upstream_path):
"""
Perform a git checkout of Alpine's aports and get the path to the aport.
:param upstream_path: where the aport is in the git repository, e.g.
"main/gcc"
:returns: absolute path on disk where the Alpine aport is checked out
example: /opt/pmbootstrap_work/cache_git/aports/upstream/main/gcc
"""
# APKBUILD
pmb.helpers.git.clone(args, "aports_upstream")
aport_path = (args.work + "/cache_git/aports_upstream/" + upstream_path)
apkbuild = pmb.parse.apkbuild(args, aport_path + "/APKBUILD",
check_pkgname=False)
apkbuild_version = apkbuild["pkgver"] + "-r" + apkbuild["pkgrel"]
# Binary package
split = upstream_path.split("/", 1)
repo = split[0]
pkgname = split[1]
index_path = pmb.helpers.repo.alpine_apkindex_path(args, repo)
package = pmb.parse.apkindex.package(args, pkgname, indexes=[index_path])
# Compare version (return when equal)
compare = pmb.parse.version.compare(apkbuild_version, package["version"])
if compare == 0:
return aport_path
# Different version message
logging.error("ERROR: Package '" + pkgname + "' has a different version in"
" local checkout of Alpine's aports (" + apkbuild_version +
") compared to Alpine's binary package (" +
package["version"] + ")!")
# APKBUILD < binary
if compare == -1:
raise RuntimeError("You can update your local checkout with:"
" 'pmbootstrap chroot --add=git --user -- git -C"
" /mnt/pmbootstrap-git/aports_upstream pull'")
# APKBUILD > binary
raise RuntimeError("You can force an update of your binary package"
" APKINDEX files with: 'pmbootstrap update'")