ninja is faster at building cmake packages than make, and according to reports
also more reliable at handling parallel builds
This commit includes a patch that adds GNU make jobserver support, in order to
allow more precise control over the number of parallel tasks
Enable parallel build by default for packages using ninja
Signed-off-by: Felix Fietkau <nbd@nbd.name>
Signed-off-by: Rosen Penev <rosenp@gmail.com>
Before this commit, it was assumed that mkhash is in the PATH. While
this was fine for the normal build workflow, this led to some issues if
make TOPDIR="$(pwd)" -C "$pkgdir" compile
was called manually. In most of the cases, I just saw warnings like this:
make: Entering directory '/home/.../package/gluon-status-page'
bash: line 1: mkhash: command not found
bash: line 1: mkhash: command not found
bash: line 1: mkhash: command not found
bash: line 1: mkhash: command not found
bash: line 1: mkhash: command not found
bash: line 1: mkhash: command not found
bash: line 1: mkhash: command not found
bash: line 1: mkhash: command not found
[...]
While these were only warnings and the package still compiled sucessfully,
I also observed that some package even fail to build because of this.
After applying this commit, the variable $(MKHASH) is introduced. This
variable points to $(STAGING_DIR_HOST)/bin/mkhash, which is always the
correct path.
Signed-off-by: Leonardo Mörlein <me@irrelefant.net>
Fixes issue openwrt/packages#14921, whereby inline ASM wasn't getting
built as PIC; look at gmp-6.2.1/mpn/x86/pentium/popcount.asm for
example:
ifdef(`PIC',`
...
for a routine that exists in both PIC and non-PIC versions.
Make sure that wherever $(FPIC) gets passed as a variable expansion
that it gets quoted where necessary (such as setting environment
variables in shell commands).
Signed-off-by: Philip Prindeville <philipp@redfish-solutions.com>
This reverts commit b12288fa69.
The patchelf approach is too fragile, and the only users of this have been
converted to make patching unnecessary
Leave the abi_version_str variable in place in rules.mk
Signed-off-by: Felix Fietkau <nbd@nbd.name>
This makes it possible to declare a package ABI_VERSION independent from the
upstream soname by setting PKG_ABI_VERSION in the package makefile.
The library filename is fixed up for files installed to packages and to the
staging dir. References to the original from executables within the same
package are also fixed up
Signed-off-by: Felix Fietkau <nbd@nbd.name>
The license folder is a core part of OpenWrt and all GPL-2.0 licensed.
Use SPDX license tags to allow machines to check licenses.
Signed-off-by: Paul Spooren <mail@aparcar.org>
[rebase, keep some Copyright lines, sharpen commit message]
Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
Packages that are in-tree only often lack a PKG_VERSION and only use the
PKG_RELEASE to mark changes. Using COMMITCOUNT/AUTORELEASE variables
causes an issue as both variables are empty during the metadata DUMP
phase.
Instead of leaving these variables empty and causing an error message
like below, set the variables to 0 during dumping. On actual building
the variable is evaluated causing in a value above 0.
ERROR: please fix package/utils/px5g-wolfssl/Makefile - \
see logs/package/utils/px5g-wolfssl/dump.txt for details
Makefile:48: *** Package/px5g-wolfssl is missing the VERSION field. Stop.
Reported-by: Daniel Golle <daniel@makrotopia.org>
Reported-by: Stijn Segers <foss@volatilesystems.org>
Reported-by: Stijn Tintel <stijn@linux-ipv6.be>
Signed-off-by: Paul Spooren <mail@aparcar.org>
The newly added $(COMMITCOUNT) variable was wrongly increased by plus
one. The addition should have been only added to $(AUTORELEASE) as
OpenWrt traditionally starts counting at one rather than zero.
$(AUTORELEASE) counts the commits since the last bump, which is zero on
the version bump commit itself.
This commit increases $(AUTORELEASE) by one while leaving $(COMMITCOUNT)
as is.
The base-files package is the only package using $(COMMITCOUNT) so far
and requires a pseudo commit to keep the PKG_RELEASE correct. A non
functional change (Copyright bump) is done in the next commit.
Signed-off-by: Paul Spooren <mail@aparcar.org>
The lack of bumped PKG_RELEASE variables is a recurring theme on the
mailing list and in GitHub comments. This costs precious review time,
a rare good within the OpenWrt project.
Instead of relying on a manually set PKG_RELEASE this commit adds a
`commitcount` function that uses the number of Git commits to determine
the release. The function is called via the variables `$(AUTORELEASE)`
or `$(COMMITCOUNT)`. The `PKG_RELEASE` variable can be set to either of
the two.
- $(AUTORELEASE):
Release is automagically set to the number of commits since the last
commit containing either ": update to " or ": bump to ".
Example below:
$ git log packages/foobar/
foobar: fixup file location
foobar: disable docs
foobar: bump to 5.3.2
foobar: fixup copyright
Resulting package name: foobar_5.3.2-3_all.ipk, two package changes
since the last upstream version change, using a 1 based counter.
- $(COMMITCOUNT):
For non-traditional versioning (x.y.z), most prominent `base-files`,
this variable contains the total number of package commits.
The new functionality can also be used by other feeds like packages.git.
In case no build information is available, e.g. when using release
tarballs, the SOURCE_DATE_EPOCH is used to have a reproducible release
identifier.
Suggested-by: Daniel Golle <daniel@makrotopia.org>
Signed-off-by: Paul Spooren <mail@aparcar.org>
Some packages fail to build on arm64 when PKG_ASLR_PIE_ALL=y, due to
machine-specific size restrictions on the global offset table.
While the manual instructs to recompile with -fPIC if it fails with
-fpic, by doing this per package, there is still a risk of random
breakage due to version bumps or other changes, so let's use -fPIC on
arm64 by default.
While comparing the sizes of 141 packages built with -fpic vs -fPIC,
most packages are either equal or smaller in size. Only 9 of the
compared packages turned out slightly larger.
Signed-off-by: Stijn Tintel <stijn@linux-ipv6.be>
Commit f98878e4c1 ("cmake.mk: set C/CXX compiler for host builds as
well") has introduced regression as it didn't taken usage of ccache into
the account so fix it by handling ccache use cases as well.
In order to get this working we need to export HOSTCXX_NOCACHE in
rules.mk as well.
Fixes: f98878e4c1 ("cmake.mk: set C/CXX compiler for host builds as well")
Reported-by: Ansuel Smith <ansuelsmth@gmail.com>
Tested-by: Ansuel Smith <ansuelsmth@gmail.com>
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Drop our local sstrip copy and use the current ELFKickers upstream
version.
Patch the original makefile in order to avoid building elftoc, since it
fails with musl's elf.h. This is fine, since we only need sstrip anyway.
Finally, add the possibility to pass additional arguments to sstrip and
pass -z (remove trailing zeros) by default, which matches the behaviour
of the previous version.
Signed-off-by: Rui Salvaterra <rsalvaterra@gmail.com>
[shorten long commit msg lines]
Signed-off-by: Paul Spooren <mail@aparcar.org>
Since fakeroot is patched to discover related ressources relative to the
STAGING_DIR_HOST environment variable, there is no need to pass the path
to faked or the preload library manually anymore.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
SELinux support requires setting the appropriate SELinux security context
to files and directories, which needs to happen at build time in order
to support read-only root filesystem scenarios. In order to create these
security contexts, we will have to run some SELinux-specific tools on
the host machine, but that requires root access. This adds support for
fakeroot, which the build process will use to run the SELinux security
context creation and the image creation.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Apply to current master, and adjust commit message
Thomas' original work is available at
http://lists.infradead.org/pipermail/openwrt-devel/2019-November/025976.html.
Signed-off-by: W. Michael Petullo <mike@flyn.org>
[add rules.mk FAKEROOT variable]
Signed-off-by: Paul Spooren <mail@aparcar.org>
[update, fix macos build]
Signed-off-by: Felix Fietkau <nbd@nbd.name>
Remapping the local build path in debug information makes debugging
using ./scripts/remote-gdb harder, because files no longer refer to the full
path on the build host.
For local builds, debug information does not need to be reproducible,
since it will be stripped out of packages anyway.
For buildbot builds, it makes sense to keep debug information reproducible,
since the full path is not needed (nor desired) anywhere.
Signed-off-by: Felix Fietkau <nbd@nbd.name>
Set CCACHE_DIR to $(TOPDIR)/.ccache and CCACHE_BASEDIR to $(TOPDIR).
This allows to do clean and dirclean. Cache hit rate for test build
after dirclean is ~65%.
If CCACHE is enabled stats are printed out at the end of building process.
CCACHE_DIR config variable allows to override default, which could be useful
when sharing cache with many builds.
cacheclean make target allows to clean the cache.
Changes from v1:
- remove ccache directory using CCACHE_DIR variable
- remove ccache leftovers from sdk and toolchain make files
- introduce CONFIG_CCACHE_DIR variable
- introduce cacheclean make target
Signed-off-by: Roman Yeryomin <roman@advem.lv>
"$(STAGING_DIR)/include" was carried over from buildroot-ng to OpenWrt
in commit 60c1f0f64d. buildroot has
dropped this directory a long time ago.
In OpenWrt the directory is still created by the PrepareStaging macro
and is part of the default TARGET_CPPFLAGS. But nothing at all installs
headers into this directory, nor should anything be installed under this
path.
Removing this directory from TARGET_CPPFLAGS will cut down the log noise
a bit. Not only will CPPFLAGS be shorter, there will be less warnings
set off by "-Wmissing-include-dirs" (or even failures when paired with
"-Werror"). After all the directory does not even _exist_ in the SDKs,
which are used on the build bots when building packages (see [1] and
[2]).
make[8]: Entering directory '/builder/shared-workdir/build/sdk/build_dir/target-aarch64_generic_musl/libmbim-1.20.0/src/common'
CC libmbim_common_la-mbim-common.lo
cc1: error: /builder/shared-workdir/build/sdk/staging_dir/target-aarch64_generic_musl/include: No such file or directory [-Werror=missing-include-dirs]
cc1: all warnings being treated as errors
[1] https://github.com/openwrt/packages/issues/10377
[2] https://github.com/openwrt/packages/pull/10378
Signed-off-by: Sebastian Kemper <sebastian_ml@gmx.net>
Acked-by: Jo-Philipp Wich <jo@mein.io>
Acked-by: Rosen Penev <rosenp@gmail.com>
-ffile-prefix-map=OLD=NEW is an alias for both -fdebug-prefix-map and
-fmacro-prefix-map and is available since GCC 8.
Co-Developed-by: Andre Heider <a.heider@gmail.com>
Signed-off-by: Andre Heider <a.heider@gmail.com>
Signed-off-by: Paul Spooren <mail@aparcar.org>
[refactored into separate commit]
Signed-off-by: Petr Štetiar <ynezz@true.cz>
This speeds up the packing of the imagebuilder a lot:
imagebuilder-T0.tar.xz real 0m25.199s user 2m45.967s sys 0m1.218s
imagebuilder-T1.tar.xz real 2m02.543s user 2m02.418s sys 0m1.653s
imagebuilder-T2.tar.xz real 1m03.684s user 1m59.931s sys 0m0.587s
imagebuilder-T3.tar.xz real 0m48.033s user 2m02.904s sys 0m0.637s
imagebuilder-T4.tar.xz real 0m38.963s user 2m15.521s sys 0m0.783s
imagebuilder-T5.tar.xz real 0m37.994s user 2m21.461s sys 0m0.919s
imagebuilder-T6.tar.xz real 0m39.524s user 2m48.115s sys 0m1.279s
imagebuilder-T7.tar.xz real 0m34.061s user 2m45.097s sys 0m1.174s
imagebuilder-T8.tar.xz real 0m27.286s user 2m55.449s sys 0m1.329s
imagebuilder-T9.tar.xz real 0m25.205s user 2m44.894s sys 0m1.208s
To keep the output reproducible in any case, we enforce a minimum amount
of 2 threads.
Signed-off-by: Martin Schiller <ms@dev.tdt.de>
[refactored into reusable NPROC var, more verbose commit message]
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Add option BUILD_LOG_DIR to menuconfig to change log destination.
The mix-up of *DIR* and *FOLDER* is confusing however.
Signed-off-by: Paul Spooren <mail@aparcar.org>
ESED is SED with extended regular expressions turned on.
Command line and usage are the same as for SED.
Signed-off-by: Konstantin Demin <rockdrilla@gmail.com>
Added boolean symbol for GCC 8 and higher, when we add newer GCC, we don't have
to modify rules.mk to keep things consistant.
Fixes: da9d760 ("rules.mk: replace iremap when using GCC 8")
Signed-off-by: Syrone Wong <wong.syrone@gmail.com>
In order to be able to better compare files to sync in the future, include
all BIN_DIR subdirectories in the checksum calculation.
To not break existing applications, restrict the recursive checksumming to
CONFIG_BUILDBOT for now.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
In addition to removing redundant code, this fixes various issues in
IB-generated images that have been fixed in prepare_rootfs before,
including better handling of CONFIG_CLEAN_IPKG and enabling of initscripts
from FILES.
We also reuse the opkg macro and remove --force-... flags that have been
removed from rootfs.mk as well.
Signed-off-by: Matthias Schiffer <mschiffer@universe-factory.net>
The only users of this were the python packages
from the `packages` feed.
The 2 python interpreters would export some mk
files (e.g. python-package.mk) and then other
python packages would include it via this rule.
But there's a few things wrong with this approach,
most of them drawing from the fact that python host
needs to be built first, to export these mk files.
By now all uses of include_mk have been corrected
in the feeds and this can be removed.
Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
This directory is never created, nor is it used anywhere. Using it in
HOST_LDFLAGS causes a linker warning to be emitted on macOS, which
messes with cmake configure tests.
Signed-off-by: Felix Fietkau <nbd@nbd.name>
Set TMPDIR to the same value as the existing TMP_DIR variable in order to
let gcc and various other utilities use the local temporary directory
instead of the system-wide one.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
The confvar macro is adjusted to not only consider if a variable has a
value or not, but also the value itself. Instead of creating a string of
'y' and 'n' characters, all variable names and values are concatenated
and hashed.
Signed-off-by: Matthias Schiffer <mschiffer@universe-factory.net>
Passing -iremap argument separately causes problems with projects that
use scons and its ParseFlags function. Consider this SConscript
example:
env = Environment()
d = env.ParseFlags("-iremap one:two")
ParseFlags will interpret one:two as a file name and the returned dict
d will contain only "-iremap". When the -iremap is passed to the
compiler without an argument, compilation obviously fails.
Signed-off-by: Michal Sojka <sojkam1@fel.cvut.cz>
For external toolchain, we also know the type of C library used, and the
toolchain triplet may not always be reflective of that, therefore make
$(TARGET_DIR_NAME) suffixed with _$(LIBC).
Signed-off-by: Florian Fainelli <f.fainelli@gmail.com>
Introduce a new location STAGING_DIR_IMAGE which is intended to be used by
bootloader iamges and similar image-related artifacts.
This directory is guaranteed to be persistent across kernel upgrades which
might involve a removal of KERNEL_BUILD_DIR and is guranteed to be bundled
with the image builder.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Using a single host package staging dir (and build dir) significantly speeds up
builds when multiple targets are built in succession, especially for large host
packages like NodeJS.
$(STAGING_DIR)/host is kept in addition to $(STAGING_DIR_HOSTPKG) in most
places; it is still used as destination for host files in Build/InstallDev.
Signed-off-by: Matthias Schiffer <mschiffer@universe-factory.net>
This makes it easier to unify versioning of git based package downloads.
PKG_SOURCE_DATE along with an 8-character abbreviation of the git hash
is used as PKG_VERSION, PKG_RELEASE should be used like normal packages.
Signed-off-by: Felix Fietkau <nbd@nbd.name>
This is intended to be used for a wide array of package sanity checks.
The first check that is implemented is for the hash of downloaded files.
It checks:
- Missing hash
- Use of SHA256 instead of MD5
- dl/<file> hash not matching hash in makefile
- deprecated MD5SUM variable
The deprecated MD5SUM variable check is skipped for feeds/ until OpenWrt
is updated as well
Signed-off-by: Felix Fietkau <nbd@nbd.name>
Add a STAGING_DIR_HOSTPKG variable which refers to $(STAGING_DIR)/host in order
to prepare support for relocating that directory in the future.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
