Re: [PATCH] tests: Improve 'make test': Run all build-time runnable tests

[Jan Beulich <jbeulich@xxxxxxxx>]

On 28.05.2026 17:25, Bernhard Kaindl wrote:
> --- a/Makefile
> +++ b/Makefile
> @@ -92,6 +92,7 @@ build-docs:
>  .PHONY: test
>  test:
>       $(MAKE) -C tools/python test
> +     $(MAKE) -C tools/tests test

This means they're run sequentially, when really they could run in parallel.

> --- a/tools/Rules.mk
> +++ b/tools/Rules.mk
> @@ -189,7 +189,7 @@ INSTALL_PYTHON_PROG = \
>  %.opic: %.S
>       $(CC) $(CPPFLAGS) -DPIC $(CFLAGS) -fPIC -c -o $@ $< $(APPEND_CFLAGS)
>  
> -subdirs-all subdirs-clean subdirs-install subdirs-distclean 
> subdirs-uninstall: .phony
> +subdirs-all subdirs-clean subdirs-install subdirs-distclean subdirs-run 
> subdirs-test subdirs-uninstall: .phony
>       @set -e; for subdir in $(SUBDIRS) $(SUBDIRS-y); do \
>               $(MAKE) subdir-$(patsubst subdirs-%,%,$@)-$$subdir; \
>       done
> @@ -200,6 +200,20 @@ subdir-all-% subdir-clean-% subdir-install-% 
> subdir-uninstall-%: .phony
>  subdir-distclean-%: .phony
>       $(MAKE) -C $* distclean
>  
> +subdir-run-%: .phony
> +     $(MAKE) -C $* run
> +
> +subdir-test-%: .phony
> +     $(MAKE) -C $* test
> +
> +ifeq ($(CC),$(HOSTCC))

I understand you lift this up from uses elsewhere, but I'd suggest thinking
of ways to improve things while generalizing. CC and HOSTCC can very well
be different, and the output can still be run. What wants to be sufficiently
similar is e.g. the output of -dumpmachine (whether the middle part of the
triplet needs to match is not quite clear to me; whether e.g. the last part
differing in a suffix [e.g. -linux vs -linux-gnu] may also want considering
a match I similarly can't quite say; what I can say it that pre-built
compilers on the distro I'm looking at yield x86_64-suse-linux, while my
self-built gcc-s yield x86_64-pc-linux-gnu, yet both obviously target the
same thing).

> +define RUN_TARGETS_IF_CC_IS_HOSTCC
> +set -ex; for test in $? ; do LD_LIBRARY_PATH=$(DISTDIR)/install$(libdir) 
> ./$$test ; done

If LD_LIBRARY_PATH wants fiddling with here, I think you want to prepend to
what may already be there. Since quite a few of the tests don't reference
any of our libraries, I'm not convinced though that this wants/needs doing
here (instead of perhaps per test).

> --- a/tools/tests/cpu-policy/Makefile
> +++ b/tools/tests/cpu-policy/Makefile
> @@ -15,9 +15,9 @@ endif
>  .PHONY: all
>  all: $(TARGETS)
>  
> -.PHONY: run
> -run: $(TARGETS)
> -     ./$<
> +.PHONY: run test
> +run test: $(TARGETS)
> +     $(call RUN_TARGETS_IF_CC_IS_HOSTCC)

Try (perhaps with a trivial, independent Makefile containing just

run test:
        @echo $@

) to invoke "make run test". You'll see the rule is executed twice, once for
each target. I think you want

run: test

or the other way around. Generally I'd expect "run" to be the "on build host"
target (based on how we use it right now, with ./$(TARGET) invocations). The
other mode might better run stuff from the install/ directory. However, which
one is "run" and which one is "test" is of course simply a matter of
convention / agreement.

> --- a/tools/tests/mem-claim/Makefile
> +++ b/tools/tests/mem-claim/Makefile
> @@ -11,14 +11,7 @@ all: $(TARGETS)
>  #  Can also be called with "make run-tests-mem-claim" from the toplevel.
>  .PHONY: run
>  run: $(TARGETS)
> -ifeq ($(CC),$(HOSTCC))
> -     set -e;             \
> -     for test in $? ; do \
> -             ./$$test ;  \
> -     done
> -else
> -     $(warning HOSTCC != CC, will not run test)
> -endif
> +     $(RUN_TARGETS_IF_CC_IS_HOSTCC)
>  
>  # The tests in this directory need to run in a privileged domain (Dom0)
>  # with libxenctrl and a matching the hypervisor to test running it.

Hmm, this doesn't match present staging, yet you also don't mention any
prereq-s this is to go on top of.

> --- a/tools/tests/vpci/Makefile
> +++ b/tools/tests/vpci/Makefile
> @@ -8,11 +8,13 @@ all: $(TARGET)
>  
>  .PHONY: run
>  run: $(TARGET)
> -ifeq ($(CC),$(HOSTCC))
> -     ./$(TARGET)
> -else
> -     $(warning HOSTCC != CC, will not run test)
> -endif
> +     $(call RUN_TARGETS_IF_CC_IS_HOSTCC)
> +
> +# The tests in this directory need to run in a privileged domain (Dom0)
> +# with libxenctrl and a matching the hypervisor to test running it.
> +# Therefore, they don't run with the native test target for build-time tests.
> +.PHONY: test
> +test:

I don't think this is true here.

Jan