Skip to content

GEISA-LEE-tests: improve network-related tests to test actual platform capability, and explain skipped tests - #54

Open
ndam-sfl wants to merge 5 commits into
geisa:mainfrom
ndam-sfl:exercise_lee_tests
Open

ndam-sfl wants to merge 5 commits into
geisa:mainfrom
ndam-sfl:exercise_lee_tests

Conversation

@ndam-sfl

Copy link
Copy Markdown
Contributor
  • APP-SHALL-10, 11, 12: GEISA lets each platform choose how it enforces
    the network policy. The tests no longer read the nft rules back. They
    set the policy up with nft, or with iptables when nft is not installed,
    then run geisa-test and judge the platform by how the traffic is
    handled: a granted port is reachable, a port that is not granted is
    not, and a granted port stops being reachable once the daily volume is
    used. When the board cannot set up a limit, the matching test is
    skipped instead of failing.
  • Every conditional test now says why it was skipped. The reason shows
    in the console, the JUnit XML and the PDF report. This bumps the
    cukinia and test-report-pdf submodules to the upstream commits that
    carry the skip reasons.
  • The launcher prints the board, kernel and distribution before the
    tests start, so a report can be traced back to its platform.
  • README: the manual launch section points to the same target
    requirements as the automatic launcher, plus a few typo fixes.

Before the tests start, print what we are running on: the board, the
kernel, the distribution and a few other details. This works over both
SSH and serial, and makes it easy to tell which platform a result came
from.

Change-Id: I0ec2671551d8b78584a053c2f9eb7ab1c3b97c3e
Signed-off-by: Nghia Dam <nghia.dam@savoirfairelinux.com>
…commits

Change-Id: I422a524a8574c248f7314cdfc45f2a895c94a1a0
Signed-off-by: Nghia Dam <nghia.dam@savoirfairelinux.com>
When a test is skipped, it now says why. Cukinia shows the reason in the
console and the JUnit XML, and the launcher passes --show_skip_reasons
so the PDF report shows it too. Both rely on the cukinia and
test-report-pdf submodules updated in the previous commit.

Change-Id: Ia3425d4e892662674d88c15af51f61b911e316d9
Signed-off-by: Nghia Dam <nghia.dam@savoirfairelinux.com>
GEISA lets each platform choose how it enforces the network policy. The
tests now set the policy up with nft, or with iptables when nft is not
installed, and judge the platform by how it actually handles the traffic
rather than by reading the rules back. When the board cannot set up a
limit, the matching test is skipped with the reason.

Change-Id: Ie56939abfe0376d6b100660ab258e7235f3740c6
Signed-off-by: Nghia Dam <nghia.dam@savoirfairelinux.com>
Point the manual launch section to the target requirements of the
automatic launcher, run shellcheck on the same files as the CI, and fix
a handful of typos.

Change-Id: If60c67e0b46ffee8e986733b5b848519ff9c3257
Signed-off-by: Nghia Dam <nghia.dam@savoirfairelinux.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant