From ba722ef4d869ea759922287afab546f0f88ec74f Mon Sep 17 00:00:00 2001 From: Milosz Wasilewski Date: Tue, 10 Feb 2026 13:59:19 +0000 Subject: [PATCH] Remove lmp-testplan from the docs Internal LmP test plan is now obsolete and should be removed from the docs. Signed-off-by: Milosz Wasilewski --- source/glossary/index.rst | 1 - .../reference-manual/testing/lmp-testplan.rst | 240 ------------------ source/reference-manual/testing/testing.rst | 3 - 3 files changed, 244 deletions(-) delete mode 100644 source/reference-manual/testing/lmp-testplan.rst diff --git a/source/glossary/index.rst b/source/glossary/index.rst index dd6dc535e..b6f581619 100644 --- a/source/glossary/index.rst +++ b/source/glossary/index.rst @@ -46,7 +46,6 @@ Glossary * :ref:`Reference Manual, LmP ` * :ref:`Updating, LmP ` - * :ref:`Test plan, LmP ` * :ref:`Customizing, LmP ` Target diff --git a/source/reference-manual/testing/lmp-testplan.rst b/source/reference-manual/testing/lmp-testplan.rst deleted file mode 100644 index ae8982978..000000000 --- a/source/reference-manual/testing/lmp-testplan.rst +++ /dev/null @@ -1,240 +0,0 @@ -.. _ref-lmp-testplan: - -Test Plan -######### - -Below you will find the test plan Foundries.io uses for the FoundriesFactory™ Platform. -Test results for each release can be found in the `release notes `_. - -What to Test -============ - -In the context of this test plan, **mandatory** means the testing that must be performed to release the software. -Moreover, all tests must pass before the release is announced. -**Optional** means that the tests may be performed if time allows. -If optional test results are missing, release can be announced. - -Testing focuses on the mandatory features of each FoundriesFactory release: - -* Linux microPlatform -* Base OS running with OSTree based root filesystem -* Aktualizr-lite daemon -* Docker engine running Docker compose apps -* Fioconfig - -This includes the open source Linux® microPlatform (LmP) builds on supported SoC vendor reference development evaluation kits (:term:`EVK`). - -Remaining elements of the system may be tested as optional but are not as important. -For example, testing of I/O interfaces done case-by-case depending on customer requirements and hardware capabilities - -Most tests are in the `test-definitions repository`_. - -LmP Test Plan -------------- - -Linux® microPlatform testing must at least cover: - -* Boot testing - - * Ensure kernel boots and mounts root filesystem - -* Network testing - - * Ensure that networking works correctly - * Ensure NetworkManager works correctly - (as it’s a part of default OS installation) - -* Mandatory services testing - - * Ensure that systemd runs and all mandatory services are started - as required. - - * Ensure that aktualizr daemon runs - * Ensure that fioconfig daemon runs - * Ensure that NetworkManager runs - - * Ensure that Docker engine runs - * Ensure that example Docker app runs - -OS Features ------------ - -Boot Test and Smoke Test -~~~~~~~~~~~~~~~~~~~~~~~~ - -* Mandatory: - - * Check kernel version - * Check block devices - * Check network devices - * Ping test - * Optee test — xtest (binaries built into the OS image) - * Timesyncd - - * Ensure NTP is synchronized correctly - -* Optional - - * Check device presence (i2c, spi, video, audio, gpio, …) - depending on HW capabilities - * Check driver presence - -OSTree -~~~~~~ - -* Optional - - OSTree has a runtime test suite. - It is used for validation after building libostree, but can potentially be used to test a live system. - However, this may be destructive to the filesystem. - -Docker -~~~~~~ - -* Mandatory - - * Docker smoke test - This includes running example docker containers (hello-world) - -* Optional - - * Docker-compose integration test - * Docker networking - - * Default bridge - * User-defined bridge - * Host - -Networking on Host -~~~~~~~~~~~~~~~~~~ - -* Mandatory - - * DHCP - - * Check if IPv4 address is assigned properly - * Check if IPv6 address is assigned properly - * Check for wired and wireless interface (requires external hardware) - - * DNS - - * Check if name resolution works with default settings (default DNS must be validated before running tests) - -Interface Testing (Optional) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Various interfaces are tested depending on the hardware and customer requirements. -Current plan is to execute the following tests: - -* HDMI (HDMI capture device) - -Device Update -------------- - -Aktualizr (OTA API) -~~~~~~~~~~~~~~~~~~~ - -* Mandatory - - * Update - - * Update of Docker compose apps (new Target) - - * *From* previous Target - - * Update of base OS - - * From previous platform Target - * From previous *release* platform Target - - * Rollback - - * Rollback of base OS - -Device Config (Fioconfig) -~~~~~~~~~~~~~~~~~~~~~~~~~ - -* Mandatory, to test whether: - - * Factory specific configs are applied properly - * Group specific configs are applied properly - * Device specific configs are applied properly - * Both encrypted and non-encrypted configs are available on the device - -How To Test -=========== - -LmP Tests ---------- - -Boot Testing -~~~~~~~~~~~~ - -There are several kinds of tests involved. -Basic boot test should be mandatory for all subsequent tests. -If the boot test fails all other testing should be abandoned. -There are 2 scenarios for boot testing: - -* Initial provisioning - - This happens when the software is delivered to the board for the first time. - Since the aktualizr is not yet running on the board, - provisioning has to be done in some other way. - This is strongly dependent on hardware limitations and boot source. - For example, RaspberryPi can boot from an SD card, and works well with available SDMux devices. - Conversely, some boards may boot from eMMC and require SoC-specific tooling for initial flashing. - Both of these provisioning methods are supported by LAVA. - Therefore, it is proposed to use LAVA for initial provisioning, boot, and reboot testing in this scenario. - -* Software update (OS update) - - Booting after a software update can be checked in 2 ways: - with either aktualizr-lite or a container running on the board, or with an external tool. - - When checking reboot after update testing rig needs to know: - - * When the test starts (on old Target) - * What are the starting (old) and ending (new) Targets and OSTree hashes - * When the test is finished (aktualizr performs update, system is rebooted) - -Basic Tests -~~~~~~~~~~~ - -Basic tests are executed on the target either using the fiotest container (running commands on host) or LAVA. -Which tool depends on the tested scenario. -We are currently testing 2 scenarios: - -* *Manufacturing* scenario - - LAVA can execute tests in Linux shell on the Target and parse results from the serial console. - Tests are executed after flashing an image to the board. - DUT always starts fresh without any previously running software. - -* *Rolling update* scenario - - ``Test-runner.py`` is a script from test-definitions repository. - It is able to run tests on the remote OS using SSH as a connection medium. - It is used to execute tests in the ‘rolling update’ scenario. - Test results are reported to both qa-reports and FIO backend. - Reporting to FIO backend is done with fiotest. - Fiotest is also responsible for starting a test round following an OTA update. - Test plan executed in the “rolling update” scenario is limited. - Tests disabling networking and potentially corrupting the OS are disabled. - -* Docker apps update - - Testing of Docker apps update should be done using a container registered for aktualizr-lite callbacks. - This way we are as close as possible to testing a production setup. - -When to Test -============ - -A testing round is started after every merge to ``lmp-manifest``. -If the build is successful, all testing Factories pull the latest source from ``lmp-manifest`` and merge to their working branches. -A successful build in the testing Factory triggers tests on the devices. -OTA update is delivered to the *rolling update* devices. -This also triggers a testing round on the new Target. -For a release candidate build, additional manual tests are performed. - -.. _test-definitions repository: - https://github.com/linaro/test-definitions diff --git a/source/reference-manual/testing/testing.rst b/source/reference-manual/testing/testing.rst index 757a34db6..4248f4afe 100644 --- a/source/reference-manual/testing/testing.rst +++ b/source/reference-manual/testing/testing.rst @@ -5,14 +5,11 @@ Testing This section provides an overview of the testing done for the FoundriesFactory™ Platform. -:ref:`ref-lmp-testplan` describes the *internal* testing process we follow to help ensure the quality of our releases. - For testing that you may wish to perform, :ref:`ref-testing-architecture` introduces the workflow and general approach. :ref:`ref-fiotest` introduces the automated testing capabilities available to your Factory using a testing API. .. toctree:: :maxdepth: 1 - lmp-testplan testing-architecture fiotest