Convert this repository into a test case collection for NTFC #102

szafonimateusz-mi · 2026-01-03T10:58:39Z

This PR adds test cases for NTFC from https://github.com/szafonimateusz-mi/nuttx-testing

Related issue in nuttx repo: apache/nuttx#17717

nuttx-testing repository now becomes the place where NTFC compatible unit tests for Apache NuttX are kept Signed-off-by: p-szafonimateusz <p-szafonimateusz@xiaomi.com> Co-authored-by: nijie3 <nijie3@xiaomi.com> Co-authored-by: v-yangjing26 <v-yangjing26@xiaomi.com> Co-authored-by: v-wuguohui1 <v-wuguohui1@xiaomi.com> Co-authored-by: chenjiahua1 <chenjiahua1@xiaomi.com> Co-authored-by: chenzihan1 <chenzihan1@xiaomi.com> Co-authored-by: tengshuangshuang <tengshuangshuang@xiaomi.com> Co-authored-by: v-xujian5 <v-xujian5@xiaomi.com> Co-authored-by: wangxingxing <wangxingxing@xiaomi.com> Co-authored-by: guanyi <guanyi@xiaomi.com> Co-authored-by: yangsong8 <yangsong8@xiaomi.com> Co-authored-by: v-zhanhaixiang <v-zhanhaixiang@xiaomi.com> Co-authored-by: yezhonghui <yezhonghui@xiaomi.com> Co-authored-by: zhangsixuan1 <zhangsixuan1@xiaomi.com> Co-authored-by: likun17 <likun17@xiaomi.com> Co-authored-by: dongjiuzhu1 <dongjiuzhu1@xiaomi.com> Co-authored-by: chenrun1 <chenrun1@xiaomi.com> Co-authored-by: yangyumeng3 <yangyumeng3@xiaomi.com> Co-authored-by: guohao15 <guohao15@xiaomi.com> Co-authored-by: jingfei <jingfei@xiaomi.com> Co-authored-by: pengyinjie <pengyinjie@xiaomi.com> Co-authored-by: huojianchao<huojianchao@xiaomi.com> Co-authored-by: wangshaoxin <wangshaoxin@xiaomi.com> Co-authored-by: huyongkang <huyongkang@xiaomi.com> Co-authored-by: xucheng5 <xucheng5@xiaomi.com> Co-authored-by: makejian <makejian@xiaomi.com> Co-authored-by: leidonglin <leidonglin@xiaomi.com> Co-authored-by: wangmingrong1 <wangmingrong1@xiaomi.com>

CONFIG_TESTING_MM is CONFIG_TESTING_HEAP in upstream Signed-off-by: p-szafonimateusz <p-szafonimateusz@xiaomi.com>

Signed-off-by: p-szafonimateusz <p-szafonimateusz@xiaomi.com>

ltp_interfaces_sigrelse_1_1 fails for sim/citest on CI Signed-off-by: p-szafonimateusz <p-szafonimateusz@xiaomi.com>

change expected string for cat011 so it pass for rv-virt/citest64 Signed-off-by: p-szafonimateusz <p-szafonimateusz@xiaomi.com>

Signed-off-by: raiden00pl <raiden00@railab.me>

cederom

Thank you @szafonimateusz-mi we finally have separate location for testing :-)

Would it be possible to put all NTFC tests under ntfc/ directory please? We have some more ideas for testing and each of them would have its own dedicated directory that way :-)

xiaoxiang781216 · 2026-01-04T15:26:24Z

@szafonimateusz-mi so do we plan to move ALL test case from apps/ to this repo and remove them from nuttx-apps repo?

simbit18 · 2026-01-04T16:27:35Z

arch/os/performance/.conf/ddl.sql

+  `describe` varchar(24) DEFAULT '' NOT NULL COMMENT '描述',
+  `data_sync_task_id` BIGINT(11) DEFAULT 0 NOT NULL COMMENT '数据同步任务id',
+  `update_time` DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '数据更新时间'
+);


Hi @szafonimateusz-mi, this SQL script file was created by a MySQL database, correct? Is it necessary?

You're right! I need to make sure this file isn't being used. I've now found that arch/os/performance is broken anyway by one of my previous changes

szafonimateusz-mi · 2026-01-04T17:02:16Z

@cederom This needs its own repository because it's Python code, and QA will be handled by the tox.
Moreover, test cases for NTFC must be grouped per repository. NTFC uses the concept of test cases module. Each test cases module has its own configuration, which must be in repo root directory.
NTFC is intended to support the detection and execution of multiple test modules at once (for example official nuttx test cases, vendor-specific test cases and custom user test cases). Tests auto detection from pytest is used for this, which means that the repository must be fully dedicated to NTFC tests. An example test cases organization should look like this:

test-cases     (all test cases in one place)
|
|-- nuttx-testing 
|      |- ... (test cases)
|      \- ntfc.yaml (module configuration)
|
|-- vendor-testing
|    |- ... (test cases)
|    \- config.yaml (module configuration)
|
|-- user-testing
     |- ... (test cases)
     \- config.yaml (module configuration)

If this repository is to be used for other purposes, it's better to create a new one for the NTFC test cases only.

@xiaoxiang781216 I don't see the need to move apps/testing. I think it would only introduce unnecessary complications in build process.

cederom · 2026-01-04T19:10:59Z

Okay, thanks for explaining @szafonimateusz-mi.. it would be best to keep all testing tools in one repo but if there is no other way we may create a dedicated repo like nuttx-testing-ntfc or just nuttx-ntfc that should not be a problem :-)

xiaoxiang781216 · 2026-01-05T02:03:53Z

arch/atomic/test_arch_atomic_integration.py

+
+
+@pytest.mark.dep_config("CONFIG_TESTING_ATOMIC")
+@pytest.mark.cmd_check("atomic_main")


it isn't valid if we change test case to elf binary.

yes, ELF binary is not supported now. pytest.mark.cmd_check is a remnant from the Xiaomi CI, but I think it should be removed in the future. Configuration checking (pytest.mark.dep_config) should be sufficient for test auto-detection. It won't be perfect, but it will work with ELF bin. In the worst case, the test case will report a failure due to "command not found", which can also help in detecting a certain class of bugs.

Another approach is to dynamically scan whether a given directory contains ELF bin file, then we can leave pytest.mark.cmd_check and add this feature in NTFC.

simbit18 · 2026-01-05T14:59:07Z

arch/task/test_arch_task_api_integration.py

+def test_vela_sched_interface_task_test07():
+
+    cmd = "vela_sched_interface_task_test07"
+    EXPECTED_LIST = "TEST PASSED !"


Hi @szafonimateusz-mi , I think the functions in this file need to be renamed.

simbit18 · 2026-01-05T15:02:11Z

arch/pthread/test_arch_pthread_api_integration.py

+@pytest.mark.skip("No Ready")
+@pytest.mark.cmd_check("vela_sched_pthreads_pth04_main")
+def test_vela_sched_pthreads_pth04_integration():
+    cmd = "vela_sched_pthreads_pth04"


Hi @szafonimateusz-mi , I think the functions in this file need to be renamed.

simbit18 · 2026-01-05T15:04:46Z

fs/stability/test_fs_stability.py

+def test_stability_test04(self):
+    pytest.product.sendCommand("vela_fs_stability_test04", "TEST PASSED", timeout=180)
+    ret = pytest.product.sendCommand(
+        "vela_fs_stability_test04", "TEST PASSED", timeout=20


@szafonimateusz-mi, in this file too, the functions need to be renamed.

simbit18 · 2026-01-05T15:07:42Z

fs/stress/test_fs_stress_integration.py

+    @pytest.mark.cmd_check("vela_fs_stress_write_full_file_main")
+    def test_stress_write_full_file(self):
+        ret = pytest.product.sendCommand(
+            "vela_fs_stress_write_full_file /tmp", self.EXPECTED_LIST, timeout=120 * 60


@szafonimateusz-mi also in this file

szafonimateusz-mi and others added 13 commits January 3, 2026 11:51

fix heap tests

762333b

CONFIG_TESTING_MM is CONFIG_TESTING_HEAP in upstream Signed-off-by: p-szafonimateusz <p-szafonimateusz@xiaomi.com>

main_core is now core0 in ntfc

67a7c55

Signed-off-by: p-szafonimateusz <p-szafonimateusz@xiaomi.com>

add ntfc.yaml file

b41babb

Signed-off-by: p-szafonimateusz <p-szafonimateusz@xiaomi.com>

add config requirements for this test module

175e1ae

Signed-off-by: p-szafonimateusz <p-szafonimateusz@xiaomi.com>

update readme

ebf5666

Signed-off-by: p-szafonimateusz <p-szafonimateusz@xiaomi.com>

ltp tests depends on TESTING_LTP

29c686d

Signed-off-by: p-szafonimateusz <p-szafonimateusz@xiaomi.com>

add livuv tests

a6a76cf

Signed-off-by: p-szafonimateusz <p-szafonimateusz@xiaomi.com>

skip failing libuv cases

6643a8f

Signed-off-by: p-szafonimateusz <p-szafonimateusz@xiaomi.com>

add usrsocktest example test

196e1c0

Signed-off-by: p-szafonimateusz <p-szafonimateusz@xiaomi.com>

temporarily blacklist ltp_interfaces_sigrelse_1_1

4ba5e53

ltp_interfaces_sigrelse_1_1 fails for sim/citest on CI Signed-off-by: p-szafonimateusz <p-szafonimateusz@xiaomi.com>

fix test_cat011

fd96cbf

change expected string for cat011 so it pass for rv-virt/citest64 Signed-off-by: p-szafonimateusz <p-szafonimateusz@xiaomi.com>

update github description

b75969e

Signed-off-by: raiden00pl <raiden00@railab.me>

This was referenced Jan 3, 2026

initial support for automated testing with NTFC apache/nuttx#17728

Merged

[FEATURE] Migrate CI test cases to NTFC apache/nuttx#17717

Open

cederom requested changes Jan 4, 2026

View reviewed changes

cederom requested review from acassis, jerpelea, linguini1, lupyuen, rcsim, simbit18 and xiaoxiang781216 January 4, 2026 10:35

simbit18 reviewed Jan 4, 2026

View reviewed changes

xiaoxiang781216 reviewed Jan 5, 2026

View reviewed changes

simbit18 reviewed Jan 5, 2026

View reviewed changes



		@pytest.mark.dep_config("CONFIG_TESTING_ATOMIC")
		@pytest.mark.cmd_check("atomic_main")

Convert this repository into a test case collection for NTFC #102

Are you sure you want to change the base?

Convert this repository into a test case collection for NTFC #102

Uh oh!

Conversation

szafonimateusz-mi commented Jan 3, 2026

Uh oh!

cederom left a comment

Choose a reason for hiding this comment

Uh oh!

xiaoxiang781216 commented Jan 4, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

szafonimateusz-mi commented Jan 4, 2026

Uh oh!

cederom commented Jan 4, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants