Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
160 changes: 159 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,161 @@
# Processor CI Tests

[![Pylint](https://github.com/LSC-Unicamp/processor-ci-tests/actions/workflows/pylint.yml/badge.svg)](https://github.com/LSC-Unicamp/processor-ci-tests/actions/workflows/pylint.yml)
[![Pylint](https://github.com/LSC-Unicamp/processor-ci-tests/actions/workflows/pylint.yml/badge.svg)](https://github.com/LSC-Unicamp/processor-ci-tests/actions/workflows/pylint.yml)

Standalone home for the Processor CI RISC-V verification assets: ISA tests, reference memories, benchmark build support, and the serial test runner used to validate a processor core.

## What this repo contains

- `riscv_test_runner/`: Python package for the test runner
- `riscv_test_runner/core/`: runner, `-march` parsing, and small data helpers
- `main.py`: stable compatibility wrapper around the package CLI
- `tests/`: RISC-V ISA test corpus grouped by architecture extension and level
- `benchmarks/`: bare-metal benchmark sources, ports, linker scripts, and build helpers
- `third_party_repositories/`: imported or converted upstream test material
- `config.json`: example test directory configuration consumed by the runner
- `docs/`: repository layout and maintenance notes
- `requirements.txt`: Python dependencies for the runner and CI checks

## Quick start

1. Create a Python virtual environment.
2. Install dependencies:

```bash
pip install -r requirements.txt
```

3. Make sure the Processor CI communication package is available next to this repository or on `PYTHONPATH`.

4. Run the test runner:

```bash
python -m riscv_test_runner.cli \
--port /dev/ttyUSB0 \
--config config.json \
--march rv32im_zicsr \
--key 0x4e455859
```

Or use the compatibility wrapper:

```bash
python main.py \
--port /dev/ttyUSB0 \
--config config.json \
--march rv32im_zicsr \
--key 0x4e455859
```

Both entrypoints use the same parser and behavior.

## Common usage

- Show CLI help:

```bash
python -m riscv_test_runner.cli --help
```

- Run only the tests selected by an architecture string:

```bash
python -m riscv_test_runner.cli \
-p /dev/ttyUSB0 \
-c config.json \
-m rv32im
```

- Run with a longer serial timeout and core execution timeout:

```bash
python -m riscv_test_runner.cli \
-p /dev/ttyUSB0 \
-c config.json \
-m rv32im_zicsr \
-s 10 \
-t 5000
```

- Run against a core with separate instruction and data memories:

```bash
python -m riscv_test_runner.cli \
-p /dev/ttyUSB0 \
-c config.json \
-m rv32im \
--core-use-two-memories
```

- Request extra benchmark-style tests listed in `config.json`:

```bash
python -m riscv_test_runner.cli \
-p /dev/ttyUSB0 \
-c config.json \
-m rv32im \
-e coremark
```

## Test Layout

The ISA tests live under `tests/<architecture>/<level>/`.

```text
tests/RV32I/basic/src/ assembly source files
tests/RV32I/basic/memory/ compiled instruction memory hex files
tests/RV32I/basic/reference/ expected result hex files
```

Architectures are named like `RV32I`, `RV32M`, `RV64I`, or `RV32Zicsr`. Test levels are:

- `basic`: currently executed by the Python runner
- `advanced`: broader instruction and behavior coverage
- `invalid`: invalid instruction or exception-oriented cases

`config.json` maps each architecture directory to its path and breakpoint addresses. The current sample uses `/eda/processor_ci_tests/...` paths for CI/container use; adjust those paths for local runs if needed.

## Benchmarks

`benchmarks/` keeps benchmark source and porting support separate from the precompiled ISA test corpus.

- `benchmarks/coremark/`: CoreMark source tree
- `benchmarks/port/`: bare-metal boot and platform hooks
- `benchmarks/common/`: minimal C library support
- `benchmarks/target/`: target-specific library and Makefile
- `benchmarks/compile_coremark.sh`: builds CoreMark hex output with `riscv32-unknown-elf-*`
- `benchmarks/compile_common.sh`: builds the minimal common library

Generated benchmark build outputs are written under `benchmarks/build/`.

## CLI Reference

```text
usage: python -m riscv_test_runner.cli [-h] [-b BAUDRATE] -p PORT
[-s SERIAL_TIMEOUT] -c CONFIG
[-m MARCH] [-e EXTRA_TESTS] [-k KEY]
[-t TIMEOUT] [-ctm]
```

- `-p`, `--port`: serial port used to communicate with the FPGA board. Required.
- `-c`, `--config`: JSON test configuration. Required.
- `-b`, `--baudrate`: serial baudrate. Defaults to `115200`.
- `-s`, `--serial-timeout`: serial timeout in seconds. Defaults to `5`.
- `-m`, `--march`: RISC-V architecture string used to select test directories. Defaults to `rv32i`.
- `-e`, `--extra-tests`: comma-separated extra tests from `config.json`.
- `-k`, `--key`: hex sync key expected from the board.
- `-t`, `--timeout`: core execution timeout in clock cycles. Defaults to `1000`.
- `-ctm`, `--core-use-two-memories`: use separate instruction and data memories.

## Developer Notes

- `riscv_test_runner.cli.build_parser()` exposes the CLI parser for tests.
- `riscv_test_runner.cli.main(argv=None)` supports argument injection for wrappers and tests.
- `riscv_test_runner.core.runner.ProcessorCITest` is the hardware-facing runner.
- Root-level `core.*` imports remain as compatibility shims for older scripts.
- Test execution writes a JUnit report named `test_results_<timestamp>.xml`.

## Notes

- The current organization keeps large verification data in place while moving Python implementation into a package, matching the shape used by `processor_ci_discover`.
- The runner currently executes the `basic` category. `advanced` and `invalid` data are preserved for expansion and maintenance.
193 changes: 7 additions & 186 deletions core/__init__.py
Original file line number Diff line number Diff line change
@@ -1,191 +1,12 @@
import time
import os
import sys
"""Compatibility imports for the former root-level core package."""

# processor_ci_communication/core/serial.py
sys.path.append(
os.path.abspath(os.path.join(os.path.dirname(__file__), '../..'))
)
from processor_ci_communication.core.serial import ProcessorCIInterface
from junitparser import TestCase, TestSuite, JUnitXml, Skipped, Error, Failure

# TEST_CATEGORIES = ['basic', 'advanced', 'invalid']
TEST_CATEGORIES = ['basic']
def __getattr__(name: str):
if name == 'ProcessorCITest':
from riscv_test_runner.core.runner import ProcessorCITest

return ProcessorCITest
raise AttributeError(name)

class ProcessorCITest:
def __init__(
self,
port: str = '/dev/ttyUSB0',
baudrate: int = 115200,
serial_timeout: int = 1,
sync_keyword: int = 0x4E455859,
timeout: int = 1000,
directories: list[dict] = [],
extra_tests: list[dict] = [],
core_use_two_memories: bool = False,
):
self.port = port
self.baudrate = baudrate
self.serial_timeout = serial_timeout
self.sync_keyword = sync_keyword
self.timeout = timeout
self.directories = directories
self.extra_tests = extra_tests
self.core_use_two_memories = core_use_two_memories

self.xml = JUnitXml()

self.processor_ci = ProcessorCIInterface(
self.port, self.baudrate, self.serial_timeout
)

id = self.processor_ci.get_module_id()

if id is None:
print('Failed to get module ID. Exiting.')
sys.exit(1)

print(f'Connected to FPGA with ID: {id}')

print('Checking for sync keyword...')
if id != self.sync_keyword:
print(
f'Sync keyword mismatch. Expected: {self.sync_keyword}, Got: {id}'
)
sys.exit(1)

print('Sync keyword matched.')
self.processor_ci.set_timeout(self.timeout)
self.processor_ci.load_msb_accumulator(0)
self.processor_ci.load_lsb_accumulator(0)

print('Testsuite configurated.')

def _execute_test(
self, test_path: str, reference_path: str, breakpoint: int
) -> tuple[bool, int, int]: # result, expected, received
test_file = open(test_path, 'r')
reference_file = open(reference_path, 'r')

test_content = []
test_lines = test_file.readlines()
test_len = len(test_lines)

for i in range(0, breakpoint + 4, 4):
shiffted = i >> 2
if shiffted >= test_len:
test_content.append(0)
else:
test_content.append(int(test_lines[shiffted], 16))
test_file.close()

self.processor_ci.write_from_accumulator(
(breakpoint + 4) >> 2, test_content
)

if(self.core_use_two_memories):
self.processor_ci.write_memory(
breakpoint, 0, self.core_use_two_memories)

self.processor_ci.execute_until_stop()

reference_lines = reference_file.readlines()
reference_content = [int(data, 16) for data in reference_lines]

ok = True
expected = 0
received = 0

difference_address = (len(reference_content) - 1) << 2

for i in reference_content:
received = int.from_bytes(
self.processor_ci.read_memory(
breakpoint - difference_address, self.core_use_two_memories
)
)
expected = i

if expected != received:
ok = False
break

difference_address -= 4

return ok, expected, received

def _run_test_dir(self, config: dict) -> None:
path = config['path']
name = config['name']

print(f'Running tests: {name} in {path}')

if not os.path.exists(path):
raise FileNotFoundError(f'Test directory {path} does not exist.')

for category in TEST_CATEGORIES:
suite = TestSuite(f'{name}.{category}')

category_path = os.path.join(path, category)
category_breakpoint = config.get(f'{category}_breakpoint', 50)
memory_files_path = os.path.join(category_path, 'memory')
referece_files_path = os.path.join(category_path, 'reference')

self.processor_ci.set_execution_end_address(category_breakpoint)

print(
f'Running {category} tests in {category_path}, with breakpoint {category_breakpoint}'
)

memory_files = sorted(os.listdir(memory_files_path))
reference_files = sorted(os.listdir(referece_files_path))

if len(memory_files) != len(reference_files):
raise ValueError(
f'Memory files and reference files count mismatch in {category_path}.'
)

for memory_file, reference_file in zip(
memory_files, reference_files
):
memory_file_path = os.path.join(memory_files_path, memory_file)
reference_file_path = os.path.join(
referece_files_path, reference_file
)

print(f'Running test: {memory_file}')

result, expected, received = self._execute_test(
memory_file_path, reference_file_path, category_breakpoint
)

if result:
test_case = TestCase(
name=f'{memory_file}', classname=f'{name}.{category}'
)
suite.add_testcase(test_case)
else:
test_case = TestCase(
name=f'{memory_file}',
classname=f'{name}.{category}',
)
test_case.result = [Failure(
f'Expected: {expected}, Received: {received}'
)]
suite.add_testcase(test_case)

self.xml.add_testsuite(suite)

def run_tests(self):
for dir in self.directories:
self._run_test_dir(dir)

def run_extra_tests(self):
...

def generate_junit_report(self):
# Generate JUnit XML report
current_time = time.time()
self.xml.write(f'test_results_{current_time}.xml')
print(f'JUnit XML report generated: test_results_{current_time}.xml')
__all__ = ['ProcessorCITest']
Loading
Loading