Skip to content

Enable flash - #434

Open
jinwjinl wants to merge 11 commits into
vivoblueos:mainfrom
jinwjinl:enable-flash
Open

Enable flash#434
jinwjinl wants to merge 11 commits into
vivoblueos:mainfrom
jinwjinl:enable-flash

Conversation

@jinwjinl

@jinwjinl jinwjinl commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Enable to read and write nor-flash. It has been verified by connecting W25Q64 in ESP32C3. Note:
CS : GPIO5,
DI : GPIO9,
DO : GPIO10,
CLK : GPIO8.

@jinwjinl

jinwjinl commented Jul 2, 2026

Copy link
Copy Markdown
Contributor Author

build_prs #434 vivoblueos/external#27

@github-actions

github-actions Bot commented Jul 2, 2026

Copy link
Copy Markdown

@github-actions

github-actions Bot commented Jul 2, 2026

Copy link
Copy Markdown

❌ Job failed. Failed jobs: build_and_check_boards (failure), see https://github.com/vivoblueos/kernel/actions/runs/28585965485.

@jinwjinl

jinwjinl commented Jul 2, 2026

Copy link
Copy Markdown
Contributor Author

build_prs #434 vivoblueos/external#27

@github-actions

github-actions Bot commented Jul 2, 2026

Copy link
Copy Markdown

@github-actions

github-actions Bot commented Jul 2, 2026

Copy link
Copy Markdown

✅ All jobs completed successfully, see https://github.com/vivoblueos/kernel/actions/runs/28586350982.

@han-jiang277

Copy link
Copy Markdown
Contributor
  1. spi_flash.rs:162 — Silent partial read on erase-block boundary
    read_blocks returns Ok(()) after copying only the in-cache portion of a multi-sector buffer that crosses an erase-block boundary. Block::read passes the full multi-sector buffer in one call, so any read spanning two erase blocks while the first is dirty delivers truncated data with no error signal.
  2. esp32_gpio.rs:68 — GPIO 1 << PIN overflows for PIN ≥ 26
    GpioOut::DATA is 26 bits wide. For PIN in [26, 31] the bit is masked to 0 — the pin is never driven. For PIN ≥ 32Rust panics (debug) or wraps to pin PIN % 32 (release). No compile-time bound exists.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to confirm whether these commands are specific to the W25Q80 or are generic (common to other flash chips).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These command codes are JEDEC 25-series SPI NOR Flash standard commands, not specific to the W25Q80. The file header is marked with //! JEDEC 25-series SPI NOR Flash command layer., consistent with the Winbond W25Q80 datasheet, and also applicable to chips conforming to the same standard, such as W25Q64 / W25Q128.

Refer to the Linux kernel include/linux/mtd/spi-nor.h: https://raw.githubusercontent.com/torvalds/linux/master/include/linux/mtd/spi-nor.h

@jinwjinl

Copy link
Copy Markdown
Contributor Author

Description

This commit enables driving an SPI NOR flash on ESP32-C3 and registers it as a block device through the device-discovery mainline (define_bus! / register_device / probe_driver / InitDriver::init), the same framework the bme280 sensor already uses. The previous init_block_devices path that registered the block device directly is replaced.

Registration path

The flash block device is no longer registered directly by init_block_devices. Instead it goes through the device-discovery mainline, mirroring bme280 so that other SPI devices can plug in through the same framework:

  1. Bus declaration (boards/seeed_xiao_esp32c3/mod.rs): define_bus! declares spi2_bus (type BlockSpi<Spi2Impl, Esp32GpioOutputPin>) with a flash device carrying SpiFlashConfig::new(name). BlockSpi holds &'static references to the SPI peripheral and the CS pin, manages CS timing internally, and implements embedded_hal::spi::SpiDevice (via BusWrapper<BlockSpi>) as the transport for the flash command layer.
  2. Initialization (init_spi_bus, #[cfg(spi_core)]): configure pins → BlockSpi::newBus::new stored in SPI2_BUS (Once) → register_device enrolls the devices declared by define_bus! onto the bus → probe_driver(&SpiFlashDriverModule) lets the driver module self-match an enrolled device.
  3. Probe & register (drivers/flash/spi_flash.rs): SpiFlashDriverModule::probe matches DeviceData::Native + SpiFlashConfig; SpiFlashConfig::init reads the JEDEC ID, computes capacity, builds SpiFlashBlockDriver, and finally DeviceManager::register_device registers the block device into the device manager. InitDriver/DriverModule are generic over impl<T: PlatPeri + Spi<SpiConfig, ()>, G: PlatPeri + OutputPin>, decoupled from any specific board.
  4. Entry (boot.rs): #[cfg(spi_core)] boards::init_spi_bus() replaces the former #[cfg(enable_block)] init_block_devices().

spi_bus_adapter.rs is superseded by block_spi.rs: the old SpiBusAdapter did not implement BusInterface and so could not enter Bus::new, and the SpiBus (no CS) + ExclusiveDevice-wraps-CS ownership model conflicts with the shared BusWrapper used by the mainline. BlockSpi internalizes CS plus the shared lock to resolve this.

Review feedback

  • silent partial read on erase-block boundary​: read_blocks now loops while buf_off < buf.len(), chunking at erase-block granularity. On a cache hit it copies from erase_buf; on a miss it reads directly into buf via flash_cmd.read. It advances buf_off/cur_block until the whole buffer is filled before returning Ok(()), so a cross-boundary read no longer truncates silently.
  • GPIO ​1 << PIN ​overflow​: Esp32GpioOutputPin::new adds a runtime assert!(pin < 26) matching GpioOut::DATA NUMBITS(26), rejecting PIN ≥ 26 (and eliminating the PIN ≥ 32 UB).

Configuration

  • enable_block is retained (FATFS depends on it).
  • New SPI_CORE (default n) gates init_spi_bus and its boot call.
  • New USE_EMBEDDED_HAL_V1 gates the SpiDevice impl for BusWrapper<BlockSpi> and the flash InitDriver/DriverModule impls.
  • esp32c3 defconfig sets CONFIG_SPI_CORE=y.

Verification

  • esp32c3 release builds with 0 errors / 0 warnings.
  • On hardware: boots cleanly, log shows SPI flash JEDEC ID / capacity, shell is stable, and the data mount point reads/writes correctly.

Note

The stack size on the ESP32C3 is currently insufficient; executing init_vfs will trigger a panic.

@jinwjinl

Copy link
Copy Markdown
Contributor Author

build_prs

@github-actions

Copy link
Copy Markdown

@github-actions

Copy link
Copy Markdown

❌ Job failed. Failed jobs: check_format (failure), build_and_check_boards (failure), see https://github.com/vivoblueos/kernel/actions/runs/29995152327.

@jinwjinl

Copy link
Copy Markdown
Contributor Author

build_prs

@github-actions

Copy link
Copy Markdown

@github-actions

Copy link
Copy Markdown

❌ Job failed. Failed jobs: build_and_check_boards (failure), see https://github.com/vivoblueos/kernel/actions/runs/29996977500.

@jinwjinl

Copy link
Copy Markdown
Contributor Author

build_prs

@github-actions

Copy link
Copy Markdown

@github-actions

Copy link
Copy Markdown

❌ Job failed. Failed jobs: build_and_check_boards (failure), see https://github.com/vivoblueos/kernel/actions/runs/30012795185.

esp32 qemu has no external W25Q64 on GPSPI2
@jinwjinl

Copy link
Copy Markdown
Contributor Author

build_prs

@github-actions

Copy link
Copy Markdown

@github-actions

Copy link
Copy Markdown

❌ Job failed. Failed jobs: build_and_check_boards (failure), see https://github.com/vivoblueos/kernel/actions/runs/30062768436.

@jinwjinl

Copy link
Copy Markdown
Contributor Author

build_prs

@github-actions

Copy link
Copy Markdown

@github-actions

Copy link
Copy Markdown

@github-actions

Copy link
Copy Markdown

✅ All jobs completed successfully, see https://github.com/vivoblueos/kernel/actions/runs/30434038782.

@jinwjinl

Copy link
Copy Markdown
Contributor Author

build_prs

@github-actions

Copy link
Copy Markdown

@github-actions

Copy link
Copy Markdown

❌ Job failed. Failed jobs: build_and_check_boards (failure), see https://github.com/vivoblueos/kernel/actions/runs/30521709572.

@jinwjinl

Copy link
Copy Markdown
Contributor Author

build_prs

@github-actions

Copy link
Copy Markdown

@github-actions

Copy link
Copy Markdown

✅ All jobs completed successfully, see https://github.com/vivoblueos/kernel/actions/runs/30522989755.

Comment thread kernel/src/drivers/flash/spi_flash.rs Outdated
const FLASH_SECTOR_SIZE: u16 = 512;
const FLASH_ERASE_SIZE: usize = 4096;
const PAGES_PER_ERASE_BLOCK: usize = FLASH_ERASE_SIZE / 256;
const MAX_24BIT_CAPACITY: u64 = 0x0100_0000;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These parameters can be defined in config file with different flash types.

.transaction(&mut [Operation::Write(&[0x9F]), Operation::Read(&mut id_buf)])
.map_err(spi_err_to_flash)?;
let jedec_id = (id_buf[0] as u32) << 16 | (id_buf[1] as u32) << 8 | (id_buf[2] as u32);
if jedec_id == 0 || jedec_id == 0x00FF_FFFF {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is not recommended to use too many magic-words. use "#defined PARAMETER_MEANING 0x0000xxxx"

@jinwjinl

jinwjinl commented Aug 7, 2026

Copy link
Copy Markdown
Contributor Author

Description

This change keeps SPI NOR flash, block-device support, and FATFS enabled by default on ESP32-C3 while allowing ESP32 QEMU to boot without an external SPI flash device, and hardens the SPI bus so that multiple devices sharing one peripheral cannot interleave commands on ESP32-C3 where SpinLock degrades to NoLock. The flash driver erase cache is expanded from a single slot to a two-slot LRU, and the flash geometry is made Kconfig-driven.

Bus and SPI transaction locking

The bus model is borrowed from PR #442 — one shared SpiBus plus a per-device CS pin, with SpiDevice::transaction() driving CS for the duration of one transaction. It is not copied verbatim, for three reasons.

  1. Bus model mismatch. PR Features: add more functionality to esp32c3 #442 is roughly SpiBus + ExclusiveDevice<SpiBus, CS>; the mainline device-discovery framework expects drivers to obtain the shared bus through BusWrapper (path define_bus! → Bus<BlockSpi> → BusWrapper → register_device → probe_driver). ExclusiveDevice would put CS ownership on an outer wrapper, bypassing BusWrapper: the driver would receive the wrong object, calls would bypass the shared-bus lock, the CS lifetime would be incomplete, and multiple devices could not share one bus lock. So CS and the transaction lock are wired into the existing BusWrapper instead.

  2. Error handling too loose. PR Features: add more functionality to esp32c3 #442 swallows SPI/CS errors: an SPI failure still returns success, a CS low/high failure is not propagated, and a flush failure loses the original error. This is dangerous for Flash — a JEDEC read error, CRC failure, or incomplete erase propagates into FS/XIP corruption. The current code preserves error priority operation > flush > CS-high: even on operation failure it still runs flush, then CS high, then releases the transaction Mutex via the guard's Drop.

  3. Widening SpinLock scope does not fix ESP32-C3. SpinLock degenerates to NoLock on non-SMP ESP32-C3 (its lock() is an unprotected guard). DelayNs can suspend the thread, so a sequence CS low → SPI op → DelayNs (thread scheduled) → another task accesses SPI → CS changed → original continues → CS high interleaves commands and corrupts Flash/CRC. BusWrapper therefore becomes BusWrapper<B>(TinyArc<SpinLock<B>>, TinyArc<Mutex>): the SpinLock<B> guards a single hardware-register call, and the Mutex guards the full CS-low → CS-high transaction. All BusWrapper clones share one Mutex, so different devices compete for the same bus.

  • lock_transaction() returns a !Send BusTransactionGuard whose Drop posts the Mutex, so the lock is released even when an operation errors.
  • The low-level SpinLock never spans DelayNs, avoiding a spinlock that may sleep on SMP.
  • The 1 ms Operation::DelayNs(1_000_000) that was inside the 0x0B fast-read transaction is replaced by a one-byte Operation::Read(dummy), matching the 0x0B protocol (command + 3-byte address + 1 dummy byte + data) and removing the in-transaction blocking delay that could schedule the thread and let another device grab the bus. release_from_deep_power_down now takes &mut impl DelayNs and calls delay_ns(3_000) after the SPI transaction returns (CS already high), so the recovery delay no longer holds the transaction Mutex.

Performance: uncontended, each transaction adds one Mutex pend/post — a fast task-lock path dwarfed by Flash command and erase timing; contended, a second device waits for the first transaction to finish, which is required because one bus cannot carry two devices' commands simultaneously. The SpinLock is not re-acquired per operation inside a transaction, so sequential Flash reads are not slowed by per-byte locking. Each Bus::new() creates one extra Mutex; I2C's BusWrapper carries an unused one (minor memory, no behavior change); virtio does not use BusWrapper and is unaffected.

Erase cache: single slot → two-slot LRU

The flash driver previously cached one erase block at a time: erase_buf: Vec<u8>, dirty: bool, current_erase_block: Option<usize>. Any access to a different erase block forced a flush + reload, so alternating writes to two blocks (e.g. metadata ping-pong) erased and re-read on every switch. The cache is replaced by an array of two slots:

struct EraseCacheSlot {
    erase_block_id: Option<usize>,
    data: Vec<u8>,
    dirty: bool,
    last_used: u64,
}

cache: [EraseCacheSlot; ERASE_CACHE_SLOTS],   // ERASE_CACHE_SLOTS = 2
use_counter: u64,
  • Lookup cached_slot(id) returns the slot whose erase_block_id matches; a hit calls touch_slot (bump use_counter, stamp last_used) and returns without I/O.
  • Eviction is LRU: on a miss, an empty slot is preferred; otherwise the slot with the smallest last_used is evicted. Eviction flushes the victim (flush_slot) then loads the new block (load_slot).
  • flush_slot dispatches by FLASH_ERASE_SIZE: sector_erase (4 KiB) / block_erase_32k (32 KiB) / block_erase_64k (64 KiB), then programs the dirty pages in FLASH_PAGE_SIZE chunks. A slot is marked dirty only when the written bytes actually differ from the cached bytes, so no-op writes skip the erase/program cycle.
  • flush iterates all ERASE_CACHE_SLOTS and flushes each dirty slot, so a shutdown/FATFS sync writes back both.

Two slots let a workload that alternates between two erase blocks (FAT metadata, directory + file data) keep both resident instead of thrashing; use_counter is u64 and bumped via saturating_add, so it cannot overflow.

Flash geometry from Kconfig

Flash geometry is no longer hard-coded. FLASH_SECTOR_SIZE, FLASH_PAGE_SIZE, FLASH_ERASE_SIZE, and MAX_24BIT_CAPACITY are read from blueos_kconfig::CONFIG_*. A const _: () = { assert!(...) } block verifies at compile time that erase % page == 0, erase % sector == 0, the erase size is one of {4 KiB, 32 KiB, 64 KiB}, and max >= erase.

capacity_from_jedec_id now rejects a capacity smaller than one erase block, not a multiple of the erase block, or larger than the configured 24-bit range before block-device registration; the overflow case is InvalidParam (named MAX_3BYTE_ADDRESS_EXCLUSIVE) instead of AddrOverflow.

BlockSpi full-duplex read

BlockSpi::read transfers in TRANSFER_CHUNK_SIZE = 64 byte chunks instead of write-then-read, and a new #[cfg(use_embedded_hal_v1)] impl SpiBus<u8> for BlockSpi<T> bridges the inherent methods to the embedded-hal trait. read_region/write_region return ENOSYS instead of todo!().

Robustness changes

  • Invalid JEDEC response: jedec_id rejects 0x000000 and 0xFFFFFF (named INVALID_JEDEC_ID_ZERO / INVALID_JEDEC_ID_ALL_ONES), mapped to FlashError::NotReady rather than being interpreted as a one-byte flash.
  • Bounded SPI polling: the CMD::UPDATE and CMD::USR hardware-owned bits use a bounded polling helper (SPI_CMD_TIMEOUT = 10_000) with core::hint::spin_loop(); failure to clear within the limit returns HalError::Timeout instead of spinning forever.
  • GPIO pin validation: Esp32GpioOutputPin::new returns Result<Self, HalError> and rejects pins >= 26 instead of assert!-panicking; a const unsafe fn new_unchecked is provided for const contexts and used for the GPIO3 flash CS with a SAFETY comment.
  • Block::new overflow check: Block::new returns Result<Self, ErrorKind> and uses checked_mul for capacity * SECTOR_SIZE, rejecting silent overflow. init_virtio_block and both flash registration points propagate errors through )?. The hand-written unsafe impl Sync was removed. should_skip_fatfs_mount is gated behind #[cfg(fatfs)] so configs without FATFS still compile.

Initialization path (unchanged from enable-flash)

init_flash_spi_bus() configures SPI2, creates BlockSpi<Spi2Impl>, wraps it in Bus, and stores it in FLASH_SPI_BUS (spin::Once); repeated calls return the existing bus. init_block_devices enrolls define_bus! devices via register_device, then probe_driver(&SpiFlashDriverModule)SpiFlashConfig::init registers flash-storage. init_block_devices() runs before init_vfs(); Optional + ENODEV continues, required/other is fatal.

VFS behavior (unchanged from enable-flash)

BlockStoragePolicy::Optional (ESP32) vs Required (VirtIO). FATFS is skipped only when Optional + ENODEV; corruption, I/O, timeout, and required-device failures are still propagated. ESP32 QEMU (no emulated W25Q64) boots with tmpfs; VirtIO stays strict.

Configuration

kernel/src/devices/Kconfig adds, under ENABLE_BLOCK:

CONFIG_SPI_FLASH_SECTOR_SIZE   (int, default 512,       range 1..65535)
CONFIG_SPI_FLASH_PAGE_SIZE     (int, default 256,       range 1..65536)
CONFIG_SPI_FLASH_ERASE_SIZE    (choice: 4K | 32K | 64K, default 4K)
CONFIG_SPI_FLASH_MAX_CAPACITY  (int, default 16777216,  range 1..16777216)

The ESP32-C3 debug and release defconfigs keep CONFIG_ENABLE_BLOCK=y, CONFIG_SPI_CORE=y, CONFIG_FATFS=y.

Verification

New unit tests added in this change:

  • test_transfer_in_place_is_full_duplexBlockSpi transfer writes all bytes and reads back +1.
  • test_device_holds_bus_lock_for_entire_transaction — CS transitions observe the transaction Mutex held.
  • test_block_new_rejects_total_size_overflowBlock::new rejects capacity * SECTOR_SIZE overflow.
  • test_new_rejects_out_of_range_pinEsp32GpioOutputPin::new(26) returns InvalidParam.
  • test_metadata_data_ping_pong_uses_two_cache_slots — ping-ponging two erase blocks reuses both slots without re-reading.

The full kernel test suite (ninja -C out/seeed_xiao_esp32c3.release check_all), rustfmt --check, and git diff --check for these changes have not been re-run this session; fill in the pass/fail numbers from your run, or I can run them.

Note

The optional-device path currently does not print an early warning when the external flash is absent. Before the scheduler starts, both normal logging and kearly_println! use the ESP32 USB Serial polling path, which can block indefinitely under QEMU; a warning should be emitted later from a scheduled context if required.

LCD PR 439 also uses SPI2 and assigns GPIO5 as LCD DC, while the external flash uses GPIO5 as Flash CS; merging both features requires explicit SPI bus ownership and GPIO assignment handling, not just resolving the Git function-name conflict.

The extra Mutex on BusWrapper is generic, so I2C buses carry one unused Mutex (minor memory, no behavior change); virtio does not use BusWrapper and is unaffected.

@jinwjinl

jinwjinl commented Aug 7, 2026

Copy link
Copy Markdown
Contributor Author

build_prs

@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown

@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown

❌ Job failed. Failed jobs: build_and_check_boards (failure), see https://github.com/vivoblueos/kernel/actions/runs/31146224844.

@jinwjinl

jinwjinl commented Aug 7, 2026

Copy link
Copy Markdown
Contributor Author

build_prs

@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown

@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown

❌ Job failed. Failed jobs: build_and_check_boards (failure), see https://github.com/vivoblueos/kernel/actions/runs/31156036205.

@jinwjinl

jinwjinl commented Aug 7, 2026

Copy link
Copy Markdown
Contributor Author

build_prs

@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown

@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown

❌ Job failed. Failed jobs: build_and_check_boards (failure), see https://github.com/vivoblueos/kernel/actions/runs/31156811434.

@jinwjinl

jinwjinl commented Aug 7, 2026

Copy link
Copy Markdown
Contributor Author

build_prs

@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown

@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown

✅ All jobs completed successfully, see https://github.com/vivoblueos/kernel/actions/runs/31158667377.

@jinwjinl

jinwjinl commented Aug 7, 2026

Copy link
Copy Markdown
Contributor Author

build_prs

@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown

@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown

❌ Job failed. Failed jobs: build_and_check_boards (failure), see https://github.com/vivoblueos/kernel/actions/runs/31168310846.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants