sd-read/sd-write throughput: one SD command per 512 bytes, and a link that idles during writes

[joeferner]

sd-read and sd-write are slow enough that the README documents it as
a limitation. Two independent causes, and neither is a missing driver
feature in rpi-hal.

1. Every transfer is one block per SD command

rpi-hal already does multi-block transfers (CMD18/CMD25 with an
auto-CMD12 stop), and its embedded-sdmmc adapter dispatches to them
whenever it is handed more than one block:

// rpi-hal, impl embedded_sdmmc::BlockDevice for SdCard
let count = checked_block_count(blocks.len())?;
self.sd.read_blocks_pio(start_block_idx.0, count, ..)

It never is. embedded_sdmmc::BlockCache holds a single block:

// embedded-sdmmc 0.9.0, src/blockdevice.rs
pub struct BlockCache<D> {
    block: [Block; 1],
    ...
}

so every read and write through the filesystem costs a full command
round trip per 512 bytes, and the multi-block path is dead code from the
loader's point of view.

Options, roughly in increasing order of work:

• Bypass the FAT layer for bulk file data: walk the cluster chain once,
  then issue multi-block reads/writes against rpi-hal's Sd directly
  for the contiguous runs, keeping embedded-sdmmc for directory
  traversal and metadata. Most of the win, all of the complexity.
• Take it upstream — a multi-block-aware cache in embedded-sdmmc
  benefits every consumer, not just this one.

Note that the DMA path (read_blocks_dma) stays unavailable either way
while blocks arrive as separate embedded_sdmmc::Blocks, since it wants
one contiguous buffer.

2. The link idles during every SD write

The device defers each chunk's OK until it has finished writing that
chunk. That is deliberate and must stay in some form: there is no
hardware flow control, the host sends the next ~4KB the instant it sees
OK, and the PL011's 16-byte RX FIFO overflows in ~85µs at 1.5Mbaud.
Acknowledging before a slow file.write dropped bytes and desynced the
framing — multi-chunk sd-write failed deterministically at the second
chunk while single-chunk files appeared to work.

The fix is to overlap rather than remove it: double-buffer, so the
device receives chunk N while writing chunk N-1, and the OK for N-1
goes out as soon as there is somewhere to put N. Two 4KB buffers is the
whole cost.

Worth measuring first

No throughput numbers have ever been taken, so the split between these
two causes is unknown, as is whether either matters next to the card's
own write latency. A measurement — bytes/second for a large sd-write
at a fixed baud — should come before either change, and is the only way
to tell whether the second one is worth the buffer.

[joeferner]

resident-fat answers cause 1, and changes what cause 2 is worth

Since this was filed, the first option above — "walk the cluster chain
once, then issue multi-block reads/writes against rpi-hal's Sd
directly for the contiguous runs" — has been built and published as
resident-fat. It was written
for a different consumer (an OTA path that had the same problem), so for
this loader it is most of the win with none of the complexity the issue
anticipated.

The protocol already fits it. cmd_sd_write reads total_size off
the wire before the data, so the streaming shape maps onto
create_with_size(path, total_size) followed by write_at(offset, chunk)
per chunk. That gets both halves of the win rather than one: the cluster
chain is allocated contiguously in a single go because the length is known
up front, and each 4 KB chunk then goes out as one 8-block command instead
of eight single-block ones.

Cause 2 is untouched by that. The device still defers each chunk's
OK until it has finished writing that chunk, and double-buffering is
orthogonal to which filesystem is underneath.

The measurement this issue asked for, indirectly

"No throughput numbers have ever been taken" is still true of this
loader. But the same card and board have since been measured through the
OTA path, which is worth recording here because it settles the split
between the two causes:

• a single-block write costs ~17.5 ms
• a 128-block write costs ~26 ms — one and a half times as much for a
  hundred and twenty-eight times the data

The card charges per command, not per block. Applied to a 4 KB chunk
(8 blocks), and taking the wire at 1.5 Mbaud to be ~27 ms for 4 KB:

per 4 KB chunk	write	wire	serialized
today	~140 ms	~27 ms	~167 ms
after resident-fat	under 20 ms	~27 ms	~46 ms
after double-buffering too	—	~27 ms	~27 ms

These are estimates, not a measurement of this loader — they are one
consumer's per-command figures applied to another's chunk size. What they
are good enough to say is that the ratio inverts: writes currently
dominate the wire about 5:1, and after cause 1 is fixed the wire dominates
instead. So cause 2 stops being "unknown, possibly irrelevant" and becomes
the remaining term, worth roughly a further 40%. Still worth measuring
before building it, but no longer worth guessing about.

The cost to decide on deliberately

resident-fat keeps the allocation table and every directory it has read
in RAM — that is where the win comes from — so it needs a global
allocator. This loader has none: no global_allocator and no
extern crate alloc anywhere in firmware/src/. Adopting it means giving
a deliberately minimal, self-relocating loader a heap and a resident table
that is tens of megabytes on a large card.

That is affordable, since the loader has the whole machine to itself
before anything else runs. But it is an architectural change rather than a
dependency swap, and it should be a decision rather than something that
arrives as a side effect of a throughput fix.

Suggested shape

Keeping this open. Cause 1 has a ready answer whose cost is now known;
cause 2 survives it and has better information behind it than when this
was written. Splitting into one issue per cause would let the
resident-fat switch close its own.