Skip to content

Two kernel slots, so a bad update is a one-line rollback instead of a card pull #9

Description

@joeferner

An update overwrites the live boot image, so the moment it commits there
is no previous kernel left. That is the wrong shape for the failure that
actually happens.

The failure this is for

The likely failure is not a card dying mid-write — the write is verified
by a read-back, which catches that. It is a kernel that installs
perfectly, boots, and then misbehaves: a driver that hangs, a filesystem
change that cannot mount the card, a regression that only shows up
against real hardware. A verify says nothing about any of those, and the
image that worked yesterday has already been overwritten by the one that
does not work today.

Recovering from that today means pulling the card and writing a
known-good image onto it from a build tree. That has happened at least
once here, from a filesystem change that could not mount the volume it
had just been ported to.

The scheme

Write the kernel to whichever slot is not running, then move the
kernel= line in config.txt:

  • Slot A is kernel7.img / kernel8.img — deliberately the name the
    firmware loads when config.txt names none. A lost or corrupt
    config.txt therefore falls back to an image that is bootable by
    construction rather than to nothing.
  • Slot B is kernel7b.img / kernel8b.img.
  • Which is live is read from config.txt at apply time, not tracked
    in state anywhere: no kernel= line means slot A. That stays correct
    after a card is swapped, imaged, or edited by hand.
  • Order: write the inactive slot, verify it, then rewrite
    config.txt. That write is the commit — it is one cluster, and it is
    the only irreversible step.

The previous kernel survives without a copy being made, which is what
makes this cheaper than the backup file it replaces rather than more
expensive.

Recovery is a jumper, not a console

The obvious way to roll back is rpi-loader sd-read config.txt, edit one
line, sd-write. That is fine at the bench and useless in the field: it
needs a UART attached to a board that is not booting, which is harder
than swapping the card, not easier. A recovery path that costs more than
the thing it replaces will not get used.

The firmware can make the choice instead. config.txt has a GPIO
conditional filter, evaluated by start.elf while parsing the file —
before any kernel is loaded — so it can steer kernel= from the state of
a pin:

[gpio4=1]
# settings here apply if GPIO 4 is high

[gpio2=0]
# settings here apply if GPIO 2 is low

That makes recovery: power off, fit a jumper, power on. No console, no
card reader, no build tree.

The jumper is a second selector layered on the first — kernel= stays
the remote commit, because OTA has to work with nobody at the board. So
the commit writes both the live slot and the fallback:

# installer: the slot just installed
kernel=kernel8b.img

# installer: the slot that was live before it
[gpio5=0]
kernel=kernel8.img
[all]

The jumper therefore does not mean "slot A", it means boot the kernel
that worked yesterday
. That keeps the alternating slots above intact,
instead of freezing slot A as a golden image that is never updated and
rots. The missing-config.txt fallback is unchanged: unfiltered slot A
is the name the firmware loads on its own, so a lost config also lands on
the previous kernel.

Choosing the pin

Rely on the power-on pull, not a gpio= directive in the same file —
that directive's ordering against filter evaluation is not something to
bet a recovery path on. GPIO 0–8 default to pull-up and 9–27 to
pull-down, so use a pull-up pin jumpered to ground: open is normal boot,
shunt is recovery, and a lost or broken jumper fails toward normal.

GPIO5 is header pin 29 and pin 30 is ground, physically adjacent — a
single 2-pin shunt, the same part as a mode jumper on any dev board.
GPIO6 (pin 31) is the same story if 5 is taken.

Two things to get right:

  • The application must never drive that pin low as an output. GPIO
    state survives a watchdog reset, so an app that muxes the pin and then
    reboots itself would silently come back in recovery. The pin is
    reserved.
  • Duplicate kernel= resolution order. The block above assumes the
    last assignment wins. That is the usual config.txt behaviour and it
    is the one load-bearing assumption here, so it gets confirmed on a real
    board with a two-line config before the installer is written around it.

The running kernel can read the same pin, which is worth doing once the
rest works: a board that knows it is in recovery can say so — over the
network, or on the ACT LED — and can refuse to auto-apply the update that
just failed.

[tryboot], [partition=N] and bootvar0 would give a nicer version of
this, with automatic revert on the next power cycle, but they are
features of the Pi 4/5 EEPROM bootloader. The GPIO filter is the portable
one and works on the Pi 2 and 3 as well.

config.txt has to be shared, not owned

A bundle can carry config.txt — firmware options get added over time,
and a board that cannot be told disable_splash=1 without a card reader
is the situation this whole update path exists to remove. But the commit
above writes that same file.

The installer owns the slot lines and the project owns the rest. The
commit rewrites or inserts the unfiltered kernel= and the [gpio5=0]
block, into whichever config.txt is on the card after the bundle's
own copy has landed. A project's config.txt in its repository should
carry neither; if it does, the commit overwrites them, because the card
is the only thing that knows which slot is live.

That is a real parsing constraint rather than a line match: the installer
reads and rewrites the kernel= in the unfiltered region, and must not
mistake the one inside the filter block for it. Bounded, but it has to be
written down, because getting it backwards points the recovery jumper at
the image that does not boot.

One gap that needs closing with it: unlike the kernel, config.txt has
no second slot, so a bundle shipping a broken one has overwritten the
only copy. Back it up to config.bak before writing. It is about a
kilobyte — the reason a kernel backup was not worth making does not apply
at this size — and it restores the one-line-edit recovery for the file
most likely to be edited casually.

The Raspberry Pi firmware can be slotted too

config.txt supports start_file= and fixup_file=, which is the same
mechanism as kernel= for the two files that most need it: a start.elf
and a fixup.dat from different releases do not boot, and they are
replaced as a pair. Writing them as start2.elf/fixup2.dat and
pointing those two lines at them gives the previous pair the same free
survival, with the same fallback when config.txt goes missing.

If that is done, the recovery block gets the previous pair too —
otherwise the jumper goes back to yesterday's kernel under today's
firmware, which is only half a rollback.

Worth doing only after the kernel slot works, since it is the same code
with different names.

bootcode.bin has no equivalent and never will — the ROM loads it by
name, so there is no slot and no line to redirect. It is ~50 KB, it
changes almost never, and on the Pi 4 it is in SPI EEPROM and not on the
card at all, which is what makes that acceptable rather than merely
unavoidable.

Where it goes

rpi-loader-ota, behind the apply feature, beside the write/verify
path. It is not application code: every consumer wants it, none would
build it alone, and getting the ordering wrong is what leaves a board
unbootable. That half of the crate does not exist yet, so this is blocked
on it.

The pin number cannot be hard-coded there — a board that already uses
GPIO5 needs a different one — so it is a parameter of the apply call,
alongside an option to write no filter block at all for a board with no
free pin.

Not in scope

Automatic rollback. Choosing a slot by counting failed boots needs
state that survives a hang and something that runs before the kernel to
act on it, and there is none. The selection stays deliberate: a remote
kernel= edit when the board still talks, a jumper when it does not.

References

  • config.txt
    — the rendered documentation for everything below
  • Conditional filters
    — the GPIO filter, and the [tryboot]/[partition]/bootvar0 filters
    that are Pi 4/5 only
  • Boot options
    kernel, start_file/fixup_file ("a matched pair – using
    unmatched files will stop the board from booting"), os_prefix
  • GPIO on Raspberry Pi
    — 40-pin header numbering: GPIO5 on pin 29, ground on pin 30
  • BCM2835 ARM Peripherals
    — the alternate-function table carries the power-on pull state per pin
    (GPIO 0–8 up, 9–27 down)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions