`rt_sigreturn` overwrites the restored X8, so a signal taken on an `svc` re-enters it as syscall 2

[jotpalch]

Symptom

tests/test-shim-futex-toctou fails intermittently on main. The host prints:

WARN  src/syscall/syscall.c:2841: unimplemented syscall 2 (x0=0x200000000, x1=0x80, x2=0x0, x3=0x0, x4=0x0, x5=0x0)
FAIL: unexpected spin rc -38 (round 24992)
FAIL: 1 unexpected returns in the spin phase

x1=0x80 is FUTEX_WAIT|FUTEX_PRIVATE_FLAG and x0 is the futex word, so the guest
issued futex and the host dispatched syscall 2. The guest gets -ENOSYS out of a
call it made as futex. Nothing here is specific to that test's mprotect race; the
reproduction below has no mprotect in it.

Mechanism

Measured by running the reproduction below under a print-only probe copy of main,
which records per thread the last signal delivery, the last rt_sigreturn, and the
state of the restart record. It caught ten bad-X8 events across nine failing runs; nine
of the ten are identical, e.g.:

INSTR bad x8=2 elr=0x400948 restarted=0 | mark site=2 seq=792 armed=0 elr=0x400944 rewound=0x0 nr=0 | cancel out=0 seq=789 elr=0x0 rewound=0x0 | arm seq=0 elr=0x0 nr=0 | seq=792
INSTR deliv el0=1 pc=0x400944 x8=98 seq=790 | sigret pc=0x400944 x8=98 seq=791
WARN  src/syscall/syscall.c:2931: unimplemented syscall 2 (x0=0x4a1aa8, x1=0x80, x2=0x0, x3=0x0, x4=0x0, x5=0x0)
FAIL: futex returned -38 at round 404994 (1 total)

deliv is the delivery that took the EL0-preemption branch, sigret the rt_sigreturn
that returned through its frame, bad the SVC that reached dispatch as syscall 2;
arm seq=0 and cancel out=0 say syscall_restart_arm never ran on that thread and
the cancel found nothing armed. The tenth is a different shape -- delivery on the EL1
branch (el0=0), the frame already carrying X8 = 2, the saved PC still on the svc.
Same symptom, nothing armed there either, and I did not chase how that PC came to be
rewound. The probe only prints, which shifts the line number in its own WARN; on main
that is src/syscall/syscall.c:2841, and an unmodified build fails at the rates in the
table below.

0x400944 is the svc #0 of raw_futex_wait, inlined into waiter, and 0x400948 the
instruction after it (aarch64-linux-gnu-objdump -d build/repro):

  400940:	d2800c48 	mov	x8, #0x62                  	// #98
  400944:	d4000001 	svc	#0x0
  400948:	110004e7 	add	w7, w7, #0x1

1. SIGUSR1 is delivered while the vCPU is at EL0 with PC on the svc #0 of
   raw_futex_wait: X8 is already 98 (__NR_futex), the instruction has not executed.
   Delivery takes the EL0-preemption branch (src/syscall/signal.c:2187), saves
   pc = 0x400944, and skips the frame-drop marker (src/syscall/signal.c:2423-2424),
   correctly, because there is no shim frame to drop.
2. The handler returns. signal_rt_sigreturn restores all 31 GPRs from the frame, X8 = 98
   among them (src/syscall/signal.c:2629), then writes X8 = 2 unconditionally
   (src/syscall/signal.c:2705) to tell the shim to drop the frame belonging to the
   rt_sigreturn call itself.
3. The shim takes that marker to exec_drop_frame (src/core/shim.S:1846-1854), which
   pops its 256-byte frame and erets without restoring any GPR, so the X8 = 2 written
   in step 2 is what reaches EL0.
4. The ERET resumes at 0x400944, the svc that never ran. It executes with X8 = 2 and
   X0/X1 still holding the futex arguments, dispatch finds no syscall 2, and answers
   -LINUX_ENOSYS (src/syscall/syscall.c:2834-2842).

The marker travels in X8, which is also the register carrying the restored syscall
number. That is harmless for a frame whose PC is past an svc, and destroys the call
for a frame whose PC is on one. The delivery side already separates those two cases;
signal_rt_sigreturn does not. A fix still has to get the marker to the shim for the
rt_sigreturn frame itself.

This is not the restart hazard src/syscall/proc.h:369-373 names, although the symptom
is the one that comment predicts -- a rewound PC meeting a live marker. In every failure
syscall_restart_arm had never run on that thread (arm seq=0 above), so nothing was
armed and syscall_restart_cancel is not involved.

Rates

8-core Apple M1, macOS 15.6.1. The rows are disjoint sets of runs; every failure in all
four is the same -38 with the same host WARN.

program	tree	host load	failures / runs
tests/test-shim-futex-toctou	main at 5b7741e	idle	6 / 370
tests/test-shim-futex-toctou	main at 5b7741e	8 concurrent compile jobs	138 / 550
the reproduction below	main at 5b7741e	idle	2 / 400
the reproduction below	main at 5b7741e	8 concurrent compile jobs	31 / 400

Load is 8 looping aarch64-linux-gnu-gcc -D_GNU_SOURCE -static -O2 compiles of one test
file, i.e. a busy make -j8 beside the run. 5b7741e is a clean checkout with nothing
applied on top. It still reproduces on current main at 4023647: 10/400 idle and
189/400 under that load, from a fresh worktree built the same way.

Two controls, 400 runs each under that load. With sa_flags = 0 in place of
SA_RESTART: 23/400, against 23/400 for the SA_RESTART build in the same pair, so
SA_RESTART is not part of it. With the pthread_kill removed: 0/400.

Reproduction

/* repro-futex-restart.c -- a raw FUTEX_WAIT loop with a signal landing on it.
 * Expected returns are 0, -EAGAIN and -EINTR; anything else is the bug.
 */
#include <errno.h>
#include <linux/futex.h>
#include <pthread.h>
#include <signal.h>
#include <stdatomic.h>
#include <stdio.h>
#include <string.h>
#include "raw-syscall.h"

#define ROUNDS 3000
#define RESTING 0
#define MOVED 0x55667788

static atomic_int stop;
static int word;
static long woken, eagain, eintr, other, first_other;
static int first_round;

static void handler(int sig)
{
    (void) sig;
}

static void *waiter(void *arg)
{
    (void) arg;
    int round = 0;
    while (!atomic_load_explicit(&stop, memory_order_acquire)) {
        long rc = raw_futex_wait(&word, RESTING);
        round++;
        if (rc == 0)
            woken++;
        else if (rc == -EAGAIN)
            eagain++;
        else if (rc == -EINTR)
            eintr++;
        else {
            if (!other) {
                first_other = rc;
                first_round = round;
            }
            other++;
        }
    }
    return NULL;
}

static void delay(void)
{
    for (volatile int i = 0; i < 400; i++)
        ;
}

int main(void)
{
    struct sigaction sa;
    pthread_t t;

    memset(&sa, 0, sizeof sa);
    sa.sa_handler = handler;
    sa.sa_flags = SA_RESTART; /* the failure does not need it */
    sigaction(SIGUSR1, &sa, NULL);

    word = RESTING;
    pthread_create(&t, NULL, waiter, NULL);
    for (int r = 0; r < ROUNDS; r++) {
        word = RESTING;
        delay();
        if (r % 3 == 0)
            pthread_kill(t, SIGUSR1);
        delay();
        word = MOVED;
        raw_futex_wake(&word, 0x7fffffff);
    }
    atomic_store_explicit(&stop, 1, memory_order_release);
    word = MOVED;
    raw_futex_wake(&word, 0x7fffffff);
    pthread_join(t, NULL);

    if (other) {
        fprintf(stderr, "FAIL: futex returned %ld at round %d (%ld total)\n",
                first_other, first_round, other);
        return 1;
    }
    printf("OK woken=%ld eagain=%ld eintr=%ld\n", woken, eagain, eintr);
    return 0;
}

Save as tests/repro-futex-restart.c, then:

make elfuse
aarch64-linux-gnu-gcc -D_GNU_SOURCE -static -O2 -Itests \
    -o build/repro tests/repro-futex-restart.c -lpthread
while ./build/elfuse build/repro; do :; done

On 4023647 that loop stopped on iteration 1, 1 and 5 in three trials under the compile
load, each within 0.6 s. Idle it took 7, 229 and 9 iterations in three trials; one run
is ~50 ms.

What this does not show

• Which other guest paths are exposed. Any syscall whose svc is preempted in that
  one-instruction window loses X8; futex is only what issues them fast enough to be
  caught often. Not measured.
• The HVC Honor futex_waitv clockid and wake without polling #9 fault fallback writes the same marker (src/syscall/signal.c:2414-2421).
  Not tested.
• Any connection to A guest signal does not interrupt futex, epoll_wait or a finite select #378. That one is a signal waiting out a polling quantum, this one a
  signal delivered promptly that loses a register on the way back, and I did not look
  for a shared cause.

Note on the test's own output

Some failures print only FAIL: 1 unexpected returns in the spin phase and lose the
preceding FAIL: unexpected spin rc -38 (round N). That is a lost line, not a second
failure mode. A copy of the test that keeps its printing path and also records the first
unexpected rc for main to print after the join gave 88 failures over 400 runs under
load; in 29 of them the fprintf at tests/test-shim-futex-toctou.c:106 returned -1
with errno == EINTR and ferror set.
It is issued from the waiter thread while the main thread is sending SIGUSR1, and glibc
drops the buffer instead of retrying; a raw write(2) immediately after it succeeded in
all 88.