flock.bin master semaphore is not released when its holder dies, so runtime callers block while the master is still alive

Summary

If a process is terminated while it holds the master lock, master_lock() can stay unavailable to every process that is still running until a master re-creates the shared memory. On Windows, terminating a second participant shortly after it began asset_register() (after its REGISTERING marker) while the master stayed alive left hakopy.init_for_external() in a third process unable to return within 10 s in 10 of 46 trials. In the affected trials where we also probed it, hako-cmd status printed ERROR: file-lock wait timed out after 3000 ms.

master_lock() / master_unlock() in the mmap backend use a counting semaphore whose value is an int in <core_mmap_path>/flock.bin. The OS file lock (LockFileEx / flock) only guards the read-modify-write of that integer. A holder that is terminated after hako_sem_down() has taken the counter from 1 to 0, and before hako_sem_up() restores it, leaves the counter at 0. While it stays at 0, every master_lock() keeps retrying in the 20 ms loop with no limit. Inside a HakoFlockWaitScope the wait ends with HakoFlockWaitError instead. The semaphore has no owner-death detection or automatic restoration.

Apart from the normal hako_sem_up(), the counter is written only by hako_sem_init(0, 1) in flock::create(). HakoSharedMemoryMmap::create_memory() calls it after it has created the mmap object: when a master creates the master memory, and on the Start event when the PDU memory (and, with core-pro, the pro memory) is created. Each call overwrites the value without checking whether anyone holds the semaphore, so it is not owner-death recovery. The Start path also takes master_lock() first, in trigger_event(), so it cannot run at all once the counter is stuck.

hako-cmd is already bounded upstream. #60 (b8f6b38) added an opt-in bounded wait. hakoniwalab/hakoniwa-core-pro#88 (hakoniwalab/hakoniwa-core-pro@c5ebfd8) makes hako-cmd use it (--lock-timeout-ms, default 3000) after first checking that the recorded master pid is alive. That bounded wait covers hako-cmd in this case too, even when the pid check passes. #88 deliberately keeps runtime callers blocking. This report is about those runtime callers when the master is alive and the counter stays at 0, and about whether owner-death handling, or an opt-in bounded wait for runtime callers, would be appropriate.

Code references (hakoniwa-core-cpp b8f6b38 unless noted):

src/hako/utils/hako_share/impl/hako_sem_flock.cpp:54 hako_sem_down(): retries while value > 0 is false (line 66), with a 20 ms sleep (line 74).
src/hako/utils/hako_share/impl/hako_sem_flock.cpp:84 hako_sem_up(): increments the counter.
src/hako/utils/hako_share/impl/hako_sem_flock.cpp:127 hako_sem_init(0, 1): explicit reinitialisation, called from hako_shared_memory_mmap.cpp:57. The PDU-memory creation call is at src/hako/data/hako_pdu_data.hpp:580.
src/hako/hako_simevent_impl.cpp:14: trigger_event() takes the master lock before any state change, Start included.
hakoniwa-core-pro e56655e sources/assets/callback/src/hako_asset_impl.cpp:254-271: asset registration holds the master lock across shared PDU channel creation.
From reading the code, the first master_lock() on the init_for_external() path is hako::create_asset_controller() -> HAKO_LOG_INFO -> HakoMasterData::add_log() (src/hako/data/hako_master_data.hpp:426). That acquisition is skipped once the shared log is full (:423).

Environment

Windows 11, MSVC (Visual Studio 2022), PowerShell 5.1, shm_type: mmap
main: hakoniwa-core-pro e56655e (2026-09-17) with hakoniwa-core-cpp b8f6b38 (2026-08-30). hakopy was built for CPython 3.13. hako-cmd.exe, hakopy.pyd and shakoc.dll were copied from the build output into one directory, the -HakoBin argument, because the install step did not complete in our setup.
An older build, used in the comparison and counter sections: hakoniwa-core-pro dccd99f (2026-07-30) with hakoniwa-core-cpp 91119b2. It predates both Add bounded and interruptible file-lock waits for command clients #60 and Prevent hako-cmd from hanging on stale runtime and file locks hakoniwalab/hakoniwa-core-pro#88. hakopy was built for CPython 3.12, and the build was used from its install directory.

Repro: master alive (`repro-live.ps1`, main)

Process A calls conductor_start() and asset_register(), then hakopy.start(). A stays alive for the whole trial.
Process B prints REGISTERING and then calls asset_register() for an asset with 60 PDU channels.
The harness polls B's log about every 50 ms, waits for the configured delay after the marker wait returns, and terminates B (Stop-Process -Force). The time from the print to the kill includes polling and scheduling overhead, which we did not measure. The marker wait can also return on its 30 s timeout, and the script does not reject that case.
Process C calls hakopy.init_for_external(), and the harness gives C 10 s to exit. In batches 3 and 4 the harness then also runs hako-cmd status.

batch	delay after the harness saw the marker (ms)	trials	C hung (no exit within 10 s)
1	100, 200, 300, 400, 500, 700, 900, 1200	8	0
2	0–120 in steps of 10, 140, 160	15	5
3	10, 30, 70, 90, 160, 10, 30, 70	8	1
4	0–120 in steps of 10, 140, 160 (script default)	15	4
total		46	10

The script reports these trials as external attach: HUNG and counts them in RESULT wedged=N/M. Batches 1 to 3 ran earlier revisions of the script with the same kill logic: batches 1 and 2 had no hako-cmd status probe, and batch 4 is the script as published below. A later run of the published script from a fresh copy of the files in this issue hung in 2 of 15 trials, and hako-cmd status timed out in both.

C printed ATTACHED in all 36 trials where it exited. A was alive in all 46.
REGISTERED had appeared in B's log before the kill in 17 trials, and none of them hung. It had not appeared in the other 29, of which 10 hung and 19 attached. That B died while holding the master lock it takes across channel creation is an inference consistent with this. We did not instrument it.
flock.bin[0] was sampled once, about 1 s after the kill, while A was still running. It read 0 in all 10 hung trials. In trials that attached it read 0, 1, or could not be read. A single 0 sample does not establish an abandoned count, because a live holder also takes the counter to 0 while it holds the semaphore. Unreadable samples are, in our reading, reads that hit the exclusive LockFileEx that hako_flock_acquire() holds over the file. Ownership at termination was not instrumented.
In batches 3 and 4, hako-cmd status printed ERROR: file-lock wait timed out after 3000 ms in all 5 hung trials and status=stopped in the other 18.
The simulation was never started in this repro (status=stopped). We observed blocked attachment only. We did not test whether stepping of a running simulation stalls.

Comparison: master killed (`repro.ps1`)

Here the master itself is terminated right after hako-cmd stop, with no hako-cmd reset (-Order kill). On main, hako-cmd status returned in 10 of 10 trials over two runs. The output we kept, from one trial, reported ERROR: Hakoniwa master is not running (stale pid=…), which comes from the pid check in hakoniwalab/hakoniwa-core-pro#88. flock.bin[0] read 0 after the kill in 4 of 5 trials in one run (0, 0, 1, 0, 0) and in none of the later run (1, 1, 1, 2, 1). On the older build, hako-cmd status exceeded 10 s in 4 of 5 trials in one run and in 0 of 5 in a later run.

With hako-cmd stop, hako-cmd reset, a 3 s wait and then termination (-Order reset), the counter read 1 and status returned in 5 of 5 trials on each build. Every trial's master registered normally, including trials that followed one that had left the counter at 0. That fits create() reinitialising it. We did not read the counter right after re-creation, and this does not show that surviving participants would recover. So this issue is only about the case where the master stays alive.

Also observed: counter above 1 (not investigated)

In the 30 repro.ps1 trials (both orders, both builds), flock.bin[0] read 2 in 4 trials, on both builds. One of these was read while the simulation was running, before that trial's kill. A value above 1 would let two processes into the master critical section at once, because hako_sem_down() only checks value > 0 and hako_sem_up() has no upper bound. One possible mechanism, from reading the code and not verified: on Start, create() for the PDU memory rewrites the counter to 1 while another process holds the semaphore, and the holder's later hako_sem_up() leaves it at 2. In a separate run that sampled the counter 200 times before and 200 times after hako-cmd start in 10 trials (both builds, sample-counter.ps1; no process was terminated during the sampling phases, and the asset is terminated afterwards for cleanup), it never read 2.

Expected

A participant that dies inside the master critical section should not leave the semaphore unavailable to the processes that are still alive. At minimum, runtime callers such as init_for_external() would benefit from an error after a bounded wait instead of blocking forever, as hako-cmd now does.

Possible directions (for discussion)

On Windows, a named mutex: a waiter gets WAIT_ABANDONED when the owner has died. POSIX would need its own mechanism, for example a robust process-shared mutex. In both cases the new owner has to treat the shared state as possibly half-updated.
For the master slot (index 0) only, hold the OS file lock (LockFileEx / flock) for the whole critical section instead of counting in the file. The OS releases it when the holder dies (on Windows asynchronously, so possibly after a short delay). Asset slots 1..N are signalling semaphores, decremented in one process and incremented in another, and cannot become locks this way. create() currently closes and reopens the thread-local handle (hako_sem_flock.cpp:117-121), which would silently drop a held lock.
Store the holder's pid next to the counter and reclaim the count when that process no longer exists. Guard against pid reuse, for example by also storing the process start time, and do the check-and-reclaim under the OS file lock.

The POSIX mmap backend uses the same hako_sem_flock.cpp, so we expect the same behaviour there. The POSIX shm backend uses a SysV semaphore instead. Its master_lock() / master_unlock() call semop() with sem_flg = 0, without SEM_UNDO (src/hako/utils/hako_share/impl/posix/hako_sem.cpp:71-94), so by code reading a holder killed in the critical section would leave that semaphore at 0 as well. SEM_UNDO would release the count, but would not repair half-updated shared state. We tested only on Windows.

Related: #60 (opt-in bounded wait API), hakoniwalab/hakoniwa-core-pro#88 (hako-cmd uses it and rejects a dead master; #88 already names "another process retains the logical master lock" as one cause of hako-cmd hangs), and #56 (closed, which listed deciding a recovery policy for stale flock state).

Whether runtime callers such as init_for_external() are meant to keep blocking here, or whether the counter should recover from a dead holder, decides where a change belongs. We can rerun these repros on Windows against a proposed change.

Repro files (Windows, PowerShell 5.1 + Python)

Save every file below in one directory whose absolute path contains no spaces, and generate the two victim JSON files there with the snippet. -HakoBin is the directory that holds hakopy and hako-cmd.exe, and -Python must be the interpreter hakopy was built for. Each script writes its own cpp_core_config.json with a private core_mmap_path under work*\ next to the scripts. Run each script in a child PowerShell:

powershell -NoProfile -ExecutionPolicy Bypass -Command "& .\repro-live.ps1 -HakoBin '<bin dir>' -Python '<python.exe>'"
powershell -NoProfile -ExecutionPolicy Bypass -Command "& .\repro.ps1 -HakoBin '<bin dir>' -Python '<python.exe>' -Order kill"
powershell -NoProfile -ExecutionPolicy Bypass -Command "& .\sample-counter.ps1 -HakoBin '<bin dir>' -Python '<python.exe>'"

A child shell keeps the HAKO_CONFIG_PATH / HAKO_BIN the scripts set out of your session, and also works when script execution is disabled. The scripts stop only the processes they start. If a script stops with an error (for example "master did not register"), or you press Ctrl+C, the python.exe it started keeps running and holds the repro's mmap. Stop it before you rerun.

The results depend on timing. With the default delays, repro-live.ps1 hung in 5, 4 and 2 of 15 trials in three runs, and batch 3, which repeated delays that had hung before, hung in only 1 of 8. If a run shows no hang, run it again or pass more delays, for example -DelaysMs @(0,10,20,30,40,50,60,70,80,90).

custom.json (process A)

{
  "paths": [ { "id": "default", "path": "pdutypes.json" } ],
  "robots": [ { "name": "DummyRobot", "pdutypes_id": "default" } ]
}

pdutypes.json (process A)

[
  { "channel_id": 0, "pdu_size": 72, "name": "cmd", "type": "geometry_msgs/Twist" }
]

custom_victim.json and pdutypes_many.json (process B) are generated as follows:

import json
json.dump([{"channel_id": i, "pdu_size": 72, "name": f"ch{i}", "type": "geometry_msgs/Twist"} for i in range(60)], open("pdutypes_many.json", "w"), indent=1)
json.dump({"paths": [{"id": "default", "path": "pdutypes_many.json"}], "robots": [{"name": "VictimRobot", "pdutypes_id": "default"}]}, open("custom_victim.json", "w"), indent=1)

asset.py (process A)

"""Minimal asset that owns the conductor. Usage: python asset.py <custom.json>

Requires HAKO_BIN (directory with hakopy) and HAKO_CONFIG_PATH in the environment.
"""
import os
import sys

sys.stdout.reconfigure(line_buffering=True)
hako_bin = os.environ["HAKO_BIN"]
sys.path.insert(0, hako_bin)
if hasattr(os, "add_dll_directory"):
    os.add_dll_directory(hako_bin)
import hakopy  # noqa: E402

DELTA_USEC = 1000
callbacks = {
    "on_initialize": lambda context: 0,
    "on_simulation_step": lambda context: 0,
    "on_manual_timing_control": None,
    "on_reset": lambda context: 0,
}

if not hakopy.conductor_start(DELTA_USEC, DELTA_USEC):
    sys.exit("conductor_start failed")
if not hakopy.asset_register("DummyAsset", sys.argv[1], callbacks, DELTA_USEC, hakopy.HAKO_ASSET_MODEL_PLANT):
    sys.exit("asset_register failed")
print("REGISTERED")
hakopy.start()
hakopy.conductor_stop()

victim.py (process B)

"""Registers a second asset (no conductor) with many channels, then idles."""
import os
import sys

sys.stdout.reconfigure(line_buffering=True)
hako_bin = os.environ["HAKO_BIN"]
sys.path.insert(0, hako_bin)
if hasattr(os, "add_dll_directory"):
    os.add_dll_directory(hako_bin)
import hakopy  # noqa: E402

callbacks = {"on_initialize": lambda c: 0, "on_simulation_step": lambda c: 0,
             "on_manual_timing_control": None, "on_reset": lambda c: 0}
print("REGISTERING")
ok = hakopy.asset_register("VictimAsset", sys.argv[1], callbacks, 1000, hakopy.HAKO_ASSET_MODEL_PLANT)
print("REGISTERED" if ok else "REGISTER FAILED")
hakopy.start()

external.py (process C)

"""Attach to an existing Hakoniwa master as an external process. Prints ATTACHED on success."""
import os
import sys

hako_bin = os.environ["HAKO_BIN"]
sys.path.insert(0, hako_bin)
if hasattr(os, "add_dll_directory"):
    os.add_dll_directory(hako_bin)
import hakopy  # noqa: E402

print("ATTACHED" if hakopy.init_for_external() else "ATTACH FAILED", flush=True)

repro-live.ps1 (master alive)

# Repro (master alive): observe REGISTERING, wait the configured delay,
# terminate the second participant, then probe init_for_external() and hako-cmd status.
# Timing-dependent; semaphore ownership at termination is inferred.
#
#   .\repro-live.ps1 -HakoBin '<bin dir>' -Python '<python.exe>' [-DelaysMs @(10,30,70,90,160)]
param(
    [Parameter(Mandatory)][string]$HakoBin,
    [string]$Python = 'python',
    [int[]]$DelaysMs = @(0, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 110, 120, 140, 160)
)
$ErrorActionPreference = 'Stop'
$here = $PSScriptRoot
$work = Join-Path $here 'work-live'
$mmap = Join-Path $work 'mmap'
New-Item -ItemType Directory -Force -Path $mmap | Out-Null
$config = Join-Path $work 'cpp_core_config.json'
@{ shm_type = 'mmap'; core_mmap_path = $mmap.Replace([char]92, [char]47); asset_timeout_usec = 600000000 } |
    ConvertTo-Json | Set-Content -LiteralPath $config -Encoding ascii
$env:HAKO_CONFIG_PATH = $config
$env:HAKO_BIN = $HakoBin
$flock = Join-Path $mmap 'flock.bin'
function Read-MasterCounter {
    try { $s = [IO.File]::Open($flock, 'Open', 'Read', 'ReadWrite'); try { $b = New-Object byte[] 4; $null = $s.Read($b, 0, 4); [BitConverter]::ToInt32($b, 0) } finally { $s.Close() } } catch { 'unreadable' }
}
function Start-Py([string]$Script, [string[]]$More, [string]$Log) {
    Start-Process -FilePath $Python -ArgumentList (@('-B', (Join-Path $here $Script)) + $More) -NoNewWindow -PassThru `
        -RedirectStandardOutput $Log -RedirectStandardError "$Log.err"
}
function Wait-Line([string]$Log, [string]$Text, [int]$Sec) {
    $deadline = (Get-Date).AddSeconds($Sec)
    while ((Get-Date) -lt $deadline) {
        if (Select-String -LiteralPath $Log -Pattern $Text -Quiet -ErrorAction SilentlyContinue) { return $true }
        Start-Sleep -Milliseconds 50
    }
    $false
}

$wedged = 0
foreach ($delay in $DelaysMs) {
    $tag = "d$delay"
    $master = Start-Py 'asset.py' @((Join-Path $here 'custom.json')) (Join-Path $work "master-$tag.log")
    if (-not (Wait-Line (Join-Path $work "master-$tag.log") 'REGISTERED' 30)) { throw "${tag}: master did not register" }
    $victimLog = Join-Path $work "victim-$tag.log"
    $victim = Start-Py 'victim.py' @((Join-Path $here 'custom_victim.json')) $victimLog
    $null = Wait-Line $victimLog 'REGISTERING' 30
    Start-Sleep -Milliseconds $delay
    $registered = [bool](Select-String -LiteralPath $victimLog -Pattern '^REGISTERED' -Quiet -ErrorAction SilentlyContinue)
    Stop-Process -Id $victim.Id -Force
    Start-Sleep 1
    $counter = Read-MasterCounter
    # Probe with an unbounded runtime caller while the master is still alive.
    $probeLog = Join-Path $work "probe-$tag.log"
    $probe = Start-Py 'external.py' @() $probeLog
    $attached = $probe.WaitForExit(10000)
    if (-not $attached) { $probe.Kill(); $wedged++ }
    $masterAlive = -not $master.HasExited
    $cmd = Start-Process -FilePath (Join-Path $HakoBin 'hako-cmd.exe') -ArgumentList 'status' -NoNewWindow -PassThru `
        -RedirectStandardOutput (Join-Path $work "cmd-$tag.out") -RedirectStandardError (Join-Path $work "cmd-$tag.err")
    $cmdState = if ($cmd.WaitForExit(30000)) { (Get-Content (Join-Path $work "cmd-$tag.out"), (Join-Path $work "cmd-$tag.err") | Where-Object { $_ -match 'ERROR|status|state' } | Select-Object -Last 1) } else { $cmd.Kill(); 'HUNG' }
    "kill at ${delay}ms: victim registered before kill=$registered counter=$counter master alive=$masterAlive external attach: $(if ($attached) { (Get-Content $probeLog | Select-Object -Last 1) } else { 'HUNG' }) | hako-cmd status: $cmdState"
    Stop-Process -Id $master.Id -Force -ErrorAction SilentlyContinue
    Start-Sleep 1
}
"RESULT wedged=$wedged/$($DelaysMs.Count)"

repro.ps1 (master killed)

# Repro (master killed): hako-cmd stop, then terminate the master either
# immediately (-Order kill) or after hako-cmd reset and a 3 s wait (-Order reset).
# Reads flock.bin[0] after the kill and probes hako-cmd status with a 10 s limit.
#
#   .\repro.ps1 -HakoBin '<bin dir>' -Python '<python.exe>' [-Trials 5] [-Order kill|reset]
param(
    [Parameter(Mandatory)][string]$HakoBin,
    [int]$Trials = 5,
    [string]$Python = 'python',
    # kill : hako-cmd stop, then kill the asset immediately
    # reset: hako-cmd stop, hako-cmd reset, wait 3 s, then kill
    [ValidateSet('kill', 'reset')][string]$Order = 'kill'
)
$ErrorActionPreference = 'Stop'
$here = $PSScriptRoot
$work = Join-Path $here 'work'
$mmap = Join-Path $work 'mmap'
New-Item -ItemType Directory -Force -Path $mmap | Out-Null
$config = Join-Path $work 'cpp_core_config.json'
@{ shm_type = 'mmap'; core_mmap_path = $mmap.Replace([char]92, [char]47); asset_timeout_usec = 600000000 } |
    ConvertTo-Json | Set-Content -LiteralPath $config -Encoding ascii
$env:HAKO_CONFIG_PATH = $config
$env:HAKO_BIN = $HakoBin
$hakoCmd = Join-Path $HakoBin 'hako-cmd.exe'
$flock = Join-Path $mmap 'flock.bin'

function Read-MasterCounter {
    try {
        $s = [IO.File]::Open($flock, 'Open', 'Read', 'ReadWrite')
        try { $b = New-Object byte[] 4; $null = $s.Read($b, 0, 4); [BitConverter]::ToInt32($b, 0) } finally { $s.Close() }
    } catch { 'unreadable' }
}
function Invoke-HakoCmd([string]$Action, [int]$TimeoutSec = 10) {
    $p = Start-Process -FilePath $hakoCmd -ArgumentList $Action -NoNewWindow -PassThru `
        -RedirectStandardOutput (Join-Path $work "cmd.out") -RedirectStandardError (Join-Path $work "cmd.err")
    if ($p.WaitForExit($TimeoutSec * 1000)) { return 'returned' }
    $p.Kill(); return 'HUNG'
}

$hung = 0
for ($i = 1; $i -le $Trials; $i++) {
    $log = Join-Path $work "asset-$i.log"
    $asset = Start-Process -FilePath $Python -ArgumentList '-B', (Join-Path $here 'asset.py'), (Join-Path $here 'custom.json') `
        -NoNewWindow -PassThru -RedirectStandardOutput $log -RedirectStandardError "$log.err"
    $deadline = (Get-Date).AddSeconds(30)
    while (-not (Select-String -LiteralPath $log -Pattern 'REGISTERED' -Quiet -ErrorAction SilentlyContinue)) {
        if ((Get-Date) -gt $deadline -or $asset.HasExited) { throw "trial ${i}: asset did not register" }
        Start-Sleep -Milliseconds 100
    }
    $null = Invoke-HakoCmd 'start'
    Start-Sleep 2
    $before = Read-MasterCounter
    $null = Invoke-HakoCmd 'stop'
    if ($Order -eq 'reset') {
        $null = Invoke-HakoCmd 'reset'
        Start-Sleep 3
    }
    Stop-Process -Id $asset.Id -Force -ErrorAction SilentlyContinue
    Start-Sleep 1
    $after = Read-MasterCounter
    $probe = Invoke-HakoCmd 'status'
    if ($probe -eq 'HUNG') { $hung++ }
    "trial ${i}: counter before=$before after-kill=$after  hako-cmd status: $probe"
}
"RESULT order=$Order hung=$hung/$Trials"

sample-counter.ps1 (counter sampling around hako-cmd start)

# Sample flock.bin[0] before and after hako-cmd start; nothing is terminated during
# sampling (the asset is terminated afterwards for cleanup).
#
#   .\sample-counter.ps1 -HakoBin '<bin dir>' -Python '<python.exe>' [-Trials 5]
param(
    [Parameter(Mandatory)][string]$HakoBin,
    [string]$Python = 'python',
    [int]$Trials = 5,
    [int]$Samples = 200
)
$ErrorActionPreference = 'Stop'
$here = $PSScriptRoot
$work = Join-Path $here 'work-sample'
$mmap = Join-Path $work 'mmap'
New-Item -ItemType Directory -Force -Path $mmap | Out-Null
$config = Join-Path $work 'cpp_core_config.json'
@{ shm_type = 'mmap'; core_mmap_path = $mmap.Replace([char]92, [char]47); asset_timeout_usec = 600000000 } |
    ConvertTo-Json | Set-Content -LiteralPath $config -Encoding ascii
$env:HAKO_CONFIG_PATH = $config
$env:HAKO_BIN = $HakoBin
$hakoCmd = Join-Path $HakoBin 'hako-cmd.exe'
$flock = Join-Path $mmap 'flock.bin'

function Sample-Counter([int]$Count) {
    $hist = @{}
    for ($k = 0; $k -lt $Count; $k++) {
        try {
            $s = [IO.File]::Open($flock, 'Open', 'Read', 'ReadWrite')
            try { $b = New-Object byte[] 4; $null = $s.Read($b, 0, 4); $v = [string][BitConverter]::ToInt32($b, 0) } finally { $s.Close() }
        } catch { $v = 'unreadable' }
        $hist[$v] = 1 + [int]$hist[$v]
        Start-Sleep -Milliseconds 10
    }
    ($hist.GetEnumerator() | Sort-Object Name | ForEach-Object { "$($_.Name)=$($_.Value)" }) -join ' '
}

for ($i = 1; $i -le $Trials; $i++) {
    $log = Join-Path $work "asset-$i.log"
    $asset = Start-Process -FilePath $Python -ArgumentList '-B', (Join-Path $here 'asset.py'), (Join-Path $here 'custom.json') `
        -NoNewWindow -PassThru -RedirectStandardOutput $log -RedirectStandardError "$log.err"
    $deadline = (Get-Date).AddSeconds(30)
    while (-not (Select-String -LiteralPath $log -Pattern 'REGISTERED' -Quiet -ErrorAction SilentlyContinue)) {
        if ((Get-Date) -gt $deadline -or $asset.HasExited) { throw "trial ${i}: asset did not register" }
        Start-Sleep -Milliseconds 100
    }
    $pre = Sample-Counter $Samples
    $p = Start-Process -FilePath $hakoCmd -ArgumentList 'start' -NoNewWindow -PassThru -RedirectStandardOutput (Join-Path $work 'start.out') -RedirectStandardError (Join-Path $work 'start.err')
    $null = $p.WaitForExit(10000)
    $post = Sample-Counter $Samples
    "trial ${i}: before start [$pre]  after start [$post]"
    $p = Start-Process -FilePath $hakoCmd -ArgumentList 'stop' -NoNewWindow -PassThru -RedirectStandardOutput (Join-Path $work 'stop.out') -RedirectStandardError (Join-Path $work 'stop.err')
    $null = $p.WaitForExit(10000)
    $p = Start-Process -FilePath $hakoCmd -ArgumentList 'reset' -NoNewWindow -PassThru -RedirectStandardOutput (Join-Path $work 'reset.out') -RedirectStandardError (Join-Path $work 'reset.err')
    $null = $p.WaitForExit(10000)
    Start-Sleep 3
    Stop-Process -Id $asset.Id -Force -ErrorAction SilentlyContinue
    Start-Sleep 1
}

fix: let an in-flight asset reach readiness before stop-all cancels s…

feat: warn when stopping one asset leaves the core unresponsive

Follow-up: I re-ran the master-alive repro against hakoniwalab/hakoniwa-core-pro@cd12c6f, which merged hakoniwalab/hakoniwa-core-pro#95 (Windows opt-in HAKO_CORE_SHARED, a single hako.dll). The behavior did not change.

I did not expect #95 to fix this, because it changes how modules inside one process share the Core state, and the stuck counter is shared across processes through flock.bin. I checked anyway because #95 is the only upstream change in this area since the report.

Build

core-pro cd12c6f with hakoniwa-core-cpp b8f6b38 (unchanged since the report).
win-build.ps1 -CoreShared ON -EnableGTest OFF, MSVC VS2022, Release x64. hakopy is built for CPython 3.13.
dumpbin /dependents shows that hakopy.pyd, hako-cmd.exe and shakoc.dll all import hako.dll.
Baseline: core-pro e56655e, the first parent of the merge, with the default static Core. It is the same build as in the report, run again on the same day.

Result (repro-live.ps1, default delays 0–120 ms in 10 ms steps plus 140 and 160 ms, 15 trials per run)

Build	Run	Hung	Delays that hung
`cd12c6f`, `core_shared` ON	1	3/15	80, 90, 120 ms
`cd12c6f`, `core_shared` ON	2	1/15	100 ms
`e56655e`, static Core	1	4/15	60, 90, 100, 120 ms

Every hung trial matched the original report:

init_for_external() did not return within 10 s, while the master was still alive.
flock.bin[0] read 0 one second after the kill.
The second participant had not printed REGISTERED before it was killed.
hako-cmd status printed ERROR: file-lock wait timed out after 3000 ms.

In every trial that did not hang, the probe printed ATTACHED and hako-cmd status reported status=stopped.

So #62 still reproduces with the shared Core library, and the questions in the report are still open: owner-death handling for the master semaphore, or an opt-in bounded wait for runtime callers.

Thank you for the careful reproduction work, including the follow-up with HAKO_CORE_SHARED. The results make it clear that this is an inter-process lock owner-death / session-lifecycle problem, not a shared-DLL issue.

After reviewing the behavior and the current Launcher implementation, we propose the following direction for #62:

Policy: fail-stop, not automatic lock recovery

If a process dies while holding the master lock, shared simulation state may have been only partially updated. Simply restoring the semaphore counter or handing the lock to another process does not establish that the simulation is valid. We will therefore treat a confirmed abandoned master lock as a session-level ABORT rather than automatically taking over the lock.

Core (hakoniwa-core-cpp): Extend the existing flock.bin with versioned owner information (PID, with protection against PID reuse where practical). Update it consistently with the master-lock state under the short-lived OS file lock. If a runtime caller confirms that the logical master lock is still held by a dead owner, emit a lock-independent error log and raise an error rather than waiting indefinitely or recovering automatically. The existing flock::create() reinitialization behavior must also be corrected so it cannot reset a live semaphore.
Diagnostics (hakoniwa-core-pro): Add hako-cmd check, a read-only, extensible health report with statuses OK, ABORT, and UNKNOWN, initially checking only the master lock. check must not acquire the logical master lock or require attaching to a potentially broken runtime.
- Default: nonblocking OS try-lock. If the OS file lock is temporarily busy, return UNKNOWN promptly (never hang, never infer ABORT from mere contention).
- Optional: --force-wait. Explicitly enable a bounded wait, using --lock-timeout-ms (default 3000 ms). A timed-out check returns a diagnostic/UNKNOWN rather than waiting forever. The Launcher will not use --force-wait.
- Missing owner metadata in an older flock format, an absent runtime, or uncertain PID liveness means UNKNOWN, not ABORT. A recorded dead owner with the logical lock still held means ABORT. We should use a versioned JSON report suitable for future health checks.
Supervision (hakoniwa-pdu-python Launcher): Run hako-cmd check before session startup and periodically during watch (with an independent subprocess timeout). On ABORT, record a FAILED session, stop all Launcher-managed processes, verify termination, and preserve the mmap/flock files for diagnosis. UNKNOWN does not automatically abort the session.
Explicit operator recovery: Add hako-cmd recover. Only after all relevant processes have stopped, an operator may clean the known runtime artifacts (mmap-0x*.bin, flock.bin, pdu_init.lock, pro_init.lock) and then start a new simulation session. This is intentionally separate from hako-cmd reset. Launcher will never auto-recover or auto-delete the evidence.
Compatibility and CI: Preserve legacy format detection as UNKNOWN rather than silently misinterpreting owner metadata; define the supported new/old writer boundary. Test owner death, transient OS-lock contention, legacy files, session failure, and manual recovery on Windows, macOS, and Linux CI.
Versioning: Coordinate this user-visible feature with the next core-pro release (proposed v1.4.0 / hako-cmd 1.1.0); align the currently differing release tag, CMake project version, and command version when the implementation is ready.

Implementation tracking

Detect abandoned mmap master lock without auto-recovery (owner metadata and fail-stop) #63 — owner metadata, fail-stop detection, and safe initialization.
Add hako-cmd check/recover for fail-stop mmap sessions and version the CLI hakoniwalab/hakoniwa-core-pro#96 — hako-cmd check / recover, JSON/reporting, and versioning.
Launcher: watch hako-cmd check, fail-stop entire session, preserve files for manual recovery hakoniwalab/hakoniwa-pdu-python#62 — Launcher health monitoring, fail-stop/FAILED lifecycle, and preservation of crash evidence.

The key distinction is that a corrupt/uncertain simulation session must not be silently resumed. Owner-death detection lets us fail clearly; recovery is a deliberate lifecycle operation. We will track the work across those issues and use the existing reproducer as a regression test. Thank you again for making the failure mode so concrete.

Implemented and merged across all three repositories. Closing this parent issue as completed.

Merged implementations

Add fail-stop detection for abandoned master locks #64 — merge commit c48016f121185f091cda7da52341b3de13e1ad41: versioned owner metadata in flock.bin, PID/start-token identity and zombie detection, repeated owner-death checks during runtime wait, fail-stop without lock takeover, safe new-file initialization, and fixed Windows byte-range locking.
Add runtime health check and explicit recovery hakoniwalab/hakoniwa-core-pro#97 — merge commit 1c4937d98f546da48a4dcb8213474f4145bca072: read-only hako-cmd check (nonblocking by default; bounded --force-wait), versioned OK/ABORT/UNKNOWN JSON, explicit safety-checked recover, C/Python failure propagation, and CLI 1.1.0 / Core PRO 1.4.0 version updates. Includes Core CPP 25ad617 as its submodule revision.
Add Launcher fail-stop runtime supervision hakoniwalab/hakoniwa-pdu-python#63 — merge commit b8fe6cbfbdf742d58544056ad32a68ed82b24645: Launcher health supervision, persistent FAILED state and diagnostics, managed process shutdown, and crash-evidence preservation with no automatic recovery.

Validation

The latest PR workflows passed on Linux, macOS and Windows, including Core PRO's standard and alternate build manifests. The original Windows hang and a subsequent Windows integration-test file-handle leak were fixed and revalidated.

Policy / scope

This is an abandoned-owner fail-stop solution, not automatic shared-state recovery. A confirmed dead owner aborts the simulation session; recovery remains an explicit operator action after stopping all related processes. Asset-registration critical-section refactoring, heartbeat detection of a live but hung process, and SysV shm owner-death recovery remain deliberately out of scope.

Linked implementation issues #63, hakoniwalab/hakoniwa-core-pro#96 and hakoniwalab/hakoniwa-pdu-python#62 are closed. No release tag was created as part of this merge.

Follow-up hardening issues were opened after the three-repository implementation of #62 was merged. This parent issue remains completed/closed; these are separate post-merge compatibility, safety, and lifecycle improvements (not a reopening of the original owner-death reproducer).

Follow-up to #62: legacy flock compatibility, mixed-writer boundary, and UNKNOWN semantics #65 — legacy flock format, mixed-version writer boundary, diagnostic UNKNOWN vs confirmed ABORT, supported upgrade contract.
Follow-up #62: contain owner-death errors across the full runtime C/Python API hakoniwalab/hakoniwa-core-pro#98 — runtime C ABI and hakopy exception containment while a session is already running.
Follow-up #62: harden hako-cmd recover ordering, startup races, and upgrade guidance hakoniwalab/hakoniwa-core-pro#99 — recover ordering, concurrent startup/external process safety, legacy migration and version documentation.
Follow-up #62: preserve FAILED during concurrent launch, and handle uncertain health reliably hakoniwalab/hakoniwa-pdu-python#64 — FAILED state persistence across startup/watch races; safe handling of malformed/UNKNOWN health results.

Urban Mobility, drone, and drone-show recipe migration from cleanup_mmap_on_start: true is tracked separately by the owner and is intentionally excluded from these issues.

Do not auto-recover a corrupt session or silently discard crash evidence.

I re-ran the master-alive repro from the report against the merged fix on Windows. On the fixed build, none of 30 probes hung: 23 attached, and 7 exited with the new fail-stop error. The baseline hung in 2 of 15 trials under the same harness.

Builds (Windows 11, MSVC VS2022, Release x64, static Core, hakopy for CPython 3.13)

Fixed: hakoniwa-core-pro 56019b3 (includes Add runtime health check and explicit recovery hakoniwalab/hakoniwa-core-pro#97) with hakoniwa-core-cpp 3def26c (includes Add fail-stop detection for abandoned master locks #64, Harden legacy master-lock compatibility #66, flock: diagnostics to stderr ('open' wording for an existing lock file) #70, Conductor event handling takes the master lock only in transitional states #74 and Master: no allocation per execute() (one buffer for the assets' times) #75). hako-cmd --version prints 1.4.0.
Baseline: core-pro e56655e with core-cpp b8f6b38, the same binaries as in the report (hako-cmd 1.0.1).

Harness

This is the report's repro-live.ps1 with two changes:

The mmap directory is emptied before each trial. Both builds ran with this change.
The probe's elapsed time and stderr are recorded.

The harness can also run hako-cmd check after hako-cmd status. I enabled that for the fixed-build runs only. The delays are the same as in the report: 0–120 ms in 10 ms steps, then 140 and 160 ms. Each run has 15 trials. A probe that has not exited after 10 s counts as hung.

Result

Build	Run	Hung	Fail-stop	Attached
`e56655e` (baseline)	1	2/15 (10, 90 ms)	–	13
`56019b3`	1	0/15	3/15 (0, 20, 50 ms)	12
`56019b3`	2	0/15	4/15 (0, 10, 20, 30 ms)	11

The two hung baseline trials matched the original report. flock.bin[0] was 0, and hako-cmd status printed ERROR: file-lock wait timed out after 3000 ms.

On the fixed build, the post-kill counter was 0 in seven trials. In each of those seven:

The probe process exited within 250 ms of launch, with this exception. The 250 ms covers interpreter startup and exit; the longest was 247 ms.
RuntimeError: Hakoniwa runtime failure: master lock owner process is no longer alive; stop every Hakoniwa process, install one Core version, then run 'hako-cmd recover' before restarting
hako-cmd status exited 1 with HAKONIWA_FAIL_STOP health=ABORT.
hako-cmd check exited 2. Its JSON reported "status":"ABORT", "state":"LOCKED", "counter":0 and the same reason.

The other 23 fixed-build trials printed ATTACHED, and check returned OK (exit 0).

Recovery

This was a separate run that stopped at the first ABORT. The ABORT came on the 2nd trial.

The owner_pid that hako-cmd check reported matched the pid of the participant I had killed.
With the master still alive, hako-cmd recover exited 4 with "status":"REFUSED" and "a Hakoniwa process is still alive: pid=<master pid>". It left the runtime files in place.
I stopped the master myself. check still reported ABORT. recover then exited 0 with "status":"RECOVERED" and removed mmap-0xff.bin and flock.bin.
A new master registered. A new external probe printed ATTACHED, and check returned OK.

Observation (no action requested)

In the seven ABORT trials, the master was still alive when the harness checked. Its last logged state was WAIT START, and its output had no fail-stop diagnostic. The probe, status and check were what detected the abandoned lock. check only reports the lock: the master keeps running until something stops it. That matches the scope you described, in which stopping the session is the Launcher's job. This repro starts the master outside the Launcher.

Limits

The runs cover only the clean-directory variant of a timing-dependent reproducer, and scheduling is not controlled.
The counts are observed outcomes. They are not failure rates, and they do not guarantee that the build never hangs.

Thank you for the thorough Windows re-verification and the clear before/after measurements.

It is very helpful to see the original hang reproduced on the baseline, while the updated build completed all 30 probes without hanging (23 attached and 7 returned the intended fail-stop error). I also appreciate the separate recovery test confirming that recovery is refused while the master is alive and succeeds after it is stopped.

Your note about the master staying alive when launched outside the Launcher is consistent with our current responsibility boundary, and the limits of this timing-dependent test are understood.

Thanks again for taking the time to verify both the failure and recovery paths.

feat: ask hako-cmd check first when looking for a wedged core

Navigation Menu

Repository navigation

flock.bin master semaphore is not released when its holder dies, so runtime callers block while the master is still alive #62

Summary

Environment

Repro: master alive (`repro-live.ps1`, main)

Comparison: master killed (`repro.ps1`)

Also observed: counter above 1 (not investigated)

Expected

Possible directions (for discussion)

Policy: fail-stop, not automatic lock recovery

Implementation tracking

Merged implementations

Validation

Policy / scope

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

flock.bin master semaphore is not released when its holder dies, so runtime callers block while the master is still alive #62

Description

Summary

Environment

Repro: master alive (repro-live.ps1, main)

Comparison: master killed (repro.ps1)

Also observed: counter above 1 (not investigated)

Expected

Possible directions (for discussion)

Activity

ykikuchii commented on Oct 16, 2026

tmori commented on Oct 16, 2026

Policy: fail-stop, not automatic lock recovery

Implementation tracking

tmori commented on Oct 16, 2026

Merged implementations

Validation

Policy / scope

tmori commented on Oct 16, 2026

ykikuchii commented on Oct 19, 2026

tmori commented on Oct 19, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Repro: master alive (`repro-live.ps1`, main)

Comparison: master killed (`repro.ps1`)