Skip to content

perf: replace sed subprocess with bash-native SQL literal escaping (local send/inbox/history/watch path) - #897

Open
gon2018 wants to merge 1 commit into
fujibee:mainfrom
gon2018:fix/896-windows-local-send-spawn-cost
Open

perf: replace sed subprocess with bash-native SQL literal escaping (local send/inbox/history/watch path)#897
gon2018 wants to merge 1 commit into
fujibee:mainfrom
gon2018:fix/896-windows-local-send-spawn-cost

Conversation

@gon2018

@gon2018 gon2018 commented Aug 19, 2026

Copy link
Copy Markdown

Fixes #896.

What changed

sed "s/'/''/g" doubles a single quote for SQL literal embedding — a
transform simple enough that bash's own ${var//pattern/replacement} does it
without spawning a process. This swaps the 8 call sites on the ordinary local
send/inbox/history/watch path (the sqlite driver used day-to-day, not the
remote-sync path #780/#799 already fixed) from a sed subprocess to pure bash
parameter expansion:

# before
_sqlite_lit() { printf '%s' "$1" | sed "s/'/''/g"; }
# after
_sqlite_lit() { local s="$1" q="'"; printf '%s' "${s//$q/$q$q}"; }

Same technique in send.sh's roster-check escape, inbox.sh/history.sh
(x2)/watch.sh/check-inbox.sh's JSONL-array escape, and
lib/sqlpath.sh's agmsg_sql_readfile_path (the cygpath call there stays —
only the trailing sed is replaced).

Why (measured on Windows 11 / Git Bash, the environment #896 was filed from)

sed bash-native ratio
single call, 20-run average 221 ms 12 ms ~18x

storage_send alone calls _sqlite_lit 6 times (team/from/to/body/id/at)
per message, so this one function accounts for roughly 1.3s of the ~10s a
single send.sh call cost before this change.

End-to-end, same machine, same team, before/after this patch (installed
locally and exercised against a real two-agent team, not synthetic):

operation before after
send.sh <team> a b "<msg>" 10.5-11.8 s 6.4-7.6 s
full send-to-ack round trip (separate process, real second agent) 35-123 s ~20 s

What this does not fix

history.sh (and check-inbox.sh) separately recompute
storage_list_unread once per distinct recipient in the displayed slice, to
derive the unread ●/○ marker — an N+1 query pattern, unrelated to the
sed→bash swap here, and out of scope for this PR. history.sh on a
159-row store took ~31s both before and after this patch, unaffected by it.
Happy to file that separately if useful; flagging here so it is not read as
"still slow, so this didn't work."

Correctness

Before touching any live file, ${s//$q/$q$q} was checked byte-for-byte
against sed "s/'/''/g" across 19 cases (empty string, leading/trailing/
consecutive quotes, Thai text, unicode, embedded newline/tab, a SQL-injection
attempt, a 2000-char string) — all identical. Live-verified after patching by
sending a real message containing both an apostrophe and Thai text between
two real agents on a live team; it round-tripped byte-identical on both ends
(sender's history.sh and the recipient's own read).

Testing

Tried running the storage/messaging bats suite locally (npm install -g bats, Windows 11 / Git Bash) before opening this — it did not finish in a
reasonable time and produced no output beyond the version banner, which is
consistent with the same per-process spawn cost this PR is about (bats
itself forks a lot per test). Not reporting a pass/fail I did not actually
observe. What I do have is the byte-for-byte equivalence check (19 cases,
above) and the live production round-trip on a real two-agent team, both
before this PR existed. Leaning on CI here rather than guessing at a local
result.

sed "s/'/''/g" spawns a process to double a single quote for SQL literal
embedding - bash's own ${var//pattern/replacement} does the same transform
without spawning. On Windows/Git Bash a single sed call costs ~221ms vs
~12ms for the bash-native equivalent (measured, 20-run average).

Swaps the 8 call sites on the ordinary local send/inbox/history/watch path
(drivers/storage/sqlite.sh _sqlite_lit, send.sh roster-check escape,
inbox.sh/history.sh(x2)/watch.sh/check-inbox.sh JSONL-array escape,
lib/sqlpath.sh agmsg_sql_readfile_path). Does not touch the remote-sync
path (fujibee#780/fujibee#799 already fixed that one).

Measured end-to-end on a real two-agent team, same machine:
  send.sh: 10.5-11.8s -> 6.4-7.6s
  full send-to-ack round trip: 35-123s -> ~20s

Verified byte-identical to the sed version against 19 edge cases (Thai
text, unicode, embedded quotes/newlines/tabs, SQL injection attempt)
before patching, and live-verified with a real message containing an
apostrophe + Thai text round-tripping correctly between two real agents.

See fujibee#896
@fujibee

fujibee commented Aug 21, 2026

Copy link
Copy Markdown
Owner

@gon2018

Thanks for this — and sorry it sat for two days without a reply.

The macOS failures have a cause we can name. _sqlite_lit in your diff is correct; the other seven substitutions use ${var//\'/\'\'}, and bash 3.2 (/bin/bash on macOS, which the macOS runners use) leaves the backslashes in the replacement:

input     it's
bash 3.2  it\'\'s
bash 5.x  it''s

That is why the three failing shards are all quote cases — whoami: resolves team and agent names containing single quotes and the two rename-team quote tests.

We hit the identical thing on a different helper today, tracked as #928, so the q-variable form you used in _sqlite_lit is the one we settled on as well.

We're carrying this forward ourselves so it lands sooner, with your commit credited as co-author. Your scoping was right: the remote-sync path was already handled by #780/#799, and the local send/inbox/history/watch path was the part still spawning a process per escape.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Local send/inbox/history on Windows: 10-33s per call on a 159-row store, same class as #780 but a different code path

2 participants