perf: replace sed subprocess with bash-native SQL literal escaping (local send/inbox/history/watch path) - #897
Conversation
sed "s/'/''/g" spawns a process to double a single quote for SQL literal
embedding - bash's own ${var//pattern/replacement} does the same transform
without spawning. On Windows/Git Bash a single sed call costs ~221ms vs
~12ms for the bash-native equivalent (measured, 20-run average).
Swaps the 8 call sites on the ordinary local send/inbox/history/watch path
(drivers/storage/sqlite.sh _sqlite_lit, send.sh roster-check escape,
inbox.sh/history.sh(x2)/watch.sh/check-inbox.sh JSONL-array escape,
lib/sqlpath.sh agmsg_sql_readfile_path). Does not touch the remote-sync
path (fujibee#780/fujibee#799 already fixed that one).
Measured end-to-end on a real two-agent team, same machine:
send.sh: 10.5-11.8s -> 6.4-7.6s
full send-to-ack round trip: 35-123s -> ~20s
Verified byte-identical to the sed version against 19 edge cases (Thai
text, unicode, embedded quotes/newlines/tabs, SQL injection attempt)
before patching, and live-verified with a real message containing an
apostrophe + Thai text round-tripping correctly between two real agents.
See fujibee#896
|
Thanks for this — and sorry it sat for two days without a reply. The macOS failures have a cause we can name. That is why the three failing shards are all quote cases — We hit the identical thing on a different helper today, tracked as #928, so the We're carrying this forward ourselves so it lands sooner, with your commit credited as co-author. Your scoping was right: the remote-sync path was already handled by #780/#799, and the local send/inbox/history/watch path was the part still spawning a process per escape. |
Fixes #896.
What changed
sed "s/'/''/g"doubles a single quote for SQL literal embedding — atransform simple enough that bash's own
${var//pattern/replacement}does itwithout spawning a process. This swaps the 8 call sites on the ordinary local
send/inbox/history/watch path (the sqlite driver used day-to-day, not the
remote-sync path #780/#799 already fixed) from a
sedsubprocess to pure bashparameter expansion:
Same technique in
send.sh's roster-check escape,inbox.sh/history.sh(x2)/
watch.sh/check-inbox.sh's JSONL-array escape, andlib/sqlpath.sh'sagmsg_sql_readfile_path(thecygpathcall there stays —only the trailing
sedis replaced).Why (measured on Windows 11 / Git Bash, the environment #896 was filed from)
sedstorage_sendalone calls_sqlite_lit6 times (team/from/to/body/id/at)per message, so this one function accounts for roughly 1.3s of the ~10s a
single
send.shcall cost before this change.End-to-end, same machine, same team, before/after this patch (installed
locally and exercised against a real two-agent team, not synthetic):
send.sh <team> a b "<msg>"What this does not fix
history.sh(andcheck-inbox.sh) separately recomputestorage_list_unreadonce per distinct recipient in the displayed slice, toderive the unread ●/○ marker — an N+1 query pattern, unrelated to the
sed→bash swap here, and out of scope for this PR.history.shon a159-row store took ~31s both before and after this patch, unaffected by it.
Happy to file that separately if useful; flagging here so it is not read as
"still slow, so this didn't work."
Correctness
Before touching any live file,
${s//$q/$q$q}was checked byte-for-byteagainst
sed "s/'/''/g"across 19 cases (empty string, leading/trailing/consecutive quotes, Thai text, unicode, embedded newline/tab, a SQL-injection
attempt, a 2000-char string) — all identical. Live-verified after patching by
sending a real message containing both an apostrophe and Thai text between
two real agents on a live team; it round-tripped byte-identical on both ends
(sender's
history.shand the recipient's own read).Testing
Tried running the storage/messaging bats suite locally (
npm install -g bats, Windows 11 / Git Bash) before opening this — it did not finish in areasonable time and produced no output beyond the version banner, which is
consistent with the same per-process spawn cost this PR is about (bats
itself forks a lot per test). Not reporting a pass/fail I did not actually
observe. What I do have is the byte-for-byte equivalence check (19 cases,
above) and the live production round-trip on a real two-agent team, both
before this PR existed. Leaning on CI here rather than guessing at a local
result.