Skip to content

Enable Safari Remote Automation on SIP-enabled macOS 14/15 workers#1152

Open
rcurranmoz wants to merge 18 commits into
masterfrom
sip-compatible-safari-automation
Open

Enable Safari Remote Automation on SIP-enabled macOS 14/15 workers#1152
rcurranmoz wants to merge 18 commits into
masterfrom
sip-compatible-safari-automation

Conversation

@rcurranmoz
Copy link
Copy Markdown
Contributor

  • Replace bash-wrapped osascript with a direct LaunchAgent that runs osascript on an applescript file, avoiding TCC attribution to bash
  • Add safari-enable-remote-automation.applescript and com.mozilla.safari.enableautomation.plist as new deployment artifacts
  • Update macos_safaridriver::init to bootstrap the LaunchAgent into gui/ on Darwin 23/24 and poll for the semaphore file
  • Remove system TCC DB sqlite3 writes from tcc_perms.sh and add_tcc_perms.sh on macOS 14/15 (system DB is SIP-protected/read-only)
  • Add org.mozilla.ci-tcc-pppc.mobileconfig for upload to SimpleMDM as a Custom Configuration Profile to supply system-level TCC grants

aerickson
aerickson previously approved these changes Mar 26, 2026
Copy link
Copy Markdown
Member

@aerickson aerickson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice

@rcurranmoz rcurranmoz force-pushed the sip-compatible-safari-automation branch 8 times, most recently from 8d3fdea to c009881 Compare April 3, 2026 16:56
@rcurranmoz rcurranmoz marked this pull request as ready for review April 3, 2026 17:19
@rcurranmoz rcurranmoz requested a review from a team April 3, 2026 17:20
aerickson
aerickson previously approved these changes Apr 7, 2026
Copy link
Copy Markdown
Member

@aerickson aerickson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just one note about inventory.

Comment thread inventory.d/macmini-m4.yaml Outdated
@rcurranmoz
Copy link
Copy Markdown
Contributor Author

target host: macmini-m4-118.test.releng.mdc1.mozilla.com

rcurranmoz and others added 11 commits May 15, 2026 12:09
- Replace bash-wrapped osascript with a direct LaunchAgent that runs
  osascript on an applescript file, avoiding TCC attribution to bash
- Add safari-enable-remote-automation.applescript and
  com.mozilla.safari.enableautomation.plist as new deployment artifacts
- Update macos_safaridriver::init to bootstrap the LaunchAgent into
  gui/<uid> on Darwin 23/24 and poll for the semaphore file
- Remove system TCC DB sqlite3 writes from tcc_perms.sh and
  add_tcc_perms.sh on macOS 14/15 (system DB is SIP-protected/read-only)
- Add org.mozilla.ci-tcc-pppc.mobileconfig for upload to SimpleMDM as
  a Custom Configuration Profile to supply system-level TCC grants

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
macOS 12+ requires the string key Authorization: Allow rather than
the boolean Allowed: true in PPPC payload entries. The old format
caused ErrorCode 22 (invalid value) when pushed via SimpleMDM.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
StaticCode is not a valid PPPC payload key and was causing ErrorCode 22
on install. Also removed SystemPolicyDesktopFolder to reduce surface area
while iterating on getting the core services to install cleanly.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ScreenCapture is not a valid PPPC service key on macOS 15 — every
entry format tried (Authorization:Allow, Allowed:true) results in
ErrorCode 22. Confirmed via binary search on macmini-m4-184 (15.3).

Working services: SystemPolicyAllFiles, Accessibility, AppleEvents.
ScreenCapture grants for bash/Terminal must be handled separately
(user DB fallback for start-worker already in tcc_perms.sh).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Previously the plist was deployed to ~/Library/LaunchAgents/ which caused
launchd to auto-load the LaunchAgent when cltbld logged in after the first
reboot — before puppet had written the user TCC DB entries. This resulted
in osascript hitting a TCC permission prompt and blocking indefinitely.

Fix: store the plist at /usr/local/lib/ so launchd never auto-loads it.
Only puppet bootstraps it, and only after the perms script succeeds.
Also remove any previously deployed plist from ~/Library/LaunchAgents/.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The applescript creates an empty semaphore file at startup as a lock,
then writes '1' on success. The puppet unless condition only checked
file existence, so a failed run left an empty semaphore that prevented
puppet from ever retrying.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The perms script (add_tcc_perms.sh) was refreshonly, meaning it only ran
when the file changed. On first run the TCC DB didn't exist so it failed;
on subsequent runs the file was unchanged so it never re-ran, leaving
the osascript TCC entries unwritten.

Replace with an unless condition that checks if the osascript AppleEvents
entry is present in cltbld's TCC DB, so it re-runs whenever entries are
missing regardless of whether the file changed.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
BlackHole pkg install fails on fresh macOS 15 machines; comment out
from both m4 and m4-staging roles until the install issue is fixed.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
BlackHole pkg install fails on fresh macOS 15 machines; comment out
from both m4 and m4-staging roles until the install issue is fixed.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The exec command polled with `test -f semaphore` which would return
success immediately if the semaphore file existed but was empty (written
by the applescript at startup before GUI interaction completes). This
caused false success when a prior failed run left an empty semaphore.

Also increase the poll timeout from 60s to 120s to give the applescript
sufficient margin — it has ~50s of programmed delays alone.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
LaunchDaemon-spawned puppet hits "authorization denied" writing
/Users/cltbld/Library/Application Support/com.apple.TCC/TCC.db.
Existing PPPC FDA grants cover sshd/sshd-keygen-wrapper only, so the
chain works for SSH-driven puppet (inherits sshd's TCC context) but
not for launchd-driven puppet (which is what a SimpleMDM script-job
bootstrap needs). Adding bash and sqlite3 lets the user-DB sqlite3
writes in add_tcc_perms.sh and tcc_perms.sh succeed regardless of
the invoking process tree.

Validated on macmini-m4-118 2026-05-15: SSH-driven puppet succeeded
end-to-end, but LaunchDaemon-driven puppet from the SimpleMDM
bootstrap script failed at the user-DB write step before this change.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@rcurranmoz rcurranmoz force-pushed the sip-compatible-safari-automation branch from 64d8ba5 to 4c41b33 Compare May 15, 2026 16:10
rcurranmoz added a commit that referenced this pull request May 15, 2026
…emaphore write

The old applescript would write the success semaphore even if the GUI
checkbox toggle silently failed (osascript-can't-read-value-back, or
some race in Safari's Settings tab population). A bootstrapped host
could land with safaridriver effectively disabled while looking "green"
to every automated check — and we only catch it when somebody opens
Safari in VNC and eyeballs the menu. Which is exactly how we lost a
chunk of today.

After clicking each of the two toggles ("Show features for web developers"
and "Allow remote automation"), the script now reads the checkbox value
back and `error`s out if it isn't 1. The script's only path to writing
the success semaphore is a clean run through both clicks, so a silent
skip now surfaces as a puppet exec failure and the SimpleMDM driver's
outer retry loop keeps going instead of declaring victory.

---

CLAUDE HANDOFF — PR #1152 + SimpleMDM bootstrap, pick up next week

Hello! If you're reading this, you have inherited the SIP-compatible
Safari Remote Automation saga. Here's where today (2026-05-15) left
things and what to chase next. Buckle up.

THE BIG PICTURE
PR #1152 ("sip-compatible-safari-automation") replaces the bash-wrapped
osascript Safari-enable dance with a LaunchAgent that runs osascript
directly, plus a SimpleMDM-delivered PPPC mobileconfig for system-level
TCC grants. End goal: zero-touch DEP enrollment → puppet → worker live
in TC, on SIP-enabled macOS 15.3 M4 Mac minis, without the human ever
SSHing in.

WHAT GOT VALIDATED TODAY ON macmini-m4-118 (gecko_t_osx_1500_m4_staging)
- In-place pivot from existing puppet state to PR branch: green
- Fresh post-EACS + admin VNC + SSH-driven bootstrap: green
- Fresh post-EACS + admin VNC + SimpleMDM-driven bootstrap (via the
  ~/Desktop/m4-simplemdm-bootstrap-sip-safari.sh one-shot script):
  bootstrap completed, BUT the applescript silently skipped the
  "Allow remote automation" toggle on the first MDM-driven run.
  Hence this commit. We then manually re-ran the applescript and it
  landed visually correctly. The bug that hid this from us all day
  was the semaphore-on-success-without-actual-success behavior.
- Bootstrap Token custody stays on admin (cltbld DISABLED): confirmed
  via sysadminctl -secureTokenStatus + APFS crypto users listing.
  The bootstrap script's admin-BST gate (sysadminctl poll, 30 min cap)
  works as intended.
- PPPC FDA grants for /bin/bash + /usr/bin/sqlite3 land on the host
  (verify via /Library/Managed Preferences/com.apple.TCC.configuration-
  profile-policy.plist).

THE FOOTGUNS I LEFT IN THE SAND FOR YOU
1. PPPC grants for sshd/sshd-keygen-wrapper alone are NOT enough for a
   LaunchDaemon-spawned puppet to write cltbld's user TCC DB. We hit
   "authorization denied" on user-DB sqlite3 writes when the driver
   ran puppet directly. Workaround: the bootstrap driver invokes
   run-puppet.sh via `ssh root@localhost` so puppet inherits sshd's
   user-session TCC domain. One-shot ed25519 key at
   /var/root/.ssh/bootstrap_id_ed25519 during bootstrap, removed on
   sentinel write. This lives in the Desktop bootstrap script — when
   we evolve this, that workaround should probably move into a real
   puppet/PR change.
2. `launchctl asuser 0 <cmd>` does NOT actually escape the system
   domain when invoked from a LaunchDaemon. It only transitions when
   the caller is already in a user-session domain (sshd-spawned, GUI
   login). I went down this path for an embarrassing amount of time
   today before pivoting to ssh-to-localhost. Save yourself.
3. LaunchDaemon RunAtLoad fires before DNS is up at boot. First puppet
   tick on a freshly-rebooted host died on `git fetch`'s DNS lookup and
   our one-shot LaunchDaemon never retried. Driver now waits on
   dscacheutil resolving github.com before doing anything, with an
   outer while-loop retrying run-puppet.sh until both semaphores land.

ARTIFACTS WORTH BOOKMARKING
- ~/Desktop/m4-simplemdm-bootstrap-sip-safari.sh — the SimpleMDM script.
  Upload to SimpleMDM as a script-job, run-on-enrollment, scoped to the
  M4 staging group. Idempotent, self-cleans on success.
- ~/Desktop/org.mozilla.ci-tcc-pppc.mobileconfig — extracted from PR
  branch; upload to SimpleMDM as a Custom Configuration Profile,
  replacing the previous "Mozilla CI TCC Permissions". Has the new
  bash + sqlite3 FDA grants this branch added.
- modules/macos_mobileconfig_profiles/files/org.mozilla.ci-tcc-pppc.mobileconfig
  — source of truth in-tree.
- modules/macos_safaridriver/files/safari-enable-remote-automation.applescript
  — this commit's change lives here.
- ~/Desktop/m4-provisioning-handoff.md — earlier handoff from the M4
  rollout, the procedural shape still applies.

REMAINING WORK FOR ACTUAL ZERO-TOUCH
- vault.yaml delivery. Still a manual scp/drop step. Bootstrap script
  polls /var/root/vault.yaml up to 10 min. Open question: secure-fetch
  via an internal endpoint? SimpleMDM Custom Attribute with the file
  body? Conversation worth having with whoever runs SimpleMDM admin.
- admin VNC login is still required to plant the Bootstrap Token before
  puppet flips autologin to cltbld. No known mechanism to script a
  GUI/console login during DEP / Setup Assistant. If you find one,
  you become a legend.
- start-worker / generic-worker-multiuser need Developer ID signing
  to be coverable by the PPPC profile (ScreenCapture specifically).
  start-worker is currently a user-DB fallback entry in tcc_perms.sh.

VERIFICATION METHOD (per Ryan, after a long debug detour today)
Forget `defaults read com.apple.Safari AllowRemoteAutomation`. The pref
doesn't live there in any way our tooling can read. The gold standard
is opening Safari in cltbld's VNC session and visually confirming:
  - Develop menu is in the menu bar
  - Safari > Settings > Developer > "Allow remote automation" is checked
This is the dance the RelOps team has been doing since macOS 10.15.

RECOMMENDED REORIENTATION PLAYBOOK
  1. Read: this commit + the mobileconfig + the Desktop bootstrap script
  2. EACS macmini-m4-118 (or any spare staging M4)
  3. Admin VNC login → drop vault.yaml → trigger the SimpleMDM script
  4. Once bootstrap completes (sentinel /var/log/m4-bootstrap-complete),
     eyeball Safari > Settings > Developer in cltbld's VNC session
  5. If anything is weird: /var/log/m4-bootstrap.log (one-shot setup)
     and /var/log/m4-bootstrap-driver.log (boot-by-boot puppet) are
     your friends. ssh keys and LaunchDaemon plists self-clean on
     success; if they're still on disk, the driver hasn't declared
     victory yet.

Memory notes for ~/.claude/projects/-Users-rcurran/memory/ are already
updated for: [[apple-silicon-bootstrap-token]] (the BST custody fix),
[[sip-safari-automation]] (the PR's overall shape and validated state).

Good luck. May your AppleScript clicks actually persist.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
rcurranmoz added a commit that referenced this pull request May 15, 2026
…emaphore write

The old applescript would write the success semaphore even if the GUI
checkbox toggle silently failed (osascript-can't-read-value-back, or
some race in Safari's Settings tab population). A bootstrapped host
could land with safaridriver effectively disabled while looking "green"
to every automated check — and we only catch it when somebody opens
Safari in VNC and eyeballs the menu. Which is exactly how we lost a
chunk of today.

After clicking each of the two toggles ("Show features for web developers"
and "Allow remote automation"), the script now reads the checkbox value
back and `error`s out if it isn't 1. The script's only path to writing
the success semaphore is a clean run through both clicks, so a silent
skip now surfaces as a puppet exec failure and the SimpleMDM driver's
outer retry loop keeps going instead of declaring victory.

---

🎁 CLAUDE HANDOFF — PR #1152 + SimpleMDM bootstrap, pick up next week 🎁

Hello! 👋 If you're reading this, you have inherited the SIP-compatible
Safari Remote Automation saga. Here's where today (2026-05-15) left
things and what to chase next. Buckle up. 🎢

🗺️ THE BIG PICTURE
PR #1152 ("sip-compatible-safari-automation") replaces the bash-wrapped
osascript Safari-enable dance with a LaunchAgent that runs osascript
directly, plus a SimpleMDM-delivered PPPC mobileconfig for system-level
TCC grants. End goal: zero-touch DEP enrollment → puppet → worker live
in TC, on SIP-enabled macOS 15.3 M4 Mac minis, without the human ever
SSHing in. ✨

✅ WHAT GOT VALIDATED ON macmini-m4-118 (gecko_t_osx_1500_m4_staging)
- 🟢 In-place pivot from existing puppet state to PR branch
- 🟢 Fresh post-EACS + admin VNC + SSH-driven bootstrap
- 🟡 Fresh post-EACS + admin VNC + SimpleMDM-driven bootstrap (via the
  ~/Desktop/m4-simplemdm-bootstrap-sip-safari.sh one-shot script):
  bootstrap completed, BUT the applescript silently skipped the
  "Allow remote automation" toggle on the first MDM-driven run.
  Hence this commit. We then manually re-ran the applescript and it
  landed visually correctly. The bug that hid this from us all day
  was the semaphore-on-success-without-actual-success behavior. 🫠
- 🔐 Bootstrap Token custody stays on admin (cltbld DISABLED):
  confirmed via sysadminctl -secureTokenStatus + APFS crypto users.
  The bootstrap script's admin-BST gate (sysadminctl poll, 30 min cap)
  works as intended.
- 🪪 PPPC FDA grants for /bin/bash + /usr/bin/sqlite3 land on the host
  (verify via /Library/Managed Preferences/com.apple.TCC.configuration-
  profile-policy.plist).

🪤 THE FOOTGUNS I LEFT IN THE SAND FOR YOU
1. 🔒 PPPC grants for sshd/sshd-keygen-wrapper alone are NOT enough for a
   LaunchDaemon-spawned puppet to write cltbld's user TCC DB. We hit
   "authorization denied" on user-DB sqlite3 writes when the driver
   ran puppet directly. Workaround: the bootstrap driver invokes
   run-puppet.sh via \`ssh root@localhost\` so puppet inherits sshd's
   user-session TCC domain. One-shot ed25519 key at
   /var/root/.ssh/bootstrap_id_ed25519 during bootstrap, removed on
   sentinel write. This lives in the Desktop bootstrap script — when
   we evolve this, that workaround should probably move into a real
   puppet/PR change.
2. 🐚 \`launchctl asuser 0 <cmd>\` does NOT actually escape the system
   domain when invoked from a LaunchDaemon. It only transitions when
   the caller is already in a user-session domain (sshd-spawned, GUI
   login). I went down this path for an embarrassing amount of time
   today before pivoting to ssh-to-localhost. Save yourself. 😅
3. 🌐 LaunchDaemon RunAtLoad fires before DNS is up at boot. First puppet
   tick on a freshly-rebooted host died on \`git fetch\`'s DNS lookup and
   our one-shot LaunchDaemon never retried. Driver now waits on
   dscacheutil resolving github.com before doing anything, with an
   outer while-loop retrying run-puppet.sh until both semaphores land.

📦 ARTIFACTS WORTH BOOKMARKING
- ~/Desktop/m4-simplemdm-bootstrap-sip-safari.sh — the SimpleMDM script.
  Upload to SimpleMDM as a script-job, run-on-enrollment, scoped to the
  M4 staging group. Idempotent, self-cleans on success.
- ~/Desktop/org.mozilla.ci-tcc-pppc.mobileconfig — extracted from PR
  branch; upload to SimpleMDM as a Custom Configuration Profile,
  replacing the previous "Mozilla CI TCC Permissions". Has the new
  bash + sqlite3 FDA grants this branch added.
- modules/macos_mobileconfig_profiles/files/org.mozilla.ci-tcc-pppc.mobileconfig
  — source of truth in-tree.
- modules/macos_safaridriver/files/safari-enable-remote-automation.applescript
  — this commit's change lives here.
- ~/Desktop/m4-provisioning-handoff.md — earlier handoff from the M4
  rollout, the procedural shape still applies.

🚧 REMAINING WORK FOR ACTUAL ZERO-TOUCH
- 🔑 vault.yaml delivery. Still a manual scp/drop step. Bootstrap script
  polls /var/root/vault.yaml up to 10 min. Open question: secure-fetch
  via an internal endpoint? SimpleMDM Custom Attribute with the file
  body? Conversation worth having with whoever runs SimpleMDM admin.
- 👤 admin VNC login is still required to plant the Bootstrap Token
  before puppet flips autologin to cltbld. No known mechanism to script
  a GUI/console login during DEP / Setup Assistant. If you find one,
  you become a legend. 🏆
- 🖋️ start-worker / generic-worker-multiuser need Developer ID signing
  to be coverable by the PPPC profile (ScreenCapture specifically).
  start-worker is currently a user-DB fallback entry in tcc_perms.sh.

🔬 VERIFICATION METHOD (per Ryan, after a long debug detour today)
Forget \`defaults read com.apple.Safari AllowRemoteAutomation\`. The pref
doesn't live there in any way our tooling can read. The gold standard
is opening Safari in cltbld's VNC session and visually confirming:
  - Develop menu is in the menu bar
  - Safari > Settings > Developer > "Allow remote automation" is checked
This is the dance the RelOps team has been doing since macOS 10.15. 🕺

🚀 RECOMMENDED REORIENTATION PLAYBOOK
  1. Read: this commit + the mobileconfig + the Desktop bootstrap script
  2. EACS macmini-m4-118 (or any spare staging M4)
  3. Admin VNC login → drop vault.yaml → trigger the SimpleMDM script
  4. Once bootstrap completes (sentinel /var/log/m4-bootstrap-complete),
     eyeball Safari > Settings > Developer in cltbld's VNC session
  5. If anything is weird: /var/log/m4-bootstrap.log (one-shot setup)
     and /var/log/m4-bootstrap-driver.log (boot-by-boot puppet) are
     your friends 🪵. ssh keys and LaunchDaemon plists self-clean on
     success; if they're still on disk, the driver hasn't declared
     victory yet.

🧠 Memory notes for ~/.claude/projects/-Users-rcurran/memory/ are already
updated for: [[apple-silicon-bootstrap-token]] (the BST custody fix),
[[sip-safari-automation]] (the PR's overall shape and validated state).

Good luck. May your AppleScript clicks actually persist. 🍀

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@rcurranmoz rcurranmoz force-pushed the sip-compatible-safari-automation branch from 4bf1b8b to 3678adb Compare May 15, 2026 18:50
…emaphore write

The old applescript would write the success semaphore even if the GUI
checkbox toggle silently failed (osascript-can't-read-value-back, or
some race in Safari's Settings tab population). A bootstrapped host
could land with safaridriver effectively disabled while looking "green"
to every automated check — and we only catch it when somebody opens
Safari in VNC and eyeballs the menu. Which is exactly how we lost a
chunk of today.

After clicking each of the two toggles ("Show features for web developers"
and "Allow remote automation"), the script now reads the checkbox value
back and `error`s out if it isn't 1. The script's only path to writing
the success semaphore is a clean run through both clicks, so a silent
skip now surfaces as a puppet exec failure and the SimpleMDM driver's
outer retry loop keeps going instead of declaring victory.

---

🎁 CLAUDE HANDOFF — PR #1152 + SimpleMDM bootstrap, pick up next week 🎁

Hello! 👋 If you're reading this, you have inherited the SIP-compatible
Safari Remote Automation saga. Here's where today (2026-05-15) left
things and what to chase next. Buckle up. 🎢

🗺️ THE BIG PICTURE
PR #1152 ("sip-compatible-safari-automation") replaces the bash-wrapped
osascript Safari-enable dance with a LaunchAgent that runs osascript
directly, plus a SimpleMDM-delivered PPPC mobileconfig for system-level
TCC grants. End goal: zero-touch DEP enrollment → puppet → worker live
in TC, on SIP-enabled macOS 15.3 M4 Mac minis, without the human ever
SSHing in. ✨

✅ WHAT GOT VALIDATED ON macmini-m4-118 (gecko_t_osx_1500_m4_staging)
- 🟢 In-place pivot from existing puppet state to PR branch
- 🟢 Fresh post-EACS + admin VNC + SSH-driven bootstrap
- 🟡 Fresh post-EACS + admin VNC + SimpleMDM-driven bootstrap (via the
  ~/Desktop/m4-simplemdm-bootstrap-sip-safari.sh one-shot script):
  bootstrap completed, BUT the applescript silently skipped the
  "Allow remote automation" toggle on the first MDM-driven run.
  Hence this commit. We then manually re-ran the applescript and it
  landed visually correctly. The bug that hid this from us all day
  was the semaphore-on-success-without-actual-success behavior. 🫠
- 🔐 Bootstrap Token custody stays on admin (cltbld DISABLED):
  confirmed via sysadminctl -secureTokenStatus + APFS crypto users.
  The bootstrap script's admin-BST gate (sysadminctl poll, 30 min cap)
  works as intended.
- 🪪 PPPC FDA grants for /bin/bash + /usr/bin/sqlite3 land on the host
  (verify via /Library/Managed Preferences/com.apple.TCC.configuration-
  profile-policy.plist).

🪤 THE FOOTGUNS I LEFT IN THE SAND FOR YOU
1. 🔒 PPPC grants for sshd/sshd-keygen-wrapper alone are NOT enough for a
   LaunchDaemon-spawned puppet to write cltbld's user TCC DB. We hit
   "authorization denied" on user-DB sqlite3 writes when the driver
   ran puppet directly. Workaround: the bootstrap driver invokes
   run-puppet.sh via \`ssh root@localhost\` so puppet inherits sshd's
   user-session TCC domain. One-shot ed25519 key at
   /var/root/.ssh/bootstrap_id_ed25519 during bootstrap, removed on
   sentinel write. This lives in the Desktop bootstrap script — when
   we evolve this, that workaround should probably move into a real
   puppet/PR change.
2. 🐚 \`launchctl asuser 0 <cmd>\` does NOT actually escape the system
   domain when invoked from a LaunchDaemon. It only transitions when
   the caller is already in a user-session domain (sshd-spawned, GUI
   login). I went down this path for an embarrassing amount of time
   today before pivoting to ssh-to-localhost. Save yourself. 😅
3. 🌐 LaunchDaemon RunAtLoad fires before DNS is up at boot. First puppet
   tick on a freshly-rebooted host died on \`git fetch\`'s DNS lookup and
   our one-shot LaunchDaemon never retried. Driver now waits on
   dscacheutil resolving github.com before doing anything, with an
   outer while-loop retrying run-puppet.sh until both semaphores land.

📦 ARTIFACTS WORTH BOOKMARKING
- ~/Desktop/m4-simplemdm-bootstrap-sip-safari.sh — the SimpleMDM script.
  Upload to SimpleMDM as a script-job, run-on-enrollment, scoped to the
  M4 staging group. Idempotent, self-cleans on success.
- ~/Desktop/org.mozilla.ci-tcc-pppc.mobileconfig — extracted from PR
  branch; upload to SimpleMDM as a Custom Configuration Profile,
  replacing the previous "Mozilla CI TCC Permissions". Has the new
  bash + sqlite3 FDA grants this branch added.
- modules/macos_mobileconfig_profiles/files/org.mozilla.ci-tcc-pppc.mobileconfig
  — source of truth in-tree.
- modules/macos_safaridriver/files/safari-enable-remote-automation.applescript
  — this commit's change lives here.
- ~/Desktop/m4-provisioning-handoff.md — earlier handoff from the M4
  rollout, the procedural shape still applies.

🚧 REMAINING WORK FOR ACTUAL ZERO-TOUCH
- 🔑 vault.yaml delivery. Still a manual scp/drop step. Bootstrap script
  polls /var/root/vault.yaml up to 10 min. Open question: secure-fetch
  via an internal endpoint? SimpleMDM Custom Attribute with the file
  body? Conversation worth having with whoever runs SimpleMDM admin.
- 👤 admin VNC login is still required to plant the Bootstrap Token
  before puppet flips autologin to cltbld. No known mechanism to script
  a GUI/console login during DEP / Setup Assistant. If you find one,
  you become a legend. 🏆
- 🖋️ start-worker / generic-worker-multiuser need Developer ID signing
  to be coverable by the PPPC profile (ScreenCapture specifically).
  start-worker is currently a user-DB fallback entry in tcc_perms.sh.

🔬 VERIFICATION METHOD (per Ryan, after a long debug detour today)
Forget \`defaults read com.apple.Safari AllowRemoteAutomation\`. The pref
doesn't live there in any way our tooling can read. The gold standard
is opening Safari in cltbld's VNC session and visually confirming:
  - Develop menu is in the menu bar
  - Safari > Settings > Developer > "Allow remote automation" is checked
This is the dance the RelOps team has been doing since macOS 10.15. 🕺

🚀 RECOMMENDED REORIENTATION PLAYBOOK
  1. Read: this commit + the mobileconfig + the Desktop bootstrap script
  2. EACS macmini-m4-118 (or any spare staging M4)
  3. Admin VNC login → drop vault.yaml → trigger the SimpleMDM script
  4. Once bootstrap completes (sentinel /var/log/m4-bootstrap-complete),
     eyeball Safari > Settings > Developer in cltbld's VNC session
  5. If anything is weird: /var/log/m4-bootstrap.log (one-shot setup)
     and /var/log/m4-bootstrap-driver.log (boot-by-boot puppet) are
     your friends 🪵. ssh keys and LaunchDaemon plists self-clean on
     success; if they're still on disk, the driver hasn't declared
     victory yet.

🧠 Memory notes for ~/.claude/projects/-Users-rcurran/memory/ are already
updated for: [[apple-silicon-bootstrap-token]] (the BST custody fix),
[[sip-safari-automation]] (the PR's overall shape and validated state).

Good luck. May your AppleScript clicks actually persist. 🍀

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@rcurranmoz rcurranmoz force-pushed the sip-compatible-safari-automation branch from 3678adb to 85876cc Compare May 15, 2026 18:54
rcurranmoz and others added 6 commits May 15, 2026 15:13
Post-EACS-#2 the SimpleMDM-driven bootstrap landed clean — Develop menu
visible and "Allow remote automation" checked, confirmed visually in
cltbld's VNC session. That validates the verify-after-click commit
(4bf1b8b/85876cc5) end-to-end through the LaunchDaemon → ssh-to-localhost
→ puppet → LaunchAgent → osascript chain.

Also flagging the one human-touch needed on this round (manual
/var/tmp/semaphore/run-buildbot), which is already fixed in the Desktop
bootstrap script — just hadn't been re-uploaded to SimpleMDM between
iterations.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Link long-standing taskcluster/taskcluster#7413 for the start-worker /
  generic-worker-multiuser Developer ID signing gap.
- Trim the "Forget defaults read AllowRemoteAutomation" framing on the
  verification section — that was overstated; visual VNC check is still
  the gold standard but no need to bury the defaults command.
- Tighten the vault.yaml delivery note (Ryan is the SimpleMDM admin —
  the decision-maker is the same human who picks this up).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The handoff was referencing ~/Desktop/ paths on my laptop, which is
useless to anyone else picking this up. Move the SimpleMDM bootstrap
script into the repo at
modules/macos_safaridriver/simplemdm-bootstrap-sip-safari.sh
so it lives next to the puppet module it bootstraps onto, and rewrite
the HANDOFF artifact table to point at in-tree relative paths only.

Also drop the "Related memory notes" section — those pointers were to
my local Claude memory dir, equally useless to someone else.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The footnote claimed SimpleMDM had a pre-touch version of the bootstrap
script during EACS#2; per Ryan, the latest script was in fact uploaded.
Dropping the inaccurate claim. The in-tree script
(`modules/macos_safaridriver/simplemdm-bootstrap-sip-safari.sh`) has
the `touch /var/tmp/semaphore/run-buildbot` baked into the driver
cleanup block (line 206), so the next EACS should bring the worker
online on its own.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…eview

Safari Technology Preview has its own separate "Allow Remote Automation"
toggle that the stable Safari enable flow does NOT flip. Without this,
test-macosx1500-aarch64-shippable/opt-browsertime-benchmark-safari-tp-*
tasks fail with "raptor-browsertime Critical: No data to collect" — the
browsertime webdriver session can't drive Safari TP.

Mirrors the stable Safari LaunchAgent + applescript pattern:

  - files/safari-tp-enable-remote-automation.applescript
      Drives "Safari Technology Preview" through the same Settings ->
      Advanced -> Show features for web developers -> Developer ->
      Allow remote automation dance, with the same verify-after-click
      defensive gate before writing its semaphore.
  - files/com.mozilla.safari-tp.enableautomation.plist
      LaunchAgent that runs the TP applescript via osascript in cltbld's
      gui/<uid> session.
  - manifests/init.pp
      New `execute enable remote automation script (Safari TP)` exec
      bootstrapped alongside the existing stable Safari exec on macOS
      14/15. Gated on `/Applications/Safari Technology Preview.app`
      existing via `onlyif` so a host without Safari TP installed yet
      (e.g. mid-EACS, before packages::safari_preview lands) becomes a
      no-op rather than a hard puppet failure — the next puppet apply
      picks it up once the .app dir is in place.

Semaphore: /Users/cltbld/Library/Preferences/semaphore/safari-tech-preview-enable-remote-automation-has-run
(name matches the legacy pre-PR convention so any older tooling that
looked for it keeps working.)

The system-wide `safaridriver --enable` already covers both Safari and
Safari TP — no additional exec needed for that side.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Safari TP applescript + LaunchAgent landed in b36069a, and the
follow-up try push (revision 0502d2099c8a) confirms 6 of 7 safari-*
benchmark tasks pass on m4-118 — including both safari-tp-speedometer3
and safari-tp-jetstream3, which were failing for "No data to collect"
on prior pushes when the TP applescript didn't exist yet. Update the
HANDOFF to:

- Move the SimpleMDM-driven bootstrap row to unambiguously green and
  drop the "(2nd EACS round)" qualifier — it's the steady state now.
- Add a try-push validation row + a dedicated Safari TP row pointing
  at the proof-of-life push.
- Add the two new safari-tp artifacts (applescript + plist) to the
  artifacts table, and call out the matching plist for stable Safari.
- Generalize the verification method to call out BOTH Safari and
  Safari Technology Preview, and link the verified try push as a
  backup signal that's faster than a VNC eyeball.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants