Buffer messages to the frontend until the comm opens - #55
Open
ehigham wants to merge 1 commit into
Open
Conversation
If spark events arrive before the frontend opens the SparkMonitor comm, ScalaMonitor.send raises AttributeError. The exception propagates out of the socket thread's run loop, permanently killing event forwarding for the lifetime of the kernel, even after the frontend connects (swan-cern#7). Buffer messages while the comm is unset and flush them once the frontend opens the comm. All buffer access and sends are serialized with a lock, and the comm is only published after commopen is sent and the buffer is flushed, so a concurrent send can neither overtake the flush nor reach a frontend that has not finished initializing.
ehigham
added a commit
to ehigham/hail
that referenced
this pull request
Aug 19, 2026
Dataproc 3.0.x images ship Spark 4 and manage the default Python environment with pixi rather than conda. Update `hailctl dataproc` and its init action accordingly: - Serve notebooks with the JUPYTER optional component and the component gateway instead of hand-rolling a systemd unit for a tokenless classic-notebook server on port 8123. Notebooks are now JupyterLab, persisted in the cluster's staging bucket. - `hailctl dataproc connect` now opens the requested service's authenticated component gateway url (JupyterLab or the Spark History Server) in the default browser. This replaces the ssh socks tunnel and the hard-coded chrome invocation, and with them the --port flag and the HAILCTL_CHROME environment variables; it also means `connect` requires a gateway-enabled cluster. - Replace hail's jgscm fork (a classic-notebook contents manager, long unmaintained) with google's gcs-jupyter-plugin, which adds a cloud storage browser to JupyterLab. - Replace hail's patched sparkmonitor-0.0.12 wheel (fetched from gs://hail-common) with upstream sparkmonitor 3.3.0, which supports Spark 4 and JupyterLab. A sha-pinned patch fixes a race between spark events and the frontend opening its comm channel; drop it once swan-cern/sparkmonitor#55 is released. - Derive python paths from the interpreter running the init action instead of hard-coding /opt/conda/default: neither the old conda layout nor the new pixi layout is a documented interface. `hailctl dataproc modify` likewise resolves python3 from PATH. - Install libarpack2 on all nodes: Spark 4's dev.ludovic.netlib jni shims bind the system blas, lapack and arpack, and the image ships openblas but no arpack, without which jvm eigensolvers silently fall back to pure-java implementations. - Default machine types move from n1 to n4, and the hardcoded machine-type memory table is replaced by a gcloud lookup, which also handles custom machine types. - Re-enable the dataproc CI test steps. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ehigham
added a commit
to ehigham/hail
that referenced
this pull request
Aug 19, 2026
Dataproc 3.0.x images ship Spark 4 and manage the default Python environment with pixi rather than conda. Update `hailctl dataproc` and its init action accordingly: - Serve notebooks with the JUPYTER optional component and the component gateway instead of hand-rolling a systemd unit for a tokenless classic-notebook server on port 8123. Notebooks are now JupyterLab, persisted in the cluster's staging bucket. - `hailctl dataproc connect` now opens the requested service's authenticated component gateway url (JupyterLab or the Spark History Server) in the default browser. This replaces the ssh socks tunnel and the hard-coded chrome invocation, and with them the --port flag and the HAILCTL_CHROME environment variables; it also means `connect` requires a gateway-enabled cluster. - Replace hail's jgscm fork (a classic-notebook contents manager, long unmaintained) with google's gcs-jupyter-plugin, which adds a cloud storage browser to JupyterLab. - Replace hail's patched sparkmonitor-0.0.12 wheel (fetched from gs://hail-common) with upstream sparkmonitor 3.3.0, which supports Spark 4 and JupyterLab. A sha-pinned patch fixes a race between spark events and the frontend opening its comm channel; drop it once swan-cern/sparkmonitor#55 is released. - Derive python paths from the interpreter running the init action instead of hard-coding /opt/conda/default: neither the old conda layout nor the new pixi layout is a documented interface. `hailctl dataproc modify` likewise resolves python3 from PATH. - Install libarpack2 on all nodes: Spark 4's dev.ludovic.netlib jni shims bind the system blas, lapack and arpack, and the image ships openblas but no arpack, without which jvm eigensolvers silently fall back to pure-java implementations. - Default machine types move from n1 to n4, and the hardcoded machine-type memory table is replaced by a gcloud lookup, which also handles custom machine types. - Re-enable the dataproc CI test steps. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ehigham
added a commit
to ehigham/hail
that referenced
this pull request
Aug 19, 2026
Dataproc 3.0.x images ship Spark 4 and manage the default Python environment with pixi rather than conda. Update `hailctl dataproc` and its init action accordingly: - Serve notebooks with the JUPYTER optional component and the component gateway instead of hand-rolling a systemd unit for a tokenless classic-notebook server on port 8123. Notebooks are now JupyterLab, persisted in the cluster's staging bucket. - `hailctl dataproc connect` now opens the requested service's authenticated component gateway url (JupyterLab or the Spark History Server) in the default browser. This replaces the ssh socks tunnel and the hard-coded chrome invocation, and with them the --port flag and the HAILCTL_CHROME environment variables; it also means `connect` requires a gateway-enabled cluster. - Replace hail's jgscm fork (a classic-notebook contents manager, long unmaintained) with google's gcs-jupyter-plugin, which adds a cloud storage browser to JupyterLab. - Replace hail's patched sparkmonitor-0.0.12 wheel (fetched from gs://hail-common) with upstream sparkmonitor 3.3.0, which supports Spark 4 and JupyterLab. A sha-pinned patch fixes a race between spark events and the frontend opening its comm channel; drop it once swan-cern/sparkmonitor#55 is released. - Derive python paths from the interpreter running the init action instead of hard-coding /opt/conda/default: neither the old conda layout nor the new pixi layout is a documented interface. `hailctl dataproc modify` likewise resolves python3 from PATH. - Install libarpack2 on all nodes: Spark 4's dev.ludovic.netlib jni shims bind the system blas, lapack and arpack, and the image ships openblas but no arpack, without which jvm eigensolvers silently fall back to pure-java implementations. - Default machine types move from n1 to n4, and the hardcoded machine-type memory table is replaced by a gcloud lookup, which also handles custom machine types. - Re-enable the dataproc CI test steps. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ehigham
added a commit
to ehigham/hail
that referenced
this pull request
Aug 19, 2026
Dataproc 3.0.x images ship Spark 4 and manage the default Python environment with pixi rather than conda. Update `hailctl dataproc` and its init action accordingly: - Serve notebooks with the JUPYTER optional component and the component gateway instead of hand-rolling a systemd unit for a tokenless classic-notebook server on port 8123. Notebooks are now JupyterLab, persisted in the cluster's staging bucket. - `hailctl dataproc connect` now opens the requested service's authenticated component gateway url (JupyterLab or the Spark History Server) in the default browser. This replaces the ssh socks tunnel and the hard-coded chrome invocation, and with them the --port flag and the HAILCTL_CHROME environment variables; it also means `connect` requires a gateway-enabled cluster. - Replace hail's jgscm fork (a classic-notebook contents manager, long unmaintained) with google's gcs-jupyter-plugin, which adds a cloud storage browser to JupyterLab. - Replace hail's patched sparkmonitor-0.0.12 wheel (fetched from gs://hail-common) with upstream sparkmonitor 3.3.0, which supports Spark 4 and JupyterLab. A sha-pinned patch fixes a race between spark events and the frontend opening its comm channel; drop it once swan-cern/sparkmonitor#55 is released. - Derive python paths from the interpreter running the init action instead of hard-coding /opt/conda/default: neither the old conda layout nor the new pixi layout is a documented interface. `hailctl dataproc modify` likewise resolves python3 from PATH. - Install libarpack2 on all nodes: Spark 4's dev.ludovic.netlib jni shims bind the system blas, lapack and arpack, and the image ships openblas but no arpack, without which jvm eigensolvers silently fall back to pure-java implementations. - Default machine types move from n1 to n4, and the hardcoded machine-type memory table is replaced by a gcloud lookup, which also handles custom machine types. - Re-enable the dataproc CI test steps. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #7.
If spark events arrive before the frontend opens the SparkMonitor comm,
ScalaMonitor.sendraisesAttributeError, killing event forwarding forthe lifetime of the kernel. See #7 for details and diagnosis.
This change buffers messages while the comm is unset and flushes them once
the frontend opens the comm — the same approach as #32, with the
synchronization concerns from its review addressed:
sendon thesocket thread cannot mutate the buffer while
target_funcis flushing itcommopenis sent and the buffer is flushed beforeself.commispublished (last, under the lock), so a concurrent send can neither
overtake the flush nor reach a frontend that has not finished its comm
initialization
Events from the race window are delivered rather than lost, so the monitor
renders jobs that started before the frontend attached.
Unit tests cover both scenarios raised in the #32 review: a message
arriving before the comm opens, and a message arriving mid-flush.
🤖 Generated with Claude Code