Consolidate server state onto a single server-context struct

CLAUDE.md

@@ -53,16 +53,17 @@ To add a new test:
 ### Communication Flow
 1. Editor sends JSON-RPC requests via stdio
 2. `read-lsp-message` (jsonrpc/messages.lisp) parses HTTP-like headers + JSON body
-3. Requests dispatch to handlers registered in `*handlers*` hash table
-4. Handlers access global state (`*documents*`, `*workspace-root*`, symbol tables)
+3. Requests dispatch to handlers registered on the server context (`ctx:handlers`)
+4. Handlers access shared state through `clef-context` accessors (`ctx:documents`, `ctx:workspace-root`, symbol tables, ...)
 5. Responses convert to JSON-RPC and write to stdout
 
 ### Key Source Modules (src/)
 
 | Module | Purpose |
 |--------|---------|
+| `context.lisp` | Central `server-context` struct + `*server*` — all persistent state lives here |
 | `jsonrpc/` | JSON-RPC protocol implementation |
-| `lsp/server.lisp` | Main server loop, handler registration, state management |
+| `lsp/server.lisp` | Main server loop, handler dispatch |
 | `lsp/lifecycle/` | Initialize/initialized/shutdown handlers |
 | `lsp/document/` | Document handlers (completion, definition, hover, formatting, diagnostics) |
 | `lsp/workspace/` | Workspace-level handlers |
@@ -73,15 +74,32 @@ To add a new test:
 | `packages.lisp` | Package definitions and namespace exports |
 | `main.lisp` | Entry point (`clef-root:start-server`) |
 
-### Global State Variables
-
-- `*documents*` - Hash table of open files (path → full text)
-- `*handlers*` - Hash table mapping LSP methods to handler functions
-- `*lexical-scopes-by-file*` - Maps files to interval trees of lexical scopes
-- `*symbol-refs-by-file*` - Maps files to symbol references with location info
-- `*workspace-root*` - Root directory of project
-- `*client-capabilities*` - What the client editor supports
-- `*initialized*` - Boolean for LSP lifecycle state
+### Server Context (`clef-context`)
+
+All persistent server state lives on a single `server-context` struct held in
+`clef-context:*server*`. Short symbol-macro aliases (`ctx:documents`,
+`ctx:workspace-root`, `ctx:handlers`, ...) expand to struct-accessor reads on
+`*server*`, so call sites read and write them as if they were ordinary
+variables, including with `setf`.
+
+Fields on the context include:
+
+- `ctx:documents` — hash table of open files (URI → full text)
+- `ctx:handlers` — hash table mapping LSP methods to handler functions
+- `ctx:workspace-root` — project workspace root URI
+- `ctx:client-capabilities` — client capabilities reported at initialize time
+- `ctx:initialized` / `ctx:shutdown-received` — lifecycle flags
+- `ctx:output-stream` — stream for outbound LSP notifications
+- `ctx:lexical-scopes` / `ctx:symbol-refs` — per-file interval trees
+- `ctx:workspace-symbol-index` — cross-file symbol lookup table
+- `ctx:document-line-offsets` — per-file byte offset caches
+- `ctx:global-scope` — root lexical-scope (builtins + external packages)
+- `ctx:loaded-systems` / `ctx:file-to-system` / `ctx:asd-files` — ASDF state
+
+Shutdown and exit handlers call `ctx:reset-context` to atomically replace
+`*server*` with a fresh context, which also gives tests a clean slate between
+runs. No CLEF package should define its own mutable `defparameter` for
+server state — put new fields on the struct in `src/context.lisp` instead.
 
 ### Symbol Resolution
 

clef.asd

@@ -19,6 +19,7 @@
                                  :components ((:file "packages")
                                               (:file "util")
                                               (:file "log")
+                                              (:file "context")
                                               (:file "jsonrpc/types")
                                               (:file "jsonrpc/messages")
                                               (:file "parser/parser")

src/context.lisp

@@ -0,0 +1,132 @@
+(in-package :clef-context)
+
+;;; Central server context.
+;;;
+;;; Historically CLEF scattered its mutable state across many defparameters in
+;;; several packages (clef-lsp/server, clef-symbols, clef-lsp/lifecycle). That
+;;; made it hard to reset between test runs, impossible to host more than one
+;;; server in a single image, and awkward when new handlers needed to reach
+;;; across module boundaries to touch state.
+;;;
+;;; This file consolidates everything onto a single SERVER-CONTEXT struct held
+;;; in the special variable *SERVER*. Each field corresponds to one of the old
+;;; globals. Short symbol-macro aliases expand to field access on *SERVER*,
+;;; so call sites look like ordinary variable references:
+;;;
+;;;     (gethash uri (documents))          ; reads (server-context-documents *server*)
+;;;     (setf (workspace-root) path)       ; writes (server-context-workspace-root *server*)
+;;;
+;;; Tests and shutdown handlers can swap *SERVER* with a fresh context to reset
+;;; all state atomically.
+
+(defstruct server-context
+  "All persistent state for a running CLEF LSP server.
+
+Fields are grouped by subsystem:
+
+Lifecycle:
+  initialized          -- set once the client sends the `initialized' notification
+  shutdown-received    -- set when the client sends `shutdown'; used by `exit'
+  client-capabilities  -- hash table of capabilities from `initialize'
+  workspace-root       -- URI of the project root
+  output-stream        -- stream for outbound LSP messages (notifications)
+
+Handlers & documents:
+  handlers             -- method-name -> handler-function
+  documents            -- document URI -> full text string
+
+Symbol analysis (formerly in clef-symbols):
+  lexical-scopes       -- file path -> interval tree of lexical-scope's
+  symbol-refs          -- file path -> interval tree of symbol-reference's
+  workspace-symbol-index -- symbol-name -> list of symbol-definitions (cross-file)
+  document-line-offsets  -- file path -> vector of per-line byte offsets
+  global-scope         -- root lexical-scope holding builtins + externals
+
+ASDF systems (formerly in clef-lsp/lifecycle):
+  loaded-systems       -- system name -> system-info
+  file-to-system       -- absolute file path -> owning system name
+  asd-files            -- list of all discovered .asd pathnames"
+  (initialized nil :type boolean)
+  (shutdown-received nil :type boolean)
+  (client-capabilities nil)
+  (workspace-root nil)
+  (output-stream nil)
+  (handlers (make-hash-table :test 'equal))
+  (documents (make-hash-table :test 'equal))
+  (lexical-scopes (make-hash-table :test 'equal))
+  (symbol-refs (make-hash-table :test 'equal))
+  (workspace-symbol-index (make-hash-table :test 'equal))
+  (document-line-offsets (make-hash-table :test 'equal))
+  (global-scope nil)
+  (loaded-systems (make-hash-table :test 'equal))
+  (file-to-system (make-hash-table :test 'equal))
+  (asd-files nil))
+
+(defparameter *server* (make-server-context)
+  "The current CLEF LSP server context.
+
+Bound to a freshly constructed SERVER-CONTEXT at image load, replaced on
+RESET-CONTEXT, and rebindable with LET for isolated tests.")
+
+(defun reset-context ()
+  "Replace *SERVER* with a fresh context, discarding all server state.
+Called from shutdown and exit handlers and from test setup."
+  (setf *server* (make-server-context)))
+
+;;; Symbol-macro aliases.
+;;;
+;;; These let callers write `(documents)' or `(setf (workspace-root) x)' as
+;;; if they were plain functions/places, without the noise of threading
+;;; *server* through every call. Setf works transparently because each
+;;; expansion bottoms out in a struct accessor, which has a real setf
+;;; expander.
+
+(defmacro define-context-accessor (short-name field-accessor docstring)
+  "Define SHORT-NAME as a symbol-macro that reads/writes (FIELD-ACCESSOR *SERVER*)."
+  (declare (ignore docstring))
+  `(define-symbol-macro ,short-name (,field-accessor *server*)))
+
+(define-context-accessor initialized server-context-initialized
+  "Whether the client has confirmed initialization.")
+
+(define-context-accessor shutdown-received server-context-shutdown-received
+  "Whether shutdown has been requested.")
+
+(define-context-accessor client-capabilities server-context-client-capabilities
+  "Client capabilities reported at initialize time.")
+
+(define-context-accessor workspace-root server-context-workspace-root
+  "Project workspace root URI.")
+
+(define-context-accessor output-stream server-context-output-stream
+  "Outbound LSP message stream.")
+
+(define-context-accessor handlers server-context-handlers
+  "Method name -> handler function.")
+
+(define-context-accessor documents server-context-documents
+  "Document URI -> current text.")
+
+(define-context-accessor lexical-scopes server-context-lexical-scopes
+  "File path -> interval tree of lexical scopes.")
+
+(define-context-accessor symbol-refs server-context-symbol-refs
+  "File path -> interval tree of symbol references.")
+
+(define-context-accessor workspace-symbol-index server-context-workspace-symbol-index
+  "Symbol name -> list of symbol-definitions, across all files.")
+
+(define-context-accessor document-line-offsets server-context-document-line-offsets
+  "File path -> vector of byte offsets for each line.")
+
+(define-context-accessor global-scope server-context-global-scope
+  "Root lexical-scope for the workspace (holds builtins + externals).")
+
+(define-context-accessor loaded-systems server-context-loaded-systems
+  "System name -> system-info for ASDF systems discovered in the workspace.")
+
+(define-context-accessor file-to-system server-context-file-to-system
+  "Absolute file path -> system name that owns it.")
+
+(define-context-accessor asd-files server-context-asd-files
+  "List of every .asd file discovered in the workspace.")

src/lsp/document/diagnostic.lisp

@@ -58,7 +58,7 @@
        (let* ((document-uri (href (clef-jsonrpc/types:request-params message)
                                   "text-document"
                                   "uri"))
-              (document-text (gethash document-uri clef-lsp/server:*documents*))
+              (document-text (gethash document-uri ctx:documents))
               (syntax-errors (get-syntax-errors document-text))
               (compile-errors (debounced-sb-collect-diagnostics document-text document-uri))
               (items (append syntax-errors compile-errors)))

src/lsp/document/did-change.lisp

@@ -31,21 +31,21 @@
        ;; Unpack the params into the document uri and range/text data
        (let* ((params (clef-jsonrpc/types:request-params message))
               (document-uri (href params "text-document" "uri"))
-              (content-changes (href params "content-changes")))
+              (content-changes (href params "content-changes"))
+              (documents ctx:documents))
              (slog :debug "[textDocument/didChange] Document: ~A" document-uri)
-             ;; (slog :debug "[textDocument/didChange] File found: ~A" (nth-value 1 (gethash document-uri clef-lsp/server:*documents*)))
 
              (dotimes (i (length content-changes))
                       (let* ((content-change (aref content-changes i))
                              (new-document-text (href content-change "text")))
-                            (setf (gethash document-uri clef-lsp/server:*documents*) new-document-text)))
+                            (setf (gethash document-uri documents) new-document-text)))
 
              ;; Reprocess the symbol-map. This is terribly jank and inefficient to do on every single change; needs debounced at the very least
              (slog :debug "[textDocument/didChange] Rebuilding symbol map for document: ~A..." document-uri)
              (let ((start-time (get-internal-real-time)))
                   (clef-symbols:build-file-symbol-map
                     (clef-util:cleanup-path document-uri)
-                    (gethash document-uri clef-lsp/server:*documents*))
+                    (gethash document-uri documents))
                   (slog :debug "[textDocument/didChange] Rebuilt symbol map in ~A ms."
                         (/ (- (get-internal-real-time) start-time) 1000.0)))))
 

src/lsp/document/did-open.lisp

@@ -4,6 +4,4 @@
        (let* ((params-hash (clef-jsonrpc/types:request-params message))
               (document-uri (href params-hash "text-document" "uri"))
               (document-text (href params-hash "text-document" "text")))
-
-             ;; (slog :debug "opened text: ~A" document-text)
-             (setf (gethash document-uri clef-lsp/server:*documents*) document-text)))
+             (setf (gethash document-uri ctx:documents) document-text)))

src/lsp/document/did-save.lisp

@@ -9,19 +9,19 @@
                    (reload-asd-file document-uri))
              ;; Rebuild symbol map for the saved file
              (let ((document-text (gethash (format nil "file://~A" document-uri)
-                                           clef-lsp/server:*documents*)))
+                                           ctx:documents)))
                   (when document-text
                         (clef-symbols:build-file-symbol-map document-uri document-text)))))
 
 (defun reload-asd-file (asd-path)
        "Re-parse an .asd file and reload any changed systems."
        (slog :debug "Reloading .asd file: ~A" asd-path)
-       (let ((new-systems (clef-lsp/lifecycle::parse-asd-file asd-path)))
+       (let ((new-systems (clef-lsp/lifecycle:parse-asd-file asd-path)))
             (dolist (sys new-systems)
                     (let ((name (clef-symbols:system-info-name sys)))
                          ;; Update or add the system info
-                         (setf (gethash name clef-lsp/lifecycle::*loaded-systems*) sys)
+                         (setf (gethash name ctx:loaded-systems) sys)
                          ;; Reload the system
-                         (clef-lsp/lifecycle::load-system-with-info sys)))
+                         (clef-lsp/lifecycle:load-system-with-info sys)))
             ;; Rebuild file mapping
-            (clef-lsp/lifecycle::build-file-to-system-mapping)))
+            (clef-lsp/lifecycle:build-file-to-system-mapping)))

src/lsp/document/formatting.lisp

@@ -17,7 +17,7 @@
 (defun handle-text-document-formatting (message)
        (let* ((params (clef-jsonrpc/types:request-params message))
               (file-uri (href params "text-document" "uri"))
-              (document-text (gethash file-uri clef-lsp/server:*documents*))
+              (document-text (gethash file-uri ctx:documents))
               (pos (href params "position"))
               (options (href params "options")))
              ;; Format the entire file with cl-indentify

src/lsp/document/highlight.lisp

@@ -38,7 +38,7 @@ Returns all occurrences of the symbol under cursor in the current document."
         ;; Find all occurrences in this file
         (let ((highlights '()))
           ;; Add all references in this file
-          (let ((refs-tree (gethash file-path clef-symbols:*symbol-refs-by-file*)))
+          (let ((refs-tree (gethash file-path ctx:symbol-refs)))
             (when refs-tree
               (let ((all-refs (get-all-intervals-from-tree refs-tree)))
                 (dolist (interval all-refs)
@@ -51,7 +51,7 @@ Returns all occurrences of the symbol under cursor in the current document."
                             highlights)))))))
 
           ;; Add definitions in this file
-          (let ((scopes-tree (gethash file-path clef-symbols:*lexical-scopes-by-file*)))
+          (let ((scopes-tree (gethash file-path ctx:lexical-scopes)))
             (when scopes-tree
               (let ((all-scopes (get-all-intervals-from-tree scopes-tree)))
                 (dolist (scope-interval all-scopes)
@@ -72,7 +72,3 @@ Returns all occurrences of the symbol under cursor in the current document."
   "Create an LSP DocumentHighlight dict from a tree-sitter node."
   (dict "range" (node-to-lsp-range node)
         "kind" kind))
-
-;; Register the handler
-(setf (gethash "textDocument/documentHighlight" clef-lsp/server:*handlers*)
-      #'handle-text-document-highlight)

src/lsp/document/hover.lisp

@@ -46,10 +46,10 @@
               (hover-line (href params "position" "line"))
               (hover-char (href params "position" "character"))
               (symbol-at-pos (find-symbol-at-position
-                               (href clef-lsp/server:*documents* document-uri)
+                               (gethash document-uri ctx:documents)
                                hover-line
                                hover-char))
-              (document-text (href clef-lsp/server:*documents* document-uri))
+              (document-text (gethash document-uri ctx:documents))
               (tree (clef-parser/parser:parse-string document-text))
               (symbol-pkg (or (clef-parser/utils:find-package-declaration tree document-text)
                               *package*))

src/lsp/document/references.lisp

@@ -55,10 +55,10 @@ Returns all locations where the symbol at the given position is referenced."
 This handles the case where the cursor is on a function/variable name in a definition
 (e.g., the 'foo' in '(defun foo ...)'), rather than on a usage."
   (let* ((file-path (clef-util:cleanup-path document-uri))
-         (offset (clef-symbols::line-char-to-byte-offset file-path line character)))
+         (offset (clef-symbols:line-char-to-byte-offset file-path line character)))
     ;; Get the lexical scope at this position
     (let ((scopes (interval:find-all
-                   (gethash file-path clef-symbols:*lexical-scopes-by-file*)
+                   (gethash file-path ctx:lexical-scopes)
                    offset)))
       ;; Check each scope (from innermost to outermost) for definitions at this position
       (dolist (scope-interval scopes)
@@ -100,7 +100,7 @@ Returns a list of LSP Location dicts."
                  ;; Walk all intervals in the tree to find matching symbol names
                  (let ((file-refs (find-refs-in-tree refs-tree symbol-name file-path)))
                    (setf locations (nconc locations file-refs)))))
-             clef-symbols:*symbol-refs-by-file*)
+             ctx:symbol-refs)
     locations))
 
 (defun find-refs-in-tree (refs-tree symbol-name file-path)

src/lsp/document/signature-help.lisp

@@ -8,7 +8,7 @@ Returns signature information for the function call at the cursor position."
          (position (href params "position"))
          (line (href position "line"))
          (character (href position "character"))
-         (document-text (gethash document-uri clef-lsp/server:*documents*)))
+         (document-text (gethash document-uri ctx:documents)))
     (slog :debug "[textDocument/signatureHelp] Document: ~A" document-uri)
     (slog :debug "[textDocument/signatureHelp] Position: line ~A, char ~A" line character)
 
@@ -214,7 +214,3 @@ Tries workspace index first, then falls back to sb-introspect for loaded functio
     (dict "signatures" (vector signature)
           "activeSignature" 0
           "activeParameter" (or active-param 0))))
-
-;; Register the handler
-(setf (gethash "textDocument/signatureHelp" clef-lsp/server:*handlers*)
-      #'handle-text-document-signature-help)