Skip to content

Provide generic position-aware text primitives (word-at-position, whole-word occurrences) + a hover MarkupContent builder #2

Closed

Description

@MelbourneDeveloper

Context

Building hover/definition/references for the Osprey language server (osprey-lsp, built on lspkit/lspkit-server/lspkit-vfs/lspkit-live), we found that lspkit provides the engine contract, JSON-RPC dispatch, capability registration (including the Hover capability), and lspkit-vfs encoding/position conversion — but not the small set of generic, language-agnostic text/position primitives that every LSP backend needs. As a result osprey-lsp hand-rolls them in its own crates/osprey-lsp/src/text.rs.

These are not language-specific: they operate purely on text + PositionEncoding, so they belong in lspkit-vfs (or a lspkit-text module) rather than being reinvented per server.

Requested primitives

  1. Word / identifier span at a position — given a line (or rope + position) and a PositionEncoding, return the identifier under the cursor and its (start, end) character offsets in the negotiated encoding. Today this is text::word_at in osprey-lsp. This is the core lookup behind hover, go-to-definition, signature help, and rename.

  2. Whole-word occurrence scan — given document text + a name + encoding, return every whole-word occurrence with line/char spans. Today this is text::occurrences, used for find-references and definition anchoring.

  3. Encoding-aware column measurement — char_width(c, encoding) / measure(str, encoding) / prefix_to(line, character, encoding). lspkit-vfs already has position_to_byte internally; exposing these would let servers stop duplicating the UTF-8/UTF-16 width logic.

  4. (Minor) Hover MarkupContent builder — lspkit-server knows the Hover capability but offers no helper to build the { contents: { kind: \"markdown\", value } } response. A tiny hover_markdown(Option<String>) -> Value (null when None) would remove hand-rolled JSON in each server's wire layer.

Why this matters

The "one engine, two surfaces" value of lspkit is undercut if each backend still re-implements identifier extraction and occurrence scanning — exactly the bits most prone to off-by-one and encoding bugs. Centralizing them in lspkit-vfs would make these correct once.

Current workaround

osprey-lsp carries its own text.rs (word_at, occurrences, char_width, measure, prefix_to) and a wire::hover_result JSON builder. We'd happily migrate to lspkit equivalents once available.

Filed while implementing variable hover (type + doc comments) in osprey-lsp; the hover content is necessarily language-specific (Osprey type inference + doc-comment grammar), but the position/text plumbing above is not.

Activity

  1. MelbourneDeveloper commented on Nov 20, 2026

    @MelbourneDeveloper
    CollaboratorAuthor

    Fixed in PR #64; not yet released, so this stays open

    Not in v0.10.1: the release has no lspkit::vfs::text module and no hover_markdown.

    Added in PR #64:

    • lspkit::vfs::text ([VFS-TEXT]): word_at, occurrences, prefix_to, measure and char_width. They take the negotiated PositionEncoding (UTF-8, UTF-16 or UTF-32) and recognise identifiers by Unicode XID_Continue.
    • lspkit_server::hover_markdown ([LSP-HOVER-MARKDOWN]): returns {contents:{kind:"markdown",value}}, or null when there is nothing to show.

    Tests:

    • crates/lspkit/tests/vfs_text.rs and vfs_text_scan.rs, run in every encoding, covering:
      • astral and combining characters;
      • end-of-line and out-of-range columns;
      • whole-word matching only (foo never matches inside foobar, _foo or foo2).
    • crates/lspkit-server/tests/hover.rs.
    • Local make ci passed, and PR Fix #2, #4, #23, #42; memory-lifetime leaks and CPU session bugs #64 CI passed on 5a01304.

    Next: after #64 merges and ships in a release, I'll call these APIs from the published crates and close this issue with that evidence.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions