Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions paper.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,9 +53,9 @@ We use LLMs because many skill keywords carry a domain context that can only be

Matching uses exact search, primarily for reproducibility: as research tooling, LAiSER should not give completely different results from run to run, since that may jeopardize downstream work. Each extracted phrase is embedded with a sentence-transformer model [@Reimers2019] and compared with every entry in a Facebook AI Similarity Search (FAISS) index [@Johnson2019]. At about 27,000 skill entries, an exhaustive search is cheap next to the LLM call, and the same phrase always returns the same entry and score. The closest entry is kept only if its similarity clears a threshold (0.60 for skills by default), so a phrase with no good match, such as a bare "Airflow", is dropped rather than mismatched.

In its default configuration, LAiSER decodes at temperature zero and passes a fixed seed to every backend that accepts one, so that repeated runs with a local model should return the same output. These measures have limits. Hosted models may change under the same name, and providers do not guarantee identical output even at temperature zero; the OpenAI API also accepts no seed. Matching cannot recover a skill the model did not extract, and different models may extract different skills. The prompt asks for at most ten skills per document, a design choice to filter out noise, so long documents are summarized rather than covered exhaustively. The similarity thresholds were tuned against an internal hand-labeled sample that is not yet public. So that a batch keeps running, a document whose model call fails, or whose reply cannot be parsed, is skipped with a warning. All document types, including syllabi, use the same extraction prompt, because different prompts for different documents would give results that are hard to compare, and one prompt is easier to maintain. We therefore recommend that users record the model, temperature, seed and thresholds when reporting results, and spot-check a sample of the output.
In its default configuration, LAiSER decodes at temperature zero and passes a fixed seed to every backend that accepts one, so that repeated runs with a local model should return the same output. These measures have limits. Hosted models may change under the same name, and providers do not guarantee identical output even at temperature zero; the OpenAI Responses API also accepts no seed. Matching cannot recover a skill the model did not extract, and different models may extract different skills. The prompt asks for at most ten skills per document, a design choice to filter out noise, so long documents are summarized rather than covered exhaustively. The similarity thresholds were tuned against an internal hand-labeled sample that is not yet public. So that a batch keeps running, a document whose model call fails, or whose reply cannot be parsed, is skipped with a warning. All document types, including syllabi, use the same extraction prompt, because different prompts for different documents would give results that are hard to compare, and one prompt is easier to maintain. We therefore recommend that users record the model, temperature, seed and thresholds when reporting results, and spot-check a sample of the output.

The source code is available at [https://github.com/LAiSER-Software/extract-module](https://github.com/LAiSER-Software/extract-module) under the BSD 3-Clause License, and can be installed with `pip install laiser`, or `pip install laiser[gpu]` for GPU inference.
The source code is available at [https://github.com/LAiSER-Software/extract-module](https://github.com/LAiSER-Software/extract-module) under the BSD 3-Clause License, and can be installed with `pip install laiser`, or `pip install laiser[gpu]` for GPU inference, followed by `laiser setup`.

# Research impact statement

Expand All @@ -65,7 +65,7 @@ LAiSER received the Bronze Award in the AI in Education (Higher Education Instit

# AI usage disclosure

Generative AI was used in developing LAiSER and preparing this paper. GitHub Copilot (completion, coding agent and Autofix) and Claude Code (Claude Opus 5) wrote and refactored code and tests, all checked by the test suite and reviewed by the authors. Gemini 2.5 Pro and 3.1 Pro, Claude 3.5 Sonnet, and Claude Opus 4, 5 and 5.5 drafted portions of the documentation, which project members reviewed and edited. For the paper, Copilot's coding agent shortened a November 2025 draft, Claude 3.5 Sonnet and Claude Opus 4 drafted the March 2026 version, and Claude Code (Claude Opus 5) co-authored the September 2026 revisions. In this revision, the authors answered structured questions for the Statement of need, State of the field and Software design; Claude Code (Claude Opus 5.5) assembled those answers into paragraphs, added technical details from the code, checked claims against the code and sources, and drafted the Software design limitations paragraph, which the authors revised. The authors made all design decisions, reviewed and validated all AI-assisted content, and take full responsibility for the paper.
Generative AI was used in developing LAiSER and preparing this paper. GitHub Copilot (completion, coding agent and Autofix) and Claude Code (Claude Sonnet 4.6, Opus 5 and 5.5) wrote and refactored code and tests, all checked by the test suite and reviewed by the authors. Gemini 2.5 Pro and 3.1 Pro, Claude 3.5 Sonnet, and Claude Opus 4, 5 and 5.5 drafted portions of the documentation, which project members reviewed and edited. For the paper, Copilot's coding agent shortened a November 2025 draft, Claude 3.5 Sonnet and Claude Opus 4 drafted the March 2026 version, and Claude Code (Claude Opus 5) co-authored the September 2026 revisions. In this revision, the authors answered structured questions for the Statement of need, State of the field and Software design; Claude Code (Claude Opus 5.5) assembled those answers into paragraphs, added technical details from the code, checked claims against the code and sources, and drafted the Software design limitations paragraph, which the authors revised. The authors made all design decisions, reviewed and validated all AI-assisted content, and take full responsibility for the paper.

# Acknowledgments

Expand Down
Loading