Repository navigation
fix: deterministic decoding by default; paper updates for JOSS review #424 - #446
Conversation
Re-lands the decoding half of #426, which #428 reverted, ported onto the current router and with the problems its review found fixed. Addresses the non-determinism question in JOSS review #424. Every backend previously inherited its provider's default decoding: vLLM, OpenAI and Anthropic sample at temperature 1.0, llama.cpp used 0.2, and a Transformers model used whatever its checkpoint's generation_config.json said. Qwen2.5-0.5B-Instruct, for example, ships do_sample=true and temperature=0.7, so the local CPU path sampled even though nothing asked it to. - config: DEFAULT_TEMPERATURE = 0.0 and DEFAULT_TOP_P = 1.0, overridable via LAISER_TEMPERATURE / LAISER_TOP_P. GENERATION_SEED (42) already existed. - Gemini, OpenAI, Anthropic, vLLM, llama.cpp and Transformers entry points all default to those values. The seed is sent to Gemini and llama.cpp only when the installed client accepts one, so older releases do not break. - Transformers: at temperature 0.0, do_sample=False is passed explicitly so a checkpoint's own config cannot re-enable sampling. Above 0.0 the seed is applied before generation. - SkillExtractorRefactored, SkillExtractionService and LLMRouter take temperature and seed and pass them through. Fixed relative to #426: - LLMRouter.generate always sent the router's seed to llama.cpp, ignoring a caller's seed. Per-call temperature= and seed= now reach every backend. - #426 routed Transformers-loaded models to vLLM with no engine; that was fixed separately in #442, and decoding settings now follow both paths. - Gemini's max_output_tokens is kept (#426 dropped it). - The determinism tests read LAISER_TEMPERATURE at import, so a runner with that variable set failed them. They now reload config under a controlled environment, compare entry points against config rather than literals, and apply the `determinism` marker that #426 registered but never used. tests/test_determinism.py: shipped defaults and env overrides; every public entry point's temperature and seed defaults; the decoding parameters each backend actually receives (OpenAI and Anthropic request bodies, Gemini GenerateContentConfig, Transformers generate kwargs, vLLM SamplingParams, llama.cpp with and without seed support); router forwarding and per-call overrides on all five routes; extractor-to-router pass-through; and three alignment-stability tests on the real skills index. tests/test_llm_router.py is updated for the new keyword arguments. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- New Reproducibility page: per-backend temperature and seed, why sampling is switched off explicitly for Transformers, how to change the defaults, a reference configuration, what is not guaranteed (hosted models change, different models extract different phrases, hardware and library versions), and what tests/test_determinism.py checks. - README: temperature and seed in the runtime options table, and a "Reproducible results" section with the reference configuration, which the reply on #424 described but which was never committed. - Model providers page links to Reproducibility. The API reference picks up the new temperature and seed parameters from the docstrings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Abbreviations. Every abbreviation is now spelled out at or before first use: ESCO, LLM, AI, NER, FAISS, API, GPU, KSA, SCQF, MIT, GW and CCAS; spaCy is described as a natural language processing library and EMSI as now Lightcast. The ESCO and FAISS expansions had existed in earlier drafts but were cut when the paper was shortened in be462ee. Summary. Rewritten for a non-technical reader: it opens on the problem in plain terms, explains what a skill taxonomy is before naming one, and says who the software is for. Pipeline internals, the SCQF sentence, and the licence and installation line move out of the Summary; the licence and installation line now closes Software design, and the FAISS and SCQF citations move to where those terms are now first used. Non-determinism. Software design regains the explanation #428 removed, with corrections: decoding is greedy with a fixed seed where accepted; alignment projects generations onto a closed taxonomy vocabulary by exact search; every backend shares one output schema; and residual variability is stated, now including that hosted models change and providers do not guarantee identical output at temperature zero. Accuracy fixes made while rewriting those paragraphs: - "ESCO's 13,000+ skill entries" replaced by the bundled figure, about 27,000 skill entries across the four taxonomies. - The backend list now includes llama.cpp and OpenAI where it was missing, and "automatic fallback" is stated precisely: on a GPU, a model vLLM cannot load falls back to Transformers. - The output description no longer lists "optional SCQF proficiency levels", which extract_concepts does not produce. Paper length is 1,628 words excluding references. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The example table came from a run in #442 where Qwen2.5-0.5B-Instruct was still sampling, because its generation_config.json enables sampling and nothing turned it off. With greedy decoding now the default, the same input returns three skills rather than two, identically on every run (verified in two separate processes). The table and the note beneath it now match. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Brings in 25c1314 (GPU extra dependencies and provider docs) so the provider documentation corrected here matches dev. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Code - Gemini ignored LAISER_TOP_P: GenerateContentConfig was built without top_p, so the documented process-wide override reached every other backend but not Gemini. GeminiAPI and gemini_generate now take top_p (default DEFAULT_TOP_P) and send it; the Gemini request test asserts it. Tests - _import_or_skip skipped on any ImportError, so a broken import inside a backend module would have turned its tests into silent skips. It now skips only when ImportError.name names a genuinely optional package (google, vllm, llama_cpp) and re-raises everything else. - The module docstring claimed no model download. The alignment tests use the real sentence-transformers embedding model, which Hugging Face downloads on first use, as tests/test_alignment_core.py already does. The claim is corrected rather than the model stubbed, because repeatable embeddings are part of what those tests check. Docs and paper - Reproducibility page: the same no-download claim is corrected, and alignment "can" map different wordings to one entry. Each phrase is matched independently, so that is likely rather than guaranteed. - Paper and providers page: the GPU fallback has two steps. load_model_from_vllm retries DEFAULT_VLLM_MODEL_ID before the router falls back to Transformers; both now say so. - Paper: preprocessing is described as the extraction prompts instructing the model to disregard branding, boilerplate and benefits, not as a separate removal step. Convergence of different phrasings is softened the same way as on the Reproducibility page. Not changed: the State of the field claim that LAiSER supports SCQF proficiency classification, which the public API does not currently expose. That is left for the authors to decide. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
State of the field claimed LAiSER uniquely supports KSA decomposition with SCQF proficiency-level classification. The public API does not reach that capability: the KSA/SCQF prompt builders and parsers in services.py are never called, and extract_concepts(levels=...) has no effect. The sentence is removed, along with the related 'proficiency-level classification' and 'multi-taxonomy proficiency alignment' wording, so the paper only claims what the code does. Raised in Copilot's review of #444. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…coding fix: deterministic decoding by default; paper updates for JOSS review #424
|
Complexity Analysis Average: C — Moderate (13.75) Hotspots tracked below. These are functional, well-exercised components — listed for visibility as the codebase scales.
|
|
Code Hygiene The following items are defined but not yet in active use. Routine cleanup — no impact on functionality.
|
There was a problem hiding this comment.
🟡 Changes recommended
Address the reproducibility caveats, model revision guidance, and determinism test coverage gaps.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
Promotes deterministic decoding defaults and JOSS review updates from dev to main.
Changes:
- Adds temperature and seed controls across LLM backends.
- Adds determinism and alignment tests.
- Updates documentation, quickstart content, and the JOSS paper.
File summaries
| File | Description |
|---|---|
tests/test_llm_router.py |
Updates router test fixtures for decoding settings. |
tests/test_determinism.py |
Adds decoding and alignment stability tests. |
README.md |
Documents decoding options and reproducibility. |
pytest.ini |
Registers the determinism marker. |
paper.md |
Applies JOSS review revisions. |
mkdocs.yml |
Adds reproducibility documentation to navigation. |
laiser/skill_extractor_refactored.py |
Exposes temperature and seed controls. |
laiser/services.py |
Propagates decoding settings through the service layer. |
laiser/llm_models/openai.py |
Adds deterministic request parameters. |
laiser/llm_models/llm_router.py |
Routes decoding settings to backends. |
laiser/llm_models/llama_cpp_handler.py |
Adds decoding settings and compatibility handling. |
laiser/llm_models/hugging_face_llm.py |
Controls Transformers and vLLM decoding. |
laiser/llm_models/gemini.py |
Adds Gemini decoding configuration. |
laiser/llm_models/anthropic.py |
Adds Anthropic temperature and top-p settings. |
laiser/config.py |
Defines decoding defaults and environment overrides. |
docs/reproducibility.md |
Documents defaults, limitations, and verification. |
docs/quickstart.md |
Updates the local example output. |
docs/providers.md |
Documents provider behavior and fallback order. |
Review details
Suppressed comments (7)
README.md:196
- These new public constructor options are added to the README, but
docs/usage.mdstill omitstemperatureandseedeven though the next paragraph says every option is described there. Add them to the usage guide or update the link/claim so users can discover the controls.
| `temperature` | decoding temperature for every backend (default `0.0`, greedy) |
| `seed` | seed for backends that accept one (default `42`) |
docs/reproducibility.md:72
- The reproducibility guidance asks users to record/pin a Hugging Face model revision, but the loader always requests
revision="main"and neitherSkillExtractorRefactorednor the router exposes a revision argument. A later update under the same model ID can therefore change results even when users follow this guidance; expose and forward a revision, or document a supported way to load the recorded revision.
- **the LAiSER version** — it fixes the bundled taxonomy data and the embedding model used for alignment,
`sentence-transformers/all-MiniLM-L6-v2`;
- **the model revision**, for Hugging Face models that may be updated under the same name.
docs/reproducibility.md:4
- The opening says LAiSER prevents repeated runs from differing, but the caveat below correctly notes that provider, hardware, and library changes can still alter results. Describe the defaults as improving reproducibility rather than guaranteeing it.
Language models generate text by sampling, so left to their providers' defaults two runs over the same input
can return different skills. LAiSER is configured so that, by default, they do not.
laiser/llm_models/llama_cpp_handler.py:106
- The new
LlamaCppBackend.generatewrapper resolves the constructor defaults and forwards per-call settings, but the tests only exercisellama_cpp_chatdirectly. Add a direct-wrapper test for both the default and override paths so this public backend cannot regress while the lower-level helper tests continue to pass.
temperature = self.temperature if temperature is None else temperature
seed = self.seed if seed is None else seed
paper.md:54
- The paper says a fixed seed makes runs reproducible even when temperature is raised, then immediately says hosted providers do not guarantee identical output. The code also omits a seed for greedy Transformers generation. Qualify this as improved/best-effort reproducibility rather than a guarantee.
Because LLMs sample their output, identical inputs can yield different generations across runs, and different backends can yield different generations from the same input. LAiSER addresses this in three ways. First, decoding is deterministic by default: every backend decodes greedily at temperature zero, and a fixed seed is supplied to every backend that accepts one, so runs stay reproducible even when a user deliberately raises the temperature. Temperature and seed are constructor arguments, so sampled variation is available when wanted but is never the silent default. Second, the alignment stage acts as a normalizing projection. Generated phrases are embedded with a fixed sentence-transformer model and matched by exact search against a precomputed index, so surface variation in the generated text collapses onto a closed, finite vocabulary of taxonomy entries, and different phrasings of the same competency can converge on the same aligned record, although because each phrase is matched independently this is likely rather than guaranteed. Third, every backend produces the same output schema, which makes results directly comparable across models and lets cross-model agreement be measured on the aligned columns. The test suite checks each backend's decoding defaults, the decoding parameters each backend actually receives, and that repeated alignment of a fixed input returns an identical result, similarity scores included.
tests/test_determinism.py:459
- Skipping an empty result makes this stability test pass when alignment returns no rows, so it does not protect the claimed taxonomy behavior. The fixture already handles an unavailable index; assert that these fixed common phrases produce output instead.
if aligned.empty: # pragma: no cover - environment dependent
pytest.skip("Alignment returned no rows")
tests/test_determinism.py:473
- This skip has the same problem as the earlier alignment test: a regression that makes repeated phrases produce no matches is reported as skipped rather than failed. Assert non-empty output so the per-phrase determinism checks actually run.
if aligned.empty: # pragma: no cover - environment dependent
pytest.skip("Alignment returned no rows")
- Files reviewed: 18/18 changed files
- Comments generated: 4
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
|
||
| ### Reproducible results | ||
|
|
||
| Every backend decodes greedily by default (temperature `0.0`) and uses a fixed seed where the backend accepts one, so repeated runs over the same input give the same result. To make results reproducible for others, pin every setting that affects them: |
| Every backend decodes greedily: at temperature `0.0` the model always takes its most probable next token, so | ||
| the same prompt produces the same text. A fixed seed (`42`) is also supplied wherever a backend accepts one, | ||
| which keeps runs reproducible if you deliberately raise the temperature. |
| `tests/test_determinism.py` runs in CI without a GPU, an API key or a language model download. The alignment | ||
| checks use the real embedding model, which Hugging Face downloads on first use. It checks that: |
| Language models sample their output, so the same input can produce different | ||
| generations across runs and across backends (JOSS review #424). LAiSER limits | ||
| this in two ways, and these tests check both without a GPU, an API key or a |
Brings the #424 work from
devtomain. It was reviewed and merged intodevin #444, including fixes for everything Copilot raised there; this PR only promotes it.devmerges intomaincleanly.Addresses #424, the JOSS paper review. The issue stays open for the reviewer to close.
What reaches
main494938afeat(llm): decode deterministically by default on every backend. Every backend defaults to temperature 0.0, with seed 42 wherever the backend accepts one. Both are exposed astemperatureandseedonSkillExtractorRefactored. Previously most backends used their provider's default, typically temperature 1.0, and the local Transformers path followed the checkpoint's own config — which forQwen2.5-0.5B-Instructenables sampling. Addstests/test_determinism.py.26a7b88docs: document reproducible results and the decoding defaults. New Reproducibility page, and a README section with a reference configuration.5b7d302docs(paper): respond to JOSS review [JOSS] Suggestions for paper improvement #424. Abbreviations are spelled out at first use, the Summary is rewritten for a non-technical reader, and the non-determinism discussion is restored.9e6ea34docs(quickstart): show the greedy output for the local example.c1fd5b7fix: address review findings on fix: deterministic decoding by default; paper updates for JOSS review #424 #444.top_p.05bc8bddocs(paper): remove the unsupported SCQF proficiency claim. The public API does not reach that capability.Full detail, including the item-by-item mapping to #424 and the review responses, is in #444.
Verification (from #444)
black,isortandflake8are clean.-m "not index": 58 passed, 5 skipped (need API keys), 1 deselected.mkdocs build --strictpasses.After merge
main.Not included: the vLLM AWQ loading bug found during review is tracked separately in #445.
🤖 Generated with Claude Code