Repository navigation
fix: deterministic decoding by default; paper updates for JOSS review #424 - #444
Conversation
Re-lands the decoding half of #426, which #428 reverted, ported onto the current router and with the problems its review found fixed. Addresses the non-determinism question in JOSS review #424. Every backend previously inherited its provider's default decoding: vLLM, OpenAI and Anthropic sample at temperature 1.0, llama.cpp used 0.2, and a Transformers model used whatever its checkpoint's generation_config.json said. Qwen2.5-0.5B-Instruct, for example, ships do_sample=true and temperature=0.7, so the local CPU path sampled even though nothing asked it to. - config: DEFAULT_TEMPERATURE = 0.0 and DEFAULT_TOP_P = 1.0, overridable via LAISER_TEMPERATURE / LAISER_TOP_P. GENERATION_SEED (42) already existed. - Gemini, OpenAI, Anthropic, vLLM, llama.cpp and Transformers entry points all default to those values. The seed is sent to Gemini and llama.cpp only when the installed client accepts one, so older releases do not break. - Transformers: at temperature 0.0, do_sample=False is passed explicitly so a checkpoint's own config cannot re-enable sampling. Above 0.0 the seed is applied before generation. - SkillExtractorRefactored, SkillExtractionService and LLMRouter take temperature and seed and pass them through. Fixed relative to #426: - LLMRouter.generate always sent the router's seed to llama.cpp, ignoring a caller's seed. Per-call temperature= and seed= now reach every backend. - #426 routed Transformers-loaded models to vLLM with no engine; that was fixed separately in #442, and decoding settings now follow both paths. - Gemini's max_output_tokens is kept (#426 dropped it). - The determinism tests read LAISER_TEMPERATURE at import, so a runner with that variable set failed them. They now reload config under a controlled environment, compare entry points against config rather than literals, and apply the `determinism` marker that #426 registered but never used. tests/test_determinism.py: shipped defaults and env overrides; every public entry point's temperature and seed defaults; the decoding parameters each backend actually receives (OpenAI and Anthropic request bodies, Gemini GenerateContentConfig, Transformers generate kwargs, vLLM SamplingParams, llama.cpp with and without seed support); router forwarding and per-call overrides on all five routes; extractor-to-router pass-through; and three alignment-stability tests on the real skills index. tests/test_llm_router.py is updated for the new keyword arguments. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- New Reproducibility page: per-backend temperature and seed, why sampling is switched off explicitly for Transformers, how to change the defaults, a reference configuration, what is not guaranteed (hosted models change, different models extract different phrases, hardware and library versions), and what tests/test_determinism.py checks. - README: temperature and seed in the runtime options table, and a "Reproducible results" section with the reference configuration, which the reply on #424 described but which was never committed. - Model providers page links to Reproducibility. The API reference picks up the new temperature and seed parameters from the docstrings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Abbreviations. Every abbreviation is now spelled out at or before first use: ESCO, LLM, AI, NER, FAISS, API, GPU, KSA, SCQF, MIT, GW and CCAS; spaCy is described as a natural language processing library and EMSI as now Lightcast. The ESCO and FAISS expansions had existed in earlier drafts but were cut when the paper was shortened in be462ee. Summary. Rewritten for a non-technical reader: it opens on the problem in plain terms, explains what a skill taxonomy is before naming one, and says who the software is for. Pipeline internals, the SCQF sentence, and the licence and installation line move out of the Summary; the licence and installation line now closes Software design, and the FAISS and SCQF citations move to where those terms are now first used. Non-determinism. Software design regains the explanation #428 removed, with corrections: decoding is greedy with a fixed seed where accepted; alignment projects generations onto a closed taxonomy vocabulary by exact search; every backend shares one output schema; and residual variability is stated, now including that hosted models change and providers do not guarantee identical output at temperature zero. Accuracy fixes made while rewriting those paragraphs: - "ESCO's 13,000+ skill entries" replaced by the bundled figure, about 27,000 skill entries across the four taxonomies. - The backend list now includes llama.cpp and OpenAI where it was missing, and "automatic fallback" is stated precisely: on a GPU, a model vLLM cannot load falls back to Transformers. - The output description no longer lists "optional SCQF proficiency levels", which extract_concepts does not produce. Paper length is 1,628 words excluding references. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The example table came from a run in #442 where Qwen2.5-0.5B-Instruct was still sampling, because its generation_config.json enables sampling and nothing turned it off. With greedy decoding now the default, the same input returns three skills rather than two, identically on every run (verified in two separate processes). The table and the note beneath it now match. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Code Hygiene The following items are defined but not yet in active use. Routine cleanup — no impact on functionality.
|
|
Complexity Analysis Average: C — Moderate (13.75) Hotspots tracked below. These are functional, well-exercised components — listed for visibility as the codebase scales.
|
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved issues remain in Gemini top-p propagation, offline alignment tests, and paper claims.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
This pull request makes LLM decoding deterministic by default, adds determinism and alignment tests, and updates documentation and the JOSS paper.
Changes:
- Adds configurable temperature, top-p, and seed handling across backends.
- Adds routing, backend-parameter, and alignment tests.
- Updates README, documentation, Quickstart examples, and paper content.
File summaries
| File | Reviewed changes |
|---|---|
tests/test_llm_router.py |
Updates router fixtures for decoding parameters. |
tests/test_determinism.py |
Adds backend, routing, and alignment determinism tests. |
README.md |
Documents decoding options and reproducibility. |
pytest.ini |
Registers the determinism marker. |
paper.md |
Revises summary, terminology, backend details, and reproducibility discussion. |
mkdocs.yml |
Adds reproducibility documentation to navigation. |
laiser/skill_extractor_refactored.py |
Exposes temperature and seed settings. |
laiser/services.py |
Passes decoding settings to the router. |
laiser/llm_models/openai.py |
Adds decoding parameters to requests. |
laiser/llm_models/llm_router.py |
Forwards settings across backend routes. |
laiser/llm_models/llama_cpp_handler.py |
Adds greedy defaults and optional seed handling. |
laiser/llm_models/hugging_face_llm.py |
Controls sampling, seeds, and vLLM parameters. |
laiser/llm_models/gemini.py |
Adds temperature and seed configuration. |
laiser/llm_models/anthropic.py |
Adds temperature and top-p request parameters. |
laiser/config.py |
Defines decoding defaults and environment overrides. |
docs/reproducibility.md |
Documents reproducibility behavior and limitations. |
docs/quickstart.md |
Updates deterministic example output. |
docs/providers.md |
Links provider documentation to reproducibility guidance. |
Review details
Suppressed comments (3)
docs/reproducibility.md:80
- Nearest-neighbor alignment does not guarantee that different wordings select the same taxonomy entry; each phrase is embedded and searched independently. The documented behavior should say alignment can map wordings to a common entry, not that it does so, while retaining the caveat that missing model extractions cannot be recovered.
- **Different models extract different phrases.** Alignment maps different wordings of a skill onto the same
taxonomy entry, but it cannot recover a skill a model never extracted. Compare models on the aligned columns,
and measure their agreement rather than assuming it.
docs/reproducibility.md:86
- The new alignment checks instantiate
SentenceTransformer(DEFAULT_EMBEDDING_MODEL)on first use, so a clean runner may download the embedding weights and a no-network runner can fail. This statement is therefore false for the current test implementation; make the alignment fixture fully offline or revise the claim.
`tests/test_determinism.py` runs in CI without a GPU, a model download or an API key. It checks that:
paper.md:54
- Exact nearest-neighbor search makes repeated alignment of one query deterministic, but it does not ensure that different phrasings independently embed to the same taxonomy entry. The implementation searches each phrase separately and the new tests only establish repeatability for identical inputs, so this sentence should be softened from a guarantee that phrasings "converge" to a possibility.
Because LLMs sample their output, identical inputs can yield different generations across runs, and different backends can yield different generations from the same input. LAiSER addresses this in three ways. First, decoding is deterministic by default: every backend decodes greedily at temperature zero, and a fixed seed is supplied to every backend that accepts one, so runs stay reproducible even when a user deliberately raises the temperature. Temperature and seed are constructor arguments, so sampled variation is available when wanted but is never the silent default. Second, the alignment stage acts as a normalizing projection. Generated phrases are embedded with a fixed sentence-transformer model and matched by exact search against a precomputed index, so surface variation in the generated text collapses onto a closed, finite vocabulary of taxonomy entries, and different phrasings of the same competency converge on the same aligned record. Third, every backend produces the same output schema, which makes results directly comparable across models and lets cross-model agreement be measured on the aligned columns. The test suite checks each backend's decoding defaults, the decoding parameters each backend actually receives, and that repeated alignment of a fixed input returns an identical result, similarity scores included.
- Files reviewed: 18/18 changed files
- Comments generated: 6
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Brings in 25c1314 (GPU extra dependencies and provider docs) so the provider documentation corrected here matches dev. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Code - Gemini ignored LAISER_TOP_P: GenerateContentConfig was built without top_p, so the documented process-wide override reached every other backend but not Gemini. GeminiAPI and gemini_generate now take top_p (default DEFAULT_TOP_P) and send it; the Gemini request test asserts it. Tests - _import_or_skip skipped on any ImportError, so a broken import inside a backend module would have turned its tests into silent skips. It now skips only when ImportError.name names a genuinely optional package (google, vllm, llama_cpp) and re-raises everything else. - The module docstring claimed no model download. The alignment tests use the real sentence-transformers embedding model, which Hugging Face downloads on first use, as tests/test_alignment_core.py already does. The claim is corrected rather than the model stubbed, because repeatable embeddings are part of what those tests check. Docs and paper - Reproducibility page: the same no-download claim is corrected, and alignment "can" map different wordings to one entry. Each phrase is matched independently, so that is likely rather than guaranteed. - Paper and providers page: the GPU fallback has two steps. load_model_from_vllm retries DEFAULT_VLLM_MODEL_ID before the router falls back to Transformers; both now say so. - Paper: preprocessing is described as the extraction prompts instructing the model to disregard branding, boilerplate and benefits, not as a separate removal step. Convergence of different phrasings is softened the same way as on the Reproducibility page. Not changed: the State of the field claim that LAiSER supports SCQF proficiency classification, which the public API does not currently expose. That is left for the authors to decide. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Thanks for the review. Five of the inline threads are fixed in c1fd5b7 and have their own replies. The SCQF thread is still open: whether to remove or qualify that claim is a decision for the paper's authors. The notes that appeared only in the review summary are also fixed in c1fd5b7:
|
State of the field claimed LAiSER uniquely supports KSA decomposition with SCQF proficiency-level classification. The public API does not reach that capability: the KSA/SCQF prompt builders and parsers in services.py are never called, and extract_concepts(levels=...) has no effect. The sentence is removed, along with the related 'proficiency-level classification' and 'multi-taxonomy proficiency alignment' wording, so the paper only claims what the code does. Raised in Copilot's review of #444. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Tests that need a GPU will be handled in #445. Merging with fixes for everything else. |
Addresses #424.
The reply on #424 (Aug 4) described four changes to the repository. Two of them — deterministic decoding and
tests/test_determinism.py— were added in #426 and then removed by the revert in #428. The other two — spelling out abbreviations and rewriting the Summary — were never committed. This PR makes the reply true, re-landing #426 by hand on top of currentdevand fixing what its review found.1. Deterministic decoding —
494938aEvery backend previously used its provider's default decoding: vLLM, OpenAI and Anthropic sample at temperature 1.0, llama.cpp used 0.2, and a Transformers model followed its checkpoint's
generation_config.json.Qwen/Qwen2.5-0.5B-Instruct, for example, shipsdo_sample: true, temperature: 0.7, so the local CPU path sampled without anyone asking it to.config.py:DEFAULT_TEMPERATURE = 0.0,DEFAULT_TOP_P = 1.0, overridable withLAISER_TEMPERATURE/LAISER_TOP_P.GENERATION_SEED = 42already existed.do_sample=Falseis passed explicitly, so a checkpoint cannot re-enable sampling. Above 0.0 the seed is applied first.SkillExtractorRefactored,SkillExtractionServiceandLLMRoutertaketemperatureandseed.Fixed relative to #426:
temperature=andseed=reach every backend; tested on all five routesmax_output_tokensdropped (Copilot on #428)LAISER_TEMPERATUREat import, so a runner with it set failed (Copilot)configunder a controlled environment and compare against it, not literalsdeterminismmarker registered but never applied (Copilot)Not ported: #426's edits to
laiser/llm_methods.py, which nothing in the package or tests imports.2. Tests —
tests/test_determinism.pyRuns in CI with no GPU, API key or language model download (the alignment checks use the real embedding model, downloaded on first use):
temperatureandseeddefaults;GenerateContentConfig, Transformersgeneratekwargs, vLLMSamplingParams, and llama.cpp builds with and without seed support;3. Documentation —
26a7b88New Reproducibility page (per-backend defaults, overriding them, a reference configuration, what is not guaranteed, what the tests check). The README gains
temperature/seedin its options table and a Reproducible results section with the reference configuration the #424 reply mentioned.4. Paper —
5b7d302be462ee).extract_conceptsdoes not produce them.Length: 1,631 words excluding references.
Review follow-ups
c1fd5b7, from Copilot's review of this PR:top_p, soLAISER_TOP_Preaches every backend.google,vllm,llama_cpp) and re-raises any other import failure.SCQF claim removed (
05bc8bd). State of the field claimed LAiSER "uniquely supports KSA decomposition with SCQF proficiency-level classification". That is not reachable from the public API: the KSA/SCQF prompt builders and parsers inservices.pyare never called, andextract_concepts(levels=...)has no effect. The sentence is removed, along with the related "proficiency-level classification" and "proficiency alignment" wording, so the paper only claims what the code does. If the feature is wired back in, the claim can return with it.Verification
black,isort,flake8: cleantests/test_determinism.py+tests/test_llm_router.py: 55 passedCI selection (
-m "not index"): 58 passed, 5 skipped (credential-gated), 1 deselectedmkdocs build --strict: passes; API reference showstemperatureandseedReal model on CPU, no API key (
Qwen/Qwen2.5-0.5B-Instruct), each run in a separate Python process:With sampling now actually off, the Quickstart input extracts three skills instead of the two found by the sampled run shown in docs: move documentation into this repo; fix keyless CPU extraction #442; the Quickstart's example output is updated to match.
🤖 Generated with Claude Code