Skip to content

fix: deterministic decoding by default; paper updates for JOSS review #424 - #444

Merged
phanindra-max merged 7 commits into
devfrom
fix/424-deterministic-decoding
Sep 15, 2026
Merged

phanindra-max merged 7 commits into
devfrom
fix/424-deterministic-decoding

Conversation

@phanindra-max

@phanindra-max phanindra-max commented Sep 14, 2026 •

Copy link
Copy Markdown
Member

Addresses #424.

The reply on #424 (Aug 4) described four changes to the repository. Two of them — deterministic decoding and tests/test_determinism.py — were added in #426 and then removed by the revert in #428. The other two — spelling out abbreviations and rewriting the Summary — were never committed. This PR makes the reply true, re-landing #426 by hand on top of current dev and fixing what its review found.

1. Deterministic decoding — 494938a

Every backend previously used its provider's default decoding: vLLM, OpenAI and Anthropic sample at temperature 1.0, llama.cpp used 0.2, and a Transformers model followed its checkpoint's generation_config.json. Qwen/Qwen2.5-0.5B-Instruct, for example, ships do_sample: true, temperature: 0.7, so the local CPU path sampled without anyone asking it to.

  • config.py: DEFAULT_TEMPERATURE = 0.0, DEFAULT_TOP_P = 1.0, overridable with LAISER_TEMPERATURE / LAISER_TOP_P. GENERATION_SEED = 42 already existed.
  • Gemini, OpenAI, Anthropic, vLLM, llama.cpp and Transformers entry points default to those. The seed goes to Gemini and llama.cpp only when the installed client accepts one.
  • Transformers: at temperature 0.0, do_sample=False is passed explicitly, so a checkpoint cannot re-enable sampling. Above 0.0 the seed is applied first.
  • SkillExtractorRefactored, SkillExtractionService and LLMRouter take temperature and seed.

Fixed relative to #426:

#426 problem Now
llama.cpp always got the router's seed, ignoring a caller's (Copilot) Per-call temperature= and seed= reach every backend; tested on all five routes
Transformers-loaded models were routed to vLLM with no engine (Copilot) Fixed in #442; decoding settings follow both paths
Gemini max_output_tokens dropped (Copilot on #428) Kept
Tests read LAISER_TEMPERATURE at import, so a runner with it set failed (Copilot) Tests reload config under a controlled environment and compare against it, not literals
determinism marker registered but never applied (Copilot) Applied module-wide

Not ported: #426's edits to laiser/llm_methods.py, which nothing in the package or tests imports.

2. Tests — tests/test_determinism.py

Runs in CI with no GPU, API key or language model download (the alignment checks use the real embedding model, downloaded on first use):

  • shipped defaults, and environment overrides;
  • every public entry point's temperature and seed defaults;
  • the decoding parameters each backend actually receives: OpenAI and Anthropic request bodies, Gemini GenerateContentConfig, Transformers generate kwargs, vLLM SamplingParams, and llama.cpp builds with and without seed support;
  • router forwarding and per-call overrides on the vLLM, Transformers, llama.cpp, Gemini and OpenAI routes;
  • extractor-to-router pass-through;
  • alignment stability on the real skills index: repeated runs are identical, output entries exist in the taxonomy, and a repeated phrase always resolves to the same entry and score.

3. Documentation — 26a7b88

New Reproducibility page (per-backend defaults, overriding them, a reference configuration, what is not guaranteed, what the tests check). The README gains temperature/seed in its options table and a Reproducible results section with the reference configuration the #424 reply mentioned.

4. Paper — 5b7d302

  • Abbreviations: ESCO, LLM, AI, NER, FAISS, API, GPU, MIT, GW and CCAS are spelled out at or before first use; spaCy and EMSI are explained. ESCO and FAISS had been expanded in earlier drafts and were cut when the paper was shortened (be462ee).
  • Summary: rewritten for a non-technical reader — the problem first, what a skill taxonomy is, who the software is for. Internals, the SCQF sentence and the install line move out.
  • Non-determinism: the Software design explanation Revert "fix: default all LLM backends to deterministic greedy decoding" #428 removed is restored, adding that hosted models change and providers do not guarantee identical output at temperature zero.
  • Accuracy fixes in the paragraphs being rewritten: "ESCO's 13,000+ skill entries" becomes the bundled figure (~27,000 skill entries); llama.cpp and OpenAI join the backend list; "automatic fallback" is stated precisely; "optional SCQF proficiency levels" is dropped from the output description, because extract_concepts does not produce them.

Length: 1,631 words excluding references.

Review follow-ups

c1fd5b7, from Copilot's review of this PR:

  • Gemini now sends top_p, so LAISER_TOP_P reaches every backend.
  • The test import helper skips only for missing optional packages (google, vllm, llama_cpp) and re-raises any other import failure.
  • The no-download claim is corrected. The alignment tests use the real embedding model, downloaded on first use.
  • The GPU fallback's two steps are described: vLLM retries a default checkpoint before LAiSER falls back to Transformers.
  • Preprocessing is described as the extraction prompts instructing the model to disregard branding, boilerplate and benefits.
  • "Different phrasings converge on the same entry" is softened to likely rather than guaranteed, because each phrase is matched independently.

SCQF claim removed (05bc8bd). State of the field claimed LAiSER "uniquely supports KSA decomposition with SCQF proficiency-level classification". That is not reachable from the public API: the KSA/SCQF prompt builders and parsers in services.py are never called, and extract_concepts(levels=...) has no effect. The sentence is removed, along with the related "proficiency-level classification" and "proficiency alignment" wording, so the paper only claims what the code does. If the feature is wired back in, the claim can return with it.

Verification

  • black, isort, flake8: clean

  • tests/test_determinism.py + tests/test_llm_router.py: 55 passed

  • CI selection (-m "not index"): 58 passed, 5 skipped (credential-gated), 1 deselected

  • mkdocs build --strict: passes; API reference shows temperature and seed

  • Real model on CPU, no API key (Qwen/Qwen2.5-0.5B-Instruct), each run in a separate Python process:

    • default decoding, two job descriptions, run twice: identical output both times — 5 aligned rows, similarity scores included;
    • sampling at temperature 0.7, seed 7, run twice: identical text; seed 8: different text.

    With sampling now actually off, the Quickstart input extracts three skills instead of the two found by the sampled run shown in docs: move documentation into this repo; fix keyless CPU extraction #442; the Quickstart's example output is updated to match.

🤖 Generated with Claude Code

phanindra-max and others added 4 commits September 14, 2026 00:30
Re-lands the decoding half of #426, which #428 reverted, ported onto the
current router and with the problems its review found fixed. Addresses the
non-determinism question in JOSS review #424.

Every backend previously inherited its provider's default decoding: vLLM,
OpenAI and Anthropic sample at temperature 1.0, llama.cpp used 0.2, and a
Transformers model used whatever its checkpoint's generation_config.json
said. Qwen2.5-0.5B-Instruct, for example, ships do_sample=true and
temperature=0.7, so the local CPU path sampled even though nothing asked it
to.

- config: DEFAULT_TEMPERATURE = 0.0 and DEFAULT_TOP_P = 1.0, overridable via
  LAISER_TEMPERATURE / LAISER_TOP_P. GENERATION_SEED (42) already existed.
- Gemini, OpenAI, Anthropic, vLLM, llama.cpp and Transformers entry points
  all default to those values. The seed is sent to Gemini and llama.cpp only
  when the installed client accepts one, so older releases do not break.
- Transformers: at temperature 0.0, do_sample=False is passed explicitly so
  a checkpoint's own config cannot re-enable sampling. Above 0.0 the seed is
  applied before generation.
- SkillExtractorRefactored, SkillExtractionService and LLMRouter take
  temperature and seed and pass them through.

Fixed relative to #426:
- LLMRouter.generate always sent the router's seed to llama.cpp, ignoring a
  caller's seed. Per-call temperature= and seed= now reach every backend.
- #426 routed Transformers-loaded models to vLLM with no engine; that was
  fixed separately in #442, and decoding settings now follow both paths.
- Gemini's max_output_tokens is kept (#426 dropped it).
- The determinism tests read LAISER_TEMPERATURE at import, so a runner with
  that variable set failed them. They now reload config under a controlled
  environment, compare entry points against config rather than literals,
  and apply the `determinism` marker that #426 registered but never used.

tests/test_determinism.py: shipped defaults and env overrides; every public
entry point's temperature and seed defaults; the decoding parameters each
backend actually receives (OpenAI and Anthropic request bodies, Gemini
GenerateContentConfig, Transformers generate kwargs, vLLM SamplingParams,
llama.cpp with and without seed support); router forwarding and per-call
overrides on all five routes; extractor-to-router pass-through; and three
alignment-stability tests on the real skills index. tests/test_llm_router.py
is updated for the new keyword arguments.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- New Reproducibility page: per-backend temperature and seed, why sampling is
  switched off explicitly for Transformers, how to change the defaults, a
  reference configuration, what is not guaranteed (hosted models change,
  different models extract different phrases, hardware and library versions),
  and what tests/test_determinism.py checks.
- README: temperature and seed in the runtime options table, and a
  "Reproducible results" section with the reference configuration, which the
  reply on #424 described but which was never committed.
- Model providers page links to Reproducibility.

The API reference picks up the new temperature and seed parameters from the
docstrings.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Abbreviations. Every abbreviation is now spelled out at or before first use:
ESCO, LLM, AI, NER, FAISS, API, GPU, KSA, SCQF, MIT, GW and CCAS; spaCy is
described as a natural language processing library and EMSI as now
Lightcast. The ESCO and FAISS expansions had existed in earlier drafts but
were cut when the paper was shortened in be462ee.

Summary. Rewritten for a non-technical reader: it opens on the problem in
plain terms, explains what a skill taxonomy is before naming one, and says
who the software is for. Pipeline internals, the SCQF sentence, and the
licence and installation line move out of the Summary; the licence and
installation line now closes Software design, and the FAISS and SCQF
citations move to where those terms are now first used.

Non-determinism. Software design regains the explanation #428 removed, with
corrections: decoding is greedy with a fixed seed where accepted; alignment
projects generations onto a closed taxonomy vocabulary by exact search;
every backend shares one output schema; and residual variability is stated,
now including that hosted models change and providers do not guarantee
identical output at temperature zero.

Accuracy fixes made while rewriting those paragraphs:
- "ESCO's 13,000+ skill entries" replaced by the bundled figure, about 27,000
  skill entries across the four taxonomies.
- The backend list now includes llama.cpp and OpenAI where it was missing,
  and "automatic fallback" is stated precisely: on a GPU, a model vLLM cannot
  load falls back to Transformers.
- The output description no longer lists "optional SCQF proficiency levels",
  which extract_concepts does not produce.

Paper length is 1,628 words excluding references.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The example table came from a run in #442 where Qwen2.5-0.5B-Instruct was still sampling, because its generation_config.json enables sampling and nothing turned it off. With greedy decoding now the default, the same input returns three skills rather than two, identically on every run (verified in two separate processes). The table and the note beneath it now match.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings September 14, 2026 04:35
@github-actions

Copy link
Copy Markdown

Code Hygiene

The following items are defined but not yet in active use. Routine cleanup — no impact on functionality.

  • laiser/llm_methods.py:87 — unused variable 'args' (100% confidence)
  • laiser/llm_models/llama_cpp_handler.py:45 — unused variable 'chat_format' (100% confidence)
  • laiser/llm_models/llama_cpp_handler.py:86 — unused variable 'exc' (100% confidence)
  • laiser/llm_models/llama_cpp_handler.py:86 — unused variable 'exc_type' (100% confidence)
  • laiser/llm_models/llama_cpp_handler.py:86 — unused variable 'tb' (100% confidence)
  • laiser/llm_models/llm_router.py:71 — unused variable 'args' (100% confidence)
  • laiser/llm_models/llm_router.py:86 — unused variable 'args' (100% confidence)
  • laiser/llm_models/llm_router.py:89 — unused variable 'args' (100% confidence)
  • laiser/services.py:132 — unused variable 'raw_description' (100% confidence)

@github-actions

Copy link
Copy Markdown

Complexity Analysis

Average: C — Moderate (13.75)

Hotspots tracked below. These are functional, well-exercised components — listed for visibility as the codebase scales.

Grade Name Type Score
F FAISSIndexManager.initialize_index Method 51
E SkillExtractionService.extract_and_align_core Method 36
E FAISSIndexManager.search_similar_skills Method 32
D ResponseParser._parse_skills_from_response Method 24
D FAISSIndexManager Class 22
C ResponseParser.parse_knowledge_task_response Method 20
C _BaseTaxonomyFAISSIndexManager.search_similar Method 18
C ResponseParser Class 17
C ResponseParser.parse_ksa_extraction_response Method 16
C ResponseParser.parse_skill_extraction_response Method 15
C openai_generate Function 14
C parse_output_vllm Function 13
C AlignmentService.align Method 11
C anthropic_generate Function 11
C LLMRouter._initialize_components Method 11
B SkillExtractionService._deduplicate Method 10
B load_model_from_vllm Function 10
B GeminiAPI.generate_with_config Method 10
B _BaseTaxonomyFAISSIndexManager.initialize_index Method 9
B ResponseParser.parse_ksa_details_response Method 8
B SkillExtractionService Class 8
B _BaseTaxonomyFAISSIndexManager Class 8
B llm_generate Function 8
B SkillExtractorRefactored._expand_allowed_sources Method 7
B get_completion_vllm Function 7
B llama_cpp_chat Function 7
B LLMRouter.generate Method 7
B AlignmentService Class 6
B SkillExtractionService._normalize_mixed_concept_rows Method 6
B fetch_model_output Function 6
B LlamaCppBackend.__init__ Method 6
B GeminiAPI Class 6

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved issues remain in Gemini top-p propagation, offline alignment tests, and paper claims.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

This pull request makes LLM decoding deterministic by default, adds determinism and alignment tests, and updates documentation and the JOSS paper.

Changes:

  • Adds configurable temperature, top-p, and seed handling across backends.
  • Adds routing, backend-parameter, and alignment tests.
  • Updates README, documentation, Quickstart examples, and paper content.
File summaries
File Reviewed changes
tests/test_llm_router.py Updates router fixtures for decoding parameters.
tests/test_determinism.py Adds backend, routing, and alignment determinism tests.
README.md Documents decoding options and reproducibility.
pytest.ini Registers the determinism marker.
paper.md Revises summary, terminology, backend details, and reproducibility discussion.
mkdocs.yml Adds reproducibility documentation to navigation.
laiser/skill_extractor_refactored.py Exposes temperature and seed settings.
laiser/services.py Passes decoding settings to the router.
laiser/llm_models/openai.py Adds decoding parameters to requests.
laiser/llm_models/llm_router.py Forwards settings across backend routes.
laiser/llm_models/llama_cpp_handler.py Adds greedy defaults and optional seed handling.
laiser/llm_models/hugging_face_llm.py Controls sampling, seeds, and vLLM parameters.
laiser/llm_models/gemini.py Adds temperature and seed configuration.
laiser/llm_models/anthropic.py Adds temperature and top-p request parameters.
laiser/config.py Defines decoding defaults and environment overrides.
docs/reproducibility.md Documents reproducibility behavior and limitations.
docs/quickstart.md Updates deterministic example output.
docs/providers.md Links provider documentation to reproducibility guidance.
Review details

Suppressed comments (3)

docs/reproducibility.md:80

  • Nearest-neighbor alignment does not guarantee that different wordings select the same taxonomy entry; each phrase is embedded and searched independently. The documented behavior should say alignment can map wordings to a common entry, not that it does so, while retaining the caveat that missing model extractions cannot be recovered.
- **Different models extract different phrases.** Alignment maps different wordings of a skill onto the same
  taxonomy entry, but it cannot recover a skill a model never extracted. Compare models on the aligned columns,
  and measure their agreement rather than assuming it.

docs/reproducibility.md:86

  • The new alignment checks instantiate SentenceTransformer(DEFAULT_EMBEDDING_MODEL) on first use, so a clean runner may download the embedding weights and a no-network runner can fail. This statement is therefore false for the current test implementation; make the alignment fixture fully offline or revise the claim.
`tests/test_determinism.py` runs in CI without a GPU, a model download or an API key. It checks that:

paper.md:54

  • Exact nearest-neighbor search makes repeated alignment of one query deterministic, but it does not ensure that different phrasings independently embed to the same taxonomy entry. The implementation searches each phrase separately and the new tests only establish repeatability for identical inputs, so this sentence should be softened from a guarantee that phrasings "converge" to a possibility.
Because LLMs sample their output, identical inputs can yield different generations across runs, and different backends can yield different generations from the same input. LAiSER addresses this in three ways. First, decoding is deterministic by default: every backend decodes greedily at temperature zero, and a fixed seed is supplied to every backend that accepts one, so runs stay reproducible even when a user deliberately raises the temperature. Temperature and seed are constructor arguments, so sampled variation is available when wanted but is never the silent default. Second, the alignment stage acts as a normalizing projection. Generated phrases are embedded with a fixed sentence-transformer model and matched by exact search against a precomputed index, so surface variation in the generated text collapses onto a closed, finite vocabulary of taxonomy entries, and different phrasings of the same competency converge on the same aligned record. Third, every backend produces the same output schema, which makes results directly comparable across models and lets cross-model agreement be measured on the aligned columns. The test suite checks each backend's decoding defaults, the decoding parameters each backend actually receives, and that repeated alignment of a fixed input returns an identical result, similarity scores included.
  • Files reviewed: 18/18 changed files
  • Comments generated: 6
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread laiser/llm_models/gemini.py Resolved
Comment thread tests/test_determinism.py Resolved
Comment thread tests/test_determinism.py Resolved
Comment thread paper.md Resolved
Comment thread paper.md Resolved
Comment thread paper.md Resolved
phanindra-max and others added 2 commits September 15, 2026 02:10
Brings in 25c1314 (GPU extra dependencies and provider docs) so the provider documentation corrected here matches dev.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Code
- Gemini ignored LAISER_TOP_P: GenerateContentConfig was built without
  top_p, so the documented process-wide override reached every other
  backend but not Gemini. GeminiAPI and gemini_generate now take top_p
  (default DEFAULT_TOP_P) and send it; the Gemini request test asserts it.

Tests
- _import_or_skip skipped on any ImportError, so a broken import inside a
  backend module would have turned its tests into silent skips. It now
  skips only when ImportError.name names a genuinely optional package
  (google, vllm, llama_cpp) and re-raises everything else.
- The module docstring claimed no model download. The alignment tests use
  the real sentence-transformers embedding model, which Hugging Face
  downloads on first use, as tests/test_alignment_core.py already does. The
  claim is corrected rather than the model stubbed, because repeatable
  embeddings are part of what those tests check.

Docs and paper
- Reproducibility page: the same no-download claim is corrected, and
  alignment "can" map different wordings to one entry. Each phrase is
  matched independently, so that is likely rather than guaranteed.
- Paper and providers page: the GPU fallback has two steps.
  load_model_from_vllm retries DEFAULT_VLLM_MODEL_ID before the router
  falls back to Transformers; both now say so.
- Paper: preprocessing is described as the extraction prompts instructing
  the model to disregard branding, boilerplate and benefits, not as a
  separate removal step. Convergence of different phrasings is softened
  the same way as on the Reproducibility page.

Not changed: the State of the field claim that LAiSER supports SCQF
proficiency classification, which the public API does not currently
expose. That is left for the authors to decide.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@phanindra-max

Copy link
Copy Markdown
Member Author

Thanks for the review. Five of the inline threads are fixed in c1fd5b7 and have their own replies. The SCQF thread is still open: whether to remove or qualify that claim is a decision for the paper's authors.

The notes that appeared only in the review summary are also fixed in c1fd5b7:

  • Reproducibility page, different wordings — each phrase is embedded and searched independently, so alignment can map different wordings to a common entry, but does not guarantee it. The page now says so, and keeps the point that alignment cannot recover a skill a model never extracted.
  • Reproducibility page, no model download — the alignment checks use the real embedding model, which is downloaded on first use; the claim is corrected.
  • Paper, phrasings "converge" — softened: different phrasings can converge on the same aligned record, which is likely rather than guaranteed. The residual-variability paragraph now says alignment can normalize how a skill is expressed.

State of the field claimed LAiSER uniquely supports KSA decomposition with SCQF proficiency-level classification. The public API does not reach that capability: the KSA/SCQF prompt builders and parsers in services.py are never called, and extract_concepts(levels=...) has no effect. The sentence is removed, along with the related 'proficiency-level classification' and 'multi-taxonomy proficiency alignment' wording, so the paper only claims what the code does. Raised in Copilot's review of #444.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@phanindra-max

Copy link
Copy Markdown
Member Author

Tests that need a GPU will be handled in #445. Merging with fixes for everything else.

@phanindra-max
phanindra-max merged commit 11adcce into dev Sep 15, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants