Skip to content

fix: deterministic decoding by default; paper updates for JOSS review #424 - #446

Merged
phanindra-max merged 8 commits into
mainfrom
dev
Sep 15, 2026
Merged

phanindra-max merged 8 commits into
mainfrom
dev

Conversation

@phanindra-max

Copy link
Copy Markdown
Member

Brings the #424 work from dev to main. It was reviewed and merged into dev in #444, including fixes for everything Copilot raised there; this PR only promotes it. dev merges into main cleanly.

Addresses #424, the JOSS paper review. The issue stays open for the reviewer to close.

What reaches main

  • 494938a feat(llm): decode deterministically by default on every backend. Every backend defaults to temperature 0.0, with seed 42 wherever the backend accepts one. Both are exposed as temperature and seed on SkillExtractorRefactored. Previously most backends used their provider's default, typically temperature 1.0, and the local Transformers path followed the checkpoint's own config — which for Qwen2.5-0.5B-Instruct enables sampling. Adds tests/test_determinism.py.
  • 26a7b88 docs: document reproducible results and the decoding defaults. New Reproducibility page, and a README section with a reference configuration.
  • 5b7d302 docs(paper): respond to JOSS review [JOSS] Suggestions for paper improvement #424. Abbreviations are spelled out at first use, the Summary is rewritten for a non-technical reader, and the non-determinism discussion is restored.
  • 9e6ea34 docs(quickstart): show the greedy output for the local example.
  • c1fd5b7 fix: address review findings on fix: deterministic decoding by default; paper updates for JOSS review #424 #444.
    • Gemini now sends top_p.
    • The test import helper no longer turns real import failures into skips.
    • The no-download claim is corrected.
    • The GPU fallback's two steps are described accurately.
    • The preprocessing and phrase-convergence wording is made precise.
  • 05bc8bd docs(paper): remove the unsupported SCQF proficiency claim. The public API does not reach that capability.

Full detail, including the item-by-item mapping to #424 and the review responses, is in #444.

Verification (from #444)

  • black, isort and flake8 are clean.
  • Tests with -m "not index": 58 passed, 5 skipped (need API keys), 1 deselected.
  • mkdocs build --strict passes.
  • A real model on CPU with no API key, each run in a separate process:
    • Default decoding gave identical output across runs, similarity scores included.
    • At temperature 0.7, seed 7 gave identical text twice, and seed 8 gave different text.

After merge

Not included: the vLLM AWQ loading bug found during review is tracked separately in #445.

🤖 Generated with Claude Code

phanindra-max and others added 8 commits September 14, 2026 00:30
Re-lands the decoding half of #426, which #428 reverted, ported onto the
current router and with the problems its review found fixed. Addresses the
non-determinism question in JOSS review #424.

Every backend previously inherited its provider's default decoding: vLLM,
OpenAI and Anthropic sample at temperature 1.0, llama.cpp used 0.2, and a
Transformers model used whatever its checkpoint's generation_config.json
said. Qwen2.5-0.5B-Instruct, for example, ships do_sample=true and
temperature=0.7, so the local CPU path sampled even though nothing asked it
to.

- config: DEFAULT_TEMPERATURE = 0.0 and DEFAULT_TOP_P = 1.0, overridable via
  LAISER_TEMPERATURE / LAISER_TOP_P. GENERATION_SEED (42) already existed.
- Gemini, OpenAI, Anthropic, vLLM, llama.cpp and Transformers entry points
  all default to those values. The seed is sent to Gemini and llama.cpp only
  when the installed client accepts one, so older releases do not break.
- Transformers: at temperature 0.0, do_sample=False is passed explicitly so
  a checkpoint's own config cannot re-enable sampling. Above 0.0 the seed is
  applied before generation.
- SkillExtractorRefactored, SkillExtractionService and LLMRouter take
  temperature and seed and pass them through.

Fixed relative to #426:
- LLMRouter.generate always sent the router's seed to llama.cpp, ignoring a
  caller's seed. Per-call temperature= and seed= now reach every backend.
- #426 routed Transformers-loaded models to vLLM with no engine; that was
  fixed separately in #442, and decoding settings now follow both paths.
- Gemini's max_output_tokens is kept (#426 dropped it).
- The determinism tests read LAISER_TEMPERATURE at import, so a runner with
  that variable set failed them. They now reload config under a controlled
  environment, compare entry points against config rather than literals,
  and apply the `determinism` marker that #426 registered but never used.

tests/test_determinism.py: shipped defaults and env overrides; every public
entry point's temperature and seed defaults; the decoding parameters each
backend actually receives (OpenAI and Anthropic request bodies, Gemini
GenerateContentConfig, Transformers generate kwargs, vLLM SamplingParams,
llama.cpp with and without seed support); router forwarding and per-call
overrides on all five routes; extractor-to-router pass-through; and three
alignment-stability tests on the real skills index. tests/test_llm_router.py
is updated for the new keyword arguments.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- New Reproducibility page: per-backend temperature and seed, why sampling is
  switched off explicitly for Transformers, how to change the defaults, a
  reference configuration, what is not guaranteed (hosted models change,
  different models extract different phrases, hardware and library versions),
  and what tests/test_determinism.py checks.
- README: temperature and seed in the runtime options table, and a
  "Reproducible results" section with the reference configuration, which the
  reply on #424 described but which was never committed.
- Model providers page links to Reproducibility.

The API reference picks up the new temperature and seed parameters from the
docstrings.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Abbreviations. Every abbreviation is now spelled out at or before first use:
ESCO, LLM, AI, NER, FAISS, API, GPU, KSA, SCQF, MIT, GW and CCAS; spaCy is
described as a natural language processing library and EMSI as now
Lightcast. The ESCO and FAISS expansions had existed in earlier drafts but
were cut when the paper was shortened in be462ee.

Summary. Rewritten for a non-technical reader: it opens on the problem in
plain terms, explains what a skill taxonomy is before naming one, and says
who the software is for. Pipeline internals, the SCQF sentence, and the
licence and installation line move out of the Summary; the licence and
installation line now closes Software design, and the FAISS and SCQF
citations move to where those terms are now first used.

Non-determinism. Software design regains the explanation #428 removed, with
corrections: decoding is greedy with a fixed seed where accepted; alignment
projects generations onto a closed taxonomy vocabulary by exact search;
every backend shares one output schema; and residual variability is stated,
now including that hosted models change and providers do not guarantee
identical output at temperature zero.

Accuracy fixes made while rewriting those paragraphs:
- "ESCO's 13,000+ skill entries" replaced by the bundled figure, about 27,000
  skill entries across the four taxonomies.
- The backend list now includes llama.cpp and OpenAI where it was missing,
  and "automatic fallback" is stated precisely: on a GPU, a model vLLM cannot
  load falls back to Transformers.
- The output description no longer lists "optional SCQF proficiency levels",
  which extract_concepts does not produce.

Paper length is 1,628 words excluding references.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The example table came from a run in #442 where Qwen2.5-0.5B-Instruct was still sampling, because its generation_config.json enables sampling and nothing turned it off. With greedy decoding now the default, the same input returns three skills rather than two, identically on every run (verified in two separate processes). The table and the note beneath it now match.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Brings in 25c1314 (GPU extra dependencies and provider docs) so the provider documentation corrected here matches dev.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Code
- Gemini ignored LAISER_TOP_P: GenerateContentConfig was built without
  top_p, so the documented process-wide override reached every other
  backend but not Gemini. GeminiAPI and gemini_generate now take top_p
  (default DEFAULT_TOP_P) and send it; the Gemini request test asserts it.

Tests
- _import_or_skip skipped on any ImportError, so a broken import inside a
  backend module would have turned its tests into silent skips. It now
  skips only when ImportError.name names a genuinely optional package
  (google, vllm, llama_cpp) and re-raises everything else.
- The module docstring claimed no model download. The alignment tests use
  the real sentence-transformers embedding model, which Hugging Face
  downloads on first use, as tests/test_alignment_core.py already does. The
  claim is corrected rather than the model stubbed, because repeatable
  embeddings are part of what those tests check.

Docs and paper
- Reproducibility page: the same no-download claim is corrected, and
  alignment "can" map different wordings to one entry. Each phrase is
  matched independently, so that is likely rather than guaranteed.
- Paper and providers page: the GPU fallback has two steps.
  load_model_from_vllm retries DEFAULT_VLLM_MODEL_ID before the router
  falls back to Transformers; both now say so.
- Paper: preprocessing is described as the extraction prompts instructing
  the model to disregard branding, boilerplate and benefits, not as a
  separate removal step. Convergence of different phrasings is softened
  the same way as on the Reproducibility page.

Not changed: the State of the field claim that LAiSER supports SCQF
proficiency classification, which the public API does not currently
expose. That is left for the authors to decide.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
State of the field claimed LAiSER uniquely supports KSA decomposition with SCQF proficiency-level classification. The public API does not reach that capability: the KSA/SCQF prompt builders and parsers in services.py are never called, and extract_concepts(levels=...) has no effect. The sentence is removed, along with the related 'proficiency-level classification' and 'multi-taxonomy proficiency alignment' wording, so the paper only claims what the code does. Raised in Copilot's review of #444.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…coding

fix: deterministic decoding by default; paper updates for JOSS review #424
Copilot AI lite review requested due to automatic review settings September 15, 2026 06:28
@github-actions

Copy link
Copy Markdown

Complexity Analysis

Average: C — Moderate (13.75)

Hotspots tracked below. These are functional, well-exercised components — listed for visibility as the codebase scales.

Grade Name Type Score
F FAISSIndexManager.initialize_index Method 51
E SkillExtractionService.extract_and_align_core Method 36
E FAISSIndexManager.search_similar_skills Method 32
D ResponseParser._parse_skills_from_response Method 24
D FAISSIndexManager Class 22
C ResponseParser.parse_knowledge_task_response Method 20
C _BaseTaxonomyFAISSIndexManager.search_similar Method 18
C ResponseParser Class 17
C ResponseParser.parse_ksa_extraction_response Method 16
C ResponseParser.parse_skill_extraction_response Method 15
C openai_generate Function 14
C parse_output_vllm Function 13
C AlignmentService.align Method 11
C anthropic_generate Function 11
C LLMRouter._initialize_components Method 11
B SkillExtractionService._deduplicate Method 10
B load_model_from_vllm Function 10
B GeminiAPI.generate_with_config Method 10
B _BaseTaxonomyFAISSIndexManager.initialize_index Method 9
B ResponseParser.parse_ksa_details_response Method 8
B SkillExtractionService Class 8
B _BaseTaxonomyFAISSIndexManager Class 8
B llm_generate Function 8
B SkillExtractorRefactored._expand_allowed_sources Method 7
B get_completion_vllm Function 7
B llama_cpp_chat Function 7
B LLMRouter.generate Method 7
B AlignmentService Class 6
B SkillExtractionService._normalize_mixed_concept_rows Method 6
B fetch_model_output Function 6
B LlamaCppBackend.__init__ Method 6
B GeminiAPI Class 6

@github-actions

Copy link
Copy Markdown

Code Hygiene

The following items are defined but not yet in active use. Routine cleanup — no impact on functionality.

  • laiser/llm_methods.py:87 — unused variable 'args' (100% confidence)
  • laiser/llm_models/llama_cpp_handler.py:45 — unused variable 'chat_format' (100% confidence)
  • laiser/llm_models/llama_cpp_handler.py:86 — unused variable 'exc' (100% confidence)
  • laiser/llm_models/llama_cpp_handler.py:86 — unused variable 'exc_type' (100% confidence)
  • laiser/llm_models/llama_cpp_handler.py:86 — unused variable 'tb' (100% confidence)
  • laiser/llm_models/llm_router.py:71 — unused variable 'args' (100% confidence)
  • laiser/llm_models/llm_router.py:86 — unused variable 'args' (100% confidence)
  • laiser/llm_models/llm_router.py:89 — unused variable 'args' (100% confidence)
  • laiser/services.py:132 — unused variable 'raw_description' (100% confidence)

@phanindra-max
phanindra-max merged commit 86a2e32 into main Sep 15, 2026
34 checks passed

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Address the reproducibility caveats, model revision guidance, and determinism test coverage gaps.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Promotes deterministic decoding defaults and JOSS review updates from dev to main.

Changes:

  • Adds temperature and seed controls across LLM backends.
  • Adds determinism and alignment tests.
  • Updates documentation, quickstart content, and the JOSS paper.
File summaries
File Description
tests/test_llm_router.py Updates router test fixtures for decoding settings.
tests/test_determinism.py Adds decoding and alignment stability tests.
README.md Documents decoding options and reproducibility.
pytest.ini Registers the determinism marker.
paper.md Applies JOSS review revisions.
mkdocs.yml Adds reproducibility documentation to navigation.
laiser/skill_extractor_refactored.py Exposes temperature and seed controls.
laiser/services.py Propagates decoding settings through the service layer.
laiser/llm_models/openai.py Adds deterministic request parameters.
laiser/llm_models/llm_router.py Routes decoding settings to backends.
laiser/llm_models/llama_cpp_handler.py Adds decoding settings and compatibility handling.
laiser/llm_models/hugging_face_llm.py Controls Transformers and vLLM decoding.
laiser/llm_models/gemini.py Adds Gemini decoding configuration.
laiser/llm_models/anthropic.py Adds Anthropic temperature and top-p settings.
laiser/config.py Defines decoding defaults and environment overrides.
docs/reproducibility.md Documents defaults, limitations, and verification.
docs/quickstart.md Updates the local example output.
docs/providers.md Documents provider behavior and fallback order.
Review details

Suppressed comments (7)

README.md:196

  • These new public constructor options are added to the README, but docs/usage.md still omits temperature and seed even though the next paragraph says every option is described there. Add them to the usage guide or update the link/claim so users can discover the controls.
| `temperature` | decoding temperature for every backend (default `0.0`, greedy) |
| `seed` | seed for backends that accept one (default `42`) |

docs/reproducibility.md:72

  • The reproducibility guidance asks users to record/pin a Hugging Face model revision, but the loader always requests revision="main" and neither SkillExtractorRefactored nor the router exposes a revision argument. A later update under the same model ID can therefore change results even when users follow this guidance; expose and forward a revision, or document a supported way to load the recorded revision.
- **the LAiSER version** — it fixes the bundled taxonomy data and the embedding model used for alignment,
  `sentence-transformers/all-MiniLM-L6-v2`;
- **the model revision**, for Hugging Face models that may be updated under the same name.

docs/reproducibility.md:4

  • The opening says LAiSER prevents repeated runs from differing, but the caveat below correctly notes that provider, hardware, and library changes can still alter results. Describe the defaults as improving reproducibility rather than guaranteeing it.
Language models generate text by sampling, so left to their providers' defaults two runs over the same input
can return different skills. LAiSER is configured so that, by default, they do not.

laiser/llm_models/llama_cpp_handler.py:106

  • The new LlamaCppBackend.generate wrapper resolves the constructor defaults and forwards per-call settings, but the tests only exercise llama_cpp_chat directly. Add a direct-wrapper test for both the default and override paths so this public backend cannot regress while the lower-level helper tests continue to pass.
        temperature = self.temperature if temperature is None else temperature
        seed = self.seed if seed is None else seed

paper.md:54

  • The paper says a fixed seed makes runs reproducible even when temperature is raised, then immediately says hosted providers do not guarantee identical output. The code also omits a seed for greedy Transformers generation. Qualify this as improved/best-effort reproducibility rather than a guarantee.
Because LLMs sample their output, identical inputs can yield different generations across runs, and different backends can yield different generations from the same input. LAiSER addresses this in three ways. First, decoding is deterministic by default: every backend decodes greedily at temperature zero, and a fixed seed is supplied to every backend that accepts one, so runs stay reproducible even when a user deliberately raises the temperature. Temperature and seed are constructor arguments, so sampled variation is available when wanted but is never the silent default. Second, the alignment stage acts as a normalizing projection. Generated phrases are embedded with a fixed sentence-transformer model and matched by exact search against a precomputed index, so surface variation in the generated text collapses onto a closed, finite vocabulary of taxonomy entries, and different phrasings of the same competency can converge on the same aligned record, although because each phrase is matched independently this is likely rather than guaranteed. Third, every backend produces the same output schema, which makes results directly comparable across models and lets cross-model agreement be measured on the aligned columns. The test suite checks each backend's decoding defaults, the decoding parameters each backend actually receives, and that repeated alignment of a fixed input returns an identical result, similarity scores included.

tests/test_determinism.py:459

  • Skipping an empty result makes this stability test pass when alignment returns no rows, so it does not protect the claimed taxonomy behavior. The fixture already handles an unavailable index; assert that these fixed common phrases produce output instead.
    if aligned.empty:  # pragma: no cover - environment dependent
        pytest.skip("Alignment returned no rows")

tests/test_determinism.py:473

  • This skip has the same problem as the earlier alignment test: a regression that makes repeated phrases produce no matches is reported as skipped rather than failed. Assert non-empty output so the per-phrase determinism checks actually run.
    if aligned.empty:  # pragma: no cover - environment dependent
        pytest.skip("Alignment returned no rows")
  • Files reviewed: 18/18 changed files
  • Comments generated: 4
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread README.md

### Reproducible results

Every backend decodes greedily by default (temperature `0.0`) and uses a fixed seed where the backend accepts one, so repeated runs over the same input give the same result. To make results reproducible for others, pin every setting that affects them:
Comment thread docs/reproducibility.md
Comment on lines +8 to +10
Every backend decodes greedily: at temperature `0.0` the model always takes its most probable next token, so
the same prompt produces the same text. A fixed seed (`42`) is also supplied wherever a backend accepts one,
which keeps runs reproducible if you deliberately raise the temperature.
Comment thread docs/reproducibility.md
Comment on lines +87 to +88
`tests/test_determinism.py` runs in CI without a GPU, an API key or a language model download. The alignment
checks use the real embedding model, which Hugging Face downloads on first use. It checks that:
Comment thread tests/test_determinism.py
Comment on lines +3 to +5
Language models sample their output, so the same input can produce different
generations across runs and across backends (JOSS review #424). LAiSER limits
this in two ways, and these tests check both without a GPU, an API key or a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants