You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
What are spaCy, NER, ESCO? Please, provide the elaborate form before using abbreviations.
Summary:
Too technical. Focus on the high level functionality and purpose. This should be understandable to people without any technical background.
Software design:
LLMs are often non-deterministic, meaning running the same data through more than once through the same LLM might result into different results. Also, you support multiple LLMs, each of which might provide a different result. How are these risks addressed? Can you make some comments on that?
Thank you, three separate points, addressed in turn.
Abbreviations. ESCO, FAISS, NER and spaCy are now spelled out at first use. KSA, SCQF and LLM already were. We swept the manuscript for any remaining bare abbreviation.
Summary. Rewritten for a non-technical reader. It opens on the problem in plain language, explains what a skill taxonomy is before naming one, says who the tool is for, and moves pipeline internals and installation to later sections.
Non-determinism. Two mechanisms:
Decoding. Every backend now defaults to greedy decoding at temperature zero, with a fixed seed where the backend accepts one: vLLM, llama.cpp, Gemini, local Transformers. OpenAI and Anthropic accept no seed, so there reproducibility rests on greedy decoding alone, and we say so. Temperature and seed are constructor arguments, so sampling is an explicit choice. Previously several backends inherited their provider's default of 1.0.
Alignment. Generated text is never the final output. Each phrase is embedded and matched against a precomputed taxonomy index, so different phrasings of the same competency collapse onto the same canonical entry with a stable identifier. Every backend ends in the same schema, which makes models comparable on the aligned columns.
tests/test_determinism.py asserts the decoding defaults, the request bodies actually sent, the router's forwarding, and that repeated alignment runs return an identical frame. The README documents a reference configuration pinning model, seed, threshold and embedding model.
What this does not fix. Alignment cannot recover a skill a model never emitted, so the set of extracted skills can still differ across models. We say so in the paper, and recommend pinning model, seed and threshold when reporting results.
My previous reply described the repository as already containing these changes, and when you looked it would not have. Deterministic decoding and its tests had been merged in #426 but were later reverted in #428, and the abbreviation and Summary edits had not been committed. That was my error. All three points are now addressed in #444, where each change below can be checked.
Abbreviations. Every abbreviation is spelled out at or before its first use: ESCO, LLM, AI, NER, FAISS, API, GPU, MIT, GW and CCAS. spaCy is introduced as a natural language processing library.
Summary. Rewritten for a non-technical reader. It opens with the problem in plain terms, explains what a skill taxonomy is before naming one, and says who the software is for. Pipeline internals and installation details moved to later sections.
Non-determinism. Addressed in three ways, which the paper's Software design section now describes:
Decoding. Every backend defaults to greedy decoding at temperature zero, with a fixed seed wherever the backend accepts one (vLLM, llama.cpp, Gemini, and local Transformers models). For local models, sampling is switched off explicitly, because some checkpoints enable it in their own configuration. Temperature and seed are constructor arguments, so sampling is an explicit choice. Previously most backends used their provider's default, typically temperature 1.0.
Alignment. Generated text is not the final output. Each extracted phrase is embedded and matched by exact search against a fixed taxonomy index, so the same phrase always resolves to the same taxonomy entry with the same similarity score. Different phrasings of the same skill can resolve to the same entry too, although because each phrase is matched independently that is likely rather than guaranteed.
One schema. Every backend produces the same output columns, so results from different models can be compared directly on the aligned columns.
tests/test_determinism.py runs in CI. It checks the decoding defaults, the decoding parameters each backend actually receives, that per-call overrides reach every backend, and that repeated alignment returns an identical result. Running the same extraction twice with a local model, in separate processes, returns identical results, similarity scores included. A new Reproducibility page in the documentation and a section in the README give a reference configuration that pins the model, temperature, seed and thresholds.
What this does not fix. Alignment cannot recover a skill that a model never extracts, so the set of extracted skills can still differ between models. Hosted providers can also change the model behind an API and do not guarantee identical output even at temperature zero. The paper states both, and recommends pinning the model, temperature, seed and thresholds when reporting results.
Activity
fix: default all LLM backends to deterministic greedy decoding
fix(llm): route local generation to the backend that was actually loaded
docs(paper): make the summary accessible and align backend claims wit…
phanindra-max commentedon Aug 4, 2026
Thank you, three separate points, addressed in turn.
Abbreviations. ESCO, FAISS, NER and spaCy are now spelled out at first use. KSA, SCQF and LLM already were. We swept the manuscript for any remaining bare abbreviation.
Summary. Rewritten for a non-technical reader. It opens on the problem in plain language, explains what a skill taxonomy is before naming one, says who the tool is for, and moves pipeline internals and installation to later sections.
Non-determinism. Two mechanisms:
tests/test_determinism.pyasserts the decoding defaults, the request bodies actually sent, the router's forwarding, and that repeated alignment runs return an identical frame. TheREADMEdocuments a reference configuration pinning model, seed, threshold and embedding model.What this does not fix. Alignment cannot recover a skill a model never emitted, so the set of extracted skills can still differ across models. We say so in the paper, and recommend pinning model, seed and threshold when reporting results.
phanindra-max commentedon Sep 10, 2026
@muctadir just checking on this
phanindra-max commentedon Sep 15, 2026
@muctadir — a correction first, then an update.
My previous reply described the repository as already containing these changes, and when you looked it would not have. Deterministic decoding and its tests had been merged in #426 but were later reverted in #428, and the abbreviation and Summary edits had not been committed. That was my error. All three points are now addressed in #444, where each change below can be checked.
Abbreviations. Every abbreviation is spelled out at or before its first use: ESCO, LLM, AI, NER, FAISS, API, GPU, MIT, GW and CCAS. spaCy is introduced as a natural language processing library.
Summary. Rewritten for a non-technical reader. It opens with the problem in plain terms, explains what a skill taxonomy is before naming one, and says who the software is for. Pipeline internals and installation details moved to later sections.
Non-determinism. Addressed in three ways, which the paper's Software design section now describes:
tests/test_determinism.pyruns in CI. It checks the decoding defaults, the decoding parameters each backend actually receives, that per-call overrides reach every backend, and that repeated alignment returns an identical result. Running the same extraction twice with a local model, in separate processes, returns identical results, similarity scores included. A new Reproducibility page in the documentation and a section in the README give a reference configuration that pins the model, temperature, seed and thresholds.What this does not fix. Alignment cannot recover a skill that a model never extracts, so the set of extracted skills can still differ between models. Hosted providers can also change the model behind an API and do not guarantee identical output even at temperature zero. The paper states both, and recommends pinning the model, temperature, seed and thresholds when reporting results.
Merge pull request #444 from LAiSER-Software/fix/424-deterministic-de…
Merge pull request #446 from LAiSER-Software/dev