Skip to content

[JOSS] Suggestions for paper improvement #424

Open

Description

@muctadir

What are spaCy, NER, ESCO? Please, provide the elaborate form before using abbreviations.

Summary:
Too technical. Focus on the high level functionality and purpose. This should be understandable to people without any technical background.

Software design:
LLMs are often non-deterministic, meaning running the same data through more than once through the same LLM might result into different results. Also, you support multiple LLMs, each of which might provide a different result. How are these risks addressed? Can you make some comments on that?

openjournals/joss-reviews#10421

Activity

  1. phanindra-max commented on Aug 4, 2026

    @phanindra-max
    Member

    Thank you, three separate points, addressed in turn.

    • Abbreviations. ESCO, FAISS, NER and spaCy are now spelled out at first use. KSA, SCQF and LLM already were. We swept the manuscript for any remaining bare abbreviation.

    • Summary. Rewritten for a non-technical reader. It opens on the problem in plain language, explains what a skill taxonomy is before naming one, says who the tool is for, and moves pipeline internals and installation to later sections.

    Non-determinism. Two mechanisms:

    1. Decoding. Every backend now defaults to greedy decoding at temperature zero, with a fixed seed where the backend accepts one: vLLM, llama.cpp, Gemini, local Transformers. OpenAI and Anthropic accept no seed, so there reproducibility rests on greedy decoding alone, and we say so. Temperature and seed are constructor arguments, so sampling is an explicit choice. Previously several backends inherited their provider's default of 1.0.
    2. Alignment. Generated text is never the final output. Each phrase is embedded and matched against a precomputed taxonomy index, so different phrasings of the same competency collapse onto the same canonical entry with a stable identifier. Every backend ends in the same schema, which makes models comparable on the aligned columns.

    tests/test_determinism.py asserts the decoding defaults, the request bodies actually sent, the router's forwarding, and that repeated alignment runs return an identical frame. The README documents a reference configuration pinning model, seed, threshold and embedding model.

    What this does not fix. Alignment cannot recover a skill a model never emitted, so the set of extracted skills can still differ across models. We say so in the paper, and recommend pinning model, seed and threshold when reporting results.

  2. phanindra-max commented on Sep 10, 2026

    @phanindra-max
    Member

    @muctadir just checking on this

  3. phanindra-max commented on Sep 15, 2026

    @phanindra-max
    Member

    @muctadir — a correction first, then an update.

    My previous reply described the repository as already containing these changes, and when you looked it would not have. Deterministic decoding and its tests had been merged in #426 but were later reverted in #428, and the abbreviation and Summary edits had not been committed. That was my error. All three points are now addressed in #444, where each change below can be checked.

    Abbreviations. Every abbreviation is spelled out at or before its first use: ESCO, LLM, AI, NER, FAISS, API, GPU, MIT, GW and CCAS. spaCy is introduced as a natural language processing library.

    Summary. Rewritten for a non-technical reader. It opens with the problem in plain terms, explains what a skill taxonomy is before naming one, and says who the software is for. Pipeline internals and installation details moved to later sections.

    Non-determinism. Addressed in three ways, which the paper's Software design section now describes:

    1. Decoding. Every backend defaults to greedy decoding at temperature zero, with a fixed seed wherever the backend accepts one (vLLM, llama.cpp, Gemini, and local Transformers models). For local models, sampling is switched off explicitly, because some checkpoints enable it in their own configuration. Temperature and seed are constructor arguments, so sampling is an explicit choice. Previously most backends used their provider's default, typically temperature 1.0.
    2. Alignment. Generated text is not the final output. Each extracted phrase is embedded and matched by exact search against a fixed taxonomy index, so the same phrase always resolves to the same taxonomy entry with the same similarity score. Different phrasings of the same skill can resolve to the same entry too, although because each phrase is matched independently that is likely rather than guaranteed.
    3. One schema. Every backend produces the same output columns, so results from different models can be compared directly on the aligned columns.

    tests/test_determinism.py runs in CI. It checks the decoding defaults, the decoding parameters each backend actually receives, that per-call overrides reach every backend, and that repeated alignment returns an identical result. Running the same extraction twice with a local model, in separate processes, returns identical results, similarity scores included. A new Reproducibility page in the documentation and a section in the README give a reference configuration that pins the model, temperature, seed and thresholds.

    What this does not fix. Alignment cannot recover a skill that a model never extracts, so the set of extracted skills can still differ between models. Hosted providers can also change the model behind an API and do not guarantee identical output even at temperature zero. The paper states both, and recommends pinning the model, temperature, seed and thresholds when reporting results.

  4. added 2 commits that reference this issue on Sep 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions