Skip to content

[JOSS] Paper Feedback #447

Open
Open

Description

@jansim

Here's a list of feedback points regarding the paper:

  • I find the use of the term alignment a bit surprising, is this commonly used when matching a taxonomy?
  • "The challenge is compounded by the absence of integrated tooling that combines skill identification with alignment to recognized taxonomies, forcing researchers to assemble fragile multi-tool pipelines." [L26] this claim seems not well supported?
  • The paper does not seem represent to the state of the field very well, with several other projects combining LLMs and skill extraction. From a quick search there seem to be multiple works that should be referenced (judging only by their abstracts).
  • Parts of the paper read very AI-generated, esp. the Software Design section e.g. L90+. This is not just about style (and respect for people reading the paper), but it also negatively affects the readability of the paper. That section should be rewritten to more clearly describe what actually happens.
    • I want to give the benefit of the doubt here, but the AI statement "The paper text was drafted by the authors with AI-assisted editing for clarity." feels a bit misleading here and it feels much more like at least portions of the paper were AI-generated? Curious for @muctadir 's experience regarding this.
  • The research impact statement is strong!
  • Please link the cookbook repository mentioned in the paper
  • L29 LAiSER claims accurate extraction, but there does not seem to be any evaluation of the tool available? I think an evaluation could be very helpful, but it's also not necessarily required for a paper like this, but then the language should be watered down here.

And one minor issue I noticed in the documentation:

Activity

  1. jansim commented on Sep 15, 2026

    @jansim
    Author
  2. pinned this issue on Oct 2, 2026
  3. added a commit that references this issue on Oct 4, 2026
  4. phanindra-max commented on Oct 4, 2026

    @phanindra-max
    Member

    @jansim thank you so much for taking the time.

    Update: I have addressed some of your points in #450: defining alignment, removing the unsupported claims, adding the references you suggested, linking the cookbook, toning down the accuracy claim, and the Open Skills Network link.

    The rest of the list is taking me a bit longer than I expected. I will post an update here as soon as those are ready.

    Thank you for your patience.

  5. phanindra-max commented on Oct 4, 2026

    @phanindra-max
    Member

    One small addition: the old Open Skills Network domain appears to have lapsed and now redirects to an unrelated gambling site, so please don't follow the original link. The documentation now links to the Open Skills Management Tool repository (https://github.com/CredentialEngine/osmt) instead. The individual skill links in LAiSER's output still open their public skill records.

  6. added a commit that references this issue on Oct 10, 2026
  7. phanindra-max commented on Nov 7, 2026

    @phanindra-max
    Member

    @jansim thank you again for the careful review. The remaining items from this issue are addressed in #460.

    State of the field. The section now compares LAiSER with the LLM-based tools, based on reading those papers. SkillGPT maps to ESCO only. Skill-LLM and Nguyen et al. extract spans but do not map them to any taxonomy. LAiSER combines LLM extraction with mapping to four taxonomies: ESCO, O*NET, the UK Skills Classification and OSN. The section also explains why we built LAiSER rather than extending an existing tool: we started from Nesta's library and moved away from it after seeing domain errors, such as "Apache Airflow" matching HVAC airflow. While checking the references, we found that esco-skill-extractor and ESCOX are the same tool, so they are now described once and cited together. Finally, the section states plainly that LAiSER has no published evaluation.

    Software design. We rewrote this section from our own answers to four questions:

    • what happens to a document;
    • why we prompt an LLM instead of training an NER model;
    • why matching uses exact search;
    • what the reproducibility measures do and do not guarantee.

    Every claim was checked against the code. That check exposed one real bug: syllabi were sent to a placeholder prompt that contained no document text. It is fixed in the same PR, and a test now covers it.

    AI usage disclosure. You were right that the previous statement understated the AI involvement. The new disclosure follows the JOSS AI policy. It names each tool and model version, says which parts of the code, documentation and paper each was used for, and includes our statement of human review. It states that earlier versions of the paper were drafted with Claude. It also describes how this revision was produced: we answered structured questions for each section, and Claude Code assembled our answers into paragraphs, checked claims against the code, and drafted the limitations paragraph, which we then revised.

    The paper is now 1,748 words. The Statement of need was also rewritten, because it had the same style problems you pointed out in Software design.

  8. added a commit that references this issue on Nov 7, 2026
  9. phanindra-max commented on Nov 7, 2026

    @phanindra-max
    Member

    As @atrisovic asked in the review thread, here is each point from this issue and the change that addresses it. All of these changes are merged into main, the branch the paper is built from.

    Point Change PR
    The term "alignment" Defined at first use in the Summary #450
    The unsupported claim at L26 about fragile multi-tool pipelines Removed #450
    The state of the field, including LLM-based work SkillGPT, ESCOX, Skill-LLM and Nguyen et al. are cited (#450). The section now also states how LAiSER differs from them, why we built LAiSER rather than extending an existing library, and that esco-skill-extractor and ESCOX are the same tool, so they are now described once (#460) #450, #460
    The Software design section reads as AI-generated Rewritten from the authors' answers and checked against the code. The Statement of need, which had the same problem, is also rewritten #460
    The AI usage disclosure Rewritten to the JOSS AI policy: each tool and model version, what it was used for, and how this revision was produced #460
    The research impact statement No change —
    Link to the cookbook repository Added in the Research impact statement #450
    The "accurate extraction" claim at L29 Removed. The paper now states that LAiSER has no published evaluation and recommends spot-checking results #450, #460
    The broken openskillsnetwork.org link The domain has lapsed, so the documentation now links to the Open Skills Management Tool repository #450, #457
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions