You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Here's a list of feedback points regarding the paper:
I find the use of the term alignment a bit surprising, is this commonly used when matching a taxonomy?
"The challenge is compounded by the absence of integrated tooling that combines skill identification with alignment to recognized taxonomies, forcing researchers to assemble fragile multi-tool pipelines." [L26] this claim seems not well supported?
The paper does not seem represent to the state of the field very well, with several other projects combining LLMs and skill extraction. From a quick search there seem to be multiple works that should be referenced (judging only by their abstracts).
Parts of the paper read very AI-generated, esp. the Software Design section e.g. L90+. This is not just about style (and respect for people reading the paper), but it also negatively affects the readability of the paper. That section should be rewritten to more clearly describe what actually happens.
I want to give the benefit of the doubt here, but the AI statement "The paper text was drafted by the authors with AI-assisted editing for clarity." feels a bit misleading here and it feels much more like at least portions of the paper were AI-generated? Curious for @muctadir 's experience regarding this.
The research impact statement is strong!
Please link the cookbook repository mentioned in the paper
L29 LAiSER claims accurate extraction, but there does not seem to be any evaluation of the tool available? I think an evaluation could be very helpful, but it's also not necessarily required for a paper like this, but then the language should be watered down here.
And one minor issue I noticed in the documentation:
Update: I have addressed some of your points in #450: defining alignment, removing the unsupported claims, adding the references you suggested, linking the cookbook, toning down the accuracy claim, and the Open Skills Network link.
The rest of the list is taking me a bit longer than I expected. I will post an update here as soon as those are ready.
One small addition: the old Open Skills Network domain appears to have lapsed and now redirects to an unrelated gambling site, so please don't follow the original link. The documentation now links to the Open Skills Management Tool repository (https://github.com/CredentialEngine/osmt) instead. The individual skill links in LAiSER's output still open their public skill records.
@jansim thank you again for the careful review. The remaining items from this issue are addressed in #460.
State of the field. The section now compares LAiSER with the LLM-based tools, based on reading those papers. SkillGPT maps to ESCO only. Skill-LLM and Nguyen et al. extract spans but do not map them to any taxonomy. LAiSER combines LLM extraction with mapping to four taxonomies: ESCO, O*NET, the UK Skills Classification and OSN. The section also explains why we built LAiSER rather than extending an existing tool: we started from Nesta's library and moved away from it after seeing domain errors, such as "Apache Airflow" matching HVAC airflow. While checking the references, we found that esco-skill-extractor and ESCOX are the same tool, so they are now described once and cited together. Finally, the section states plainly that LAiSER has no published evaluation.
Software design. We rewrote this section from our own answers to four questions:
what happens to a document;
why we prompt an LLM instead of training an NER model;
why matching uses exact search;
what the reproducibility measures do and do not guarantee.
Every claim was checked against the code. That check exposed one real bug: syllabi were sent to a placeholder prompt that contained no document text. It is fixed in the same PR, and a test now covers it.
AI usage disclosure. You were right that the previous statement understated the AI involvement. The new disclosure follows the JOSS AI policy. It names each tool and model version, says which parts of the code, documentation and paper each was used for, and includes our statement of human review. It states that earlier versions of the paper were drafted with Claude. It also describes how this revision was produced: we answered structured questions for each section, and Claude Code assembled our answers into paragraphs, checked claims against the code, and drafted the limitations paragraph, which we then revised.
The paper is now 1,748 words. The Statement of need was also rewritten, because it had the same style problems you pointed out in Software design.
As @atrisovic asked in the review thread, here is each point from this issue and the change that addresses it. All of these changes are merged into main, the branch the paper is built from.
SkillGPT, ESCOX, Skill-LLM and Nguyen et al. are cited (#450). The section now also states how LAiSER differs from them, why we built LAiSER rather than extending an existing library, and that esco-skill-extractor and ESCOX are the same tool, so they are now described once (#460)
Activity
jansim commentedon Sep 15, 2026
Related to openjournals/joss-reviews#10421
Merge pull request #450 from LAiSER-Software/fix/447-paper-mechanical
phanindra-max commentedon Oct 4, 2026
@jansim thank you so much for taking the time.
Update: I have addressed some of your points in #450: defining alignment, removing the unsupported claims, adding the references you suggested, linking the cookbook, toning down the accuracy claim, and the Open Skills Network link.
The rest of the list is taking me a bit longer than I expected. I will post an update here as soon as those are ready.
Thank you for your patience.
phanindra-max commentedon Oct 4, 2026
One small addition: the old Open Skills Network domain appears to have lapsed and now redirects to an unrelated gambling site, so please don't follow the original link. The documentation now links to the Open Skills Management Tool repository (https://github.com/CredentialEngine/osmt) instead. The individual skill links in LAiSER's output still open their public skill records.
chore(paper): start #447 rewrite branch
phanindra-max commentedon Nov 7, 2026
@jansim thank you again for the careful review. The remaining items from this issue are addressed in #460.
State of the field. The section now compares LAiSER with the LLM-based tools, based on reading those papers. SkillGPT maps to ESCO only. Skill-LLM and Nguyen et al. extract spans but do not map them to any taxonomy. LAiSER combines LLM extraction with mapping to four taxonomies: ESCO, O*NET, the UK Skills Classification and OSN. The section also explains why we built LAiSER rather than extending an existing tool: we started from Nesta's library and moved away from it after seeing domain errors, such as "Apache Airflow" matching HVAC airflow. While checking the references, we found that esco-skill-extractor and ESCOX are the same tool, so they are now described once and cited together. Finally, the section states plainly that LAiSER has no published evaluation.
Software design. We rewrote this section from our own answers to four questions:
Every claim was checked against the code. That check exposed one real bug: syllabi were sent to a placeholder prompt that contained no document text. It is fixed in the same PR, and a test now covers it.
AI usage disclosure. You were right that the previous statement understated the AI involvement. The new disclosure follows the JOSS AI policy. It names each tool and model version, says which parts of the code, documentation and paper each was used for, and includes our statement of human review. It states that earlier versions of the paper were drafted with Claude. It also describes how this revision was produced: we answered structured questions for each section, and Claude Code assembled our answers into paragraphs, checked claims against the code, and drafted the limitations paragraph, which we then revised.
The paper is now 1,748 words. The Statement of need was also rewritten, because it had the same style problems you pointed out in Software design.
Merge pull request #460 from LAiSER-Software/docs/447-paper-rewrite
phanindra-max commentedon Nov 7, 2026
As @atrisovic asked in the review thread, here is each point from this issue and the change that addresses it. All of these changes are merged into
main, the branch the paper is built from.