Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 67 additions & 0 deletions .github/workflows/docs.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
name: Docs

# Documentation lives in this repository, so it is reviewed and versioned with
# the code it describes. Every PR that could change it builds the site in strict
# mode; merges to main publish it to GitHub Pages.
on:
pull_request:
paths:
- "docs/**"
- "mkdocs.yml"
- "laiser/**" # the API reference is generated from docstrings
- ".github/workflows/docs.yml"
push:
branches:
- main
paths:
- "docs/**"
- "mkdocs.yml"
- "laiser/**"
- ".github/workflows/docs.yml"
workflow_dispatch:

permissions:
contents: read

concurrency:
group: docs-${{ github.ref }}
cancel-in-progress: true

jobs:
build:
name: Build docs
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- uses: actions/setup-python@v5
with:
python-version: "3.11"

- name: Install docs toolchain
run: pip install -r docs/requirements.txt

- name: Build (strict)
# Broken links, missing anchors and unresolved API references fail here.
run: mkdocs build --strict

- name: Upload site
if: github.event_name != 'pull_request' && github.ref == 'refs/heads/main'
uses: actions/upload-pages-artifact@v3
with:
path: site

deploy:
name: Deploy to GitHub Pages
if: github.event_name != 'pull_request' && github.ref == 'refs/heads/main'
needs: build
runs-on: ubuntu-latest
permissions:
pages: write
id-token: write
environment:
name: github-pages
url: ${{ steps.deploy.outputs.page_url }}
steps:
- id: deploy
uses: actions/deploy-pages@v4
117 changes: 79 additions & 38 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,48 +5,50 @@
<div align="center">
<img src="https://i.imgur.com/XznvjNi.png" width="70%"/>
<h1>Leveraging ​Artificial ​Intelligence for ​Skill ​Extraction &​ Research (LAiSER)</h1>

<a href="https://laiser-software.github.io/extract-module/"><b>Documentation</b></a> ·
<a href="https://pypi.org/project/laiser/"><b>PyPI</b></a> ·
<a href="https://github.com/LAiSER-Software/laiser-cookbook"><b>Cookbook notebooks</b></a>
</div>

LAiSER turns free text such as job postings and course syllabi into structured skills, knowledge and tasks, each matched to an entry in ESCO, O\*NET, the UK Skills Classification or the Open Skills Network.

### Contents
LAiSER is a tool that helps learners, educators and employers share trusted and mutually intelligible information about skills​.

- [About](#about)
- [Architecture](#architecture)
- [Requirements](#requirements)
- [Setup and Installation](#setup-and-installation)
- [Usage](#usage)
- [Try it in Google Colab](#try-it-in-google-colab)
- [Funding](#funding)
- [Authors](#authors)
- [Partners](#partners)

## About

LAiSER is an innovative tool that harnesses the power of artificial intelligence to simplify the extraction and analysis of skills. It is designed for learners, educators, and employers who want to gain reliable insights into skill sets, ensuring that the information shared is both trusted and mutually intelligible across various sectors.
LAiSER is a Python package for turning unstructured text about work and learning into structured, comparable skill data.

You give it a table of documents — job postings, course descriptions, syllabi. For each document, a language model extracts the skills it describes and, optionally, the knowledge and tasks behind them. LAiSER then matches every extracted phrase to its closest entries in established taxonomies using embedding similarity. The result is a table with one row per match: the phrase as written, the taxonomy entry it corresponds to, which taxonomy that entry comes from, and a similarity score.

By leveraging state-of-the-art AI models, LAiSER automates the process of identifying and classifying skills from diverse data sources. This not only saves time but also enhances accuracy, making it easier for users to discover emerging trends and in-demand skills.
Because results point at shared taxonomy entries rather than free-text phrases, they can be counted and compared across sources — the skills employers ask for can be set against the skills a program teaches, or tracked across thousands of postings.

The tool emphasizes standardization and transparency, offering a common framework that bridges the communication gap between different stakeholders. With LAiSER, educators can better align their teaching methods with industry requirements, and employers can more effectively identify the competencies required for their teams. The result is a more efficient and strategic approach to skill development, benefiting the entire ecosystem.
The language model can be a hosted API (Gemini or OpenAI) or a model running on your own machine, including on CPU with no API key. Taxonomy alignment always runs locally against indexes bundled with the package.

## Architecture

LAiSER uses a four-stage extraction and alignment pipeline:
LAiSER runs four stages for every document:

1. Extraction
Input text is normalized by input type and passed through prompt construction and LLM inference to produce raw concept candidates.
2. Parsing and deduplication
Model output is parsed into structured concepts and filtered through exact and semantic deduplication.
3. Taxonomy alignment
Extracted concepts are matched against bundled taxonomy indexes using embedding-based similarity search and threshold filtering.
4. Output normalization
Alignment results are converted into a unified tabular schema, with optional edge generation for graph-style outputs.
1. **Extraction** — The text is placed into a prompt for its input type and sent to the language model, which returns candidate concepts.
2. **Parsing and deduplication** — The response is parsed into phrases. Exact duplicates are dropped and near-duplicates are collapsed by embedding similarity.
3. **Taxonomy alignment** — Each phrase is embedded and searched against bundled FAISS indexes of taxonomy entries. Matches above a per-type similarity threshold are kept.
4. **Output** — Matches are returned as a single table, optionally with graph edges linking knowledge areas to the tasks they enable.

## Requirements
- Python version `>=3.10`.
- The package supports the current tested matrix through Python `3.13`.
- A GPU is recommended for heavy local model workflows, but API-backed extraction can run CPU-only.
- Provider-specific environment variables may be required depending on backend:
- `GEMINI_API_KEY` or `GOOGLE_API_KEY`
- `OPENAI_API_KEY`

- Python `>=3.10`; CI tests 3.10 through 3.13.
- No GPU or API key is required. A GPU speeds up larger local models; hosted providers need their own API key.
- Supported model providers and their settings are listed in the [documentation](https://laiser-software.github.io/extract-module/providers/).

## Setup and Installation

Expand All @@ -68,9 +70,8 @@ LAiSER uses a four-stage extraction and alignment pipeline:
pip install -e ".[dev]"
```

**NOTE**: Python 3.10 or later is required. Python 3.12 or 3.13 is recommended for current development and CI parity.

You can check if your machine has a GPU available with:

```shell
python -c "import torch; print(torch.cuda.is_available())"
```
Expand All @@ -79,7 +80,40 @@ python -c "import torch; print(torch.cuda.is_available())"

LAiSER is used as a Python package. The recommended API is `SkillExtractorRefactored`.

### Basic job description extraction
### Without an API key

This runs a small open model locally on CPU. The model downloads on first use.

```python
import pandas as pd

from laiser.skill_extractor_refactored import SkillExtractorRefactored

data = pd.DataFrame(
[
{
"Research ID": "job-001",
"description": "Build production machine learning systems in Python.",
}
]
)

extractor = SkillExtractorRefactored(model_id="Qwen/Qwen2.5-0.5B-Instruct", use_gpu=False)

results = extractor.extract_concepts(
data=data,
id_column="Research ID",
text_columns=["description"],
input_type="job_desc",
concepts=["skills"],
)

print(results[["Raw Concept", "Taxonomy Concept", "Taxonomy Source", "Correlation Coefficient"]])
```

Small local models are convenient for trying LAiSER out; hosted or larger models extract more reliably.

### Job description extraction with Gemini

```python
import os
Expand Down Expand Up @@ -152,22 +186,29 @@ print(results.head())

### Common runtime options

- `model_id`
Provider or model selector such as `gemini` or `openai`
- `api_key`
API key for hosted providers
- `use_gpu`
Enables GPU-backed initialization where supported
- `allowed_sources`
Filters alignment sources. Available sources are `["esco"]`, `["onet"]`, `["osn"]`, and `["ukos"]`.
- `top_k`
Per-alignment-call cap for matched rows
- `return_edges`
Returns `{nodes, edges}` instead of only normalized rows
- `output_csv_path`
Writes CSV output only when explicitly provided

Additional examples are available in [docs/examples.md](docs/examples.md).
| Option | Description |
|---|---|
| `model_id` | `"gemini"`, `"openai"`, or a Hugging Face model id to run locally |
| `api_key` | API key for hosted providers |
| `use_gpu` | run local models on GPU where available |
| `backend` | `"llama_cpp"` to run a local GGUF model |
| `concepts` | `["skills"]` (default), or add `"knowledge"` and `"tasks"` |
| `allowed_sources` | taxonomies to align against: `"esco"`, `"onet"`, `"osn"`, `"ukos"` |
| `top_k` | maximum aligned matches per concept type, per document (default 25) |
| `return_edges` | return `{nodes, edges}` instead of only the results table |
| `output_csv_path` | also write the results to this CSV file |

Every option is described in the [usage guide](https://laiser-software.github.io/extract-module/usage/), and more snippets are in [docs/examples.md](docs/examples.md).

## Try it in Google Colab

The [cookbook](https://github.com/LAiSER-Software/laiser-cookbook) has complete analyses that open directly in Colab:

| Notebook | |
|---|---|
| Job skill analysis for job seekers | [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/LAiSER-Software/laiser-cookbook/blob/main/recipes/Job_Skill_Analysis/Job_Skill_Analysis_for_Job_Seekers.ipynb) |
| University program skill analysis | [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/LAiSER-Software/laiser-cookbook/blob/main/recipes/University_Program_Skill_Analysis/University_Program_Skill_Analysis.ipynb) |
| Pay equity analysis | [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/LAiSER-Software/laiser-cookbook/blob/main/recipes/Pay_Equity_Analysis/Pay_Equity_Analysis_Based_on_Skills.ipynb) |

## Funding
<div align="center">
Expand Down
36 changes: 36 additions & 0 deletions docs/cookbook.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# Cookbook notebooks

The [laiser-cookbook](https://github.com/LAiSER-Software/laiser-cookbook) repository holds complete
analyses built on LAiSER. Each opens directly in Google Colab, which is the quickest way to try the package
without a local install.

!!! tip "Running in Colab"
The notebooks use a hosted model, so add your provider key as a Colab secret (the key icon in the
left sidebar) before running. To avoid needing a key, switch the extractor to a small local model as
shown in the [Quickstart](quickstart.md) — expect slower extraction on Colab's CPU runtime.

## Job skill analysis

Extracts and visualizes the skills requested across a set of job postings, aimed at job seekers deciding
what to learn next.

[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/LAiSER-Software/laiser-cookbook/blob/main/recipes/Job_Skill_Analysis/Job_Skill_Analysis_for_Job_Seekers.ipynb)
&nbsp;
[View on GitHub](https://github.com/LAiSER-Software/laiser-cookbook/tree/main/recipes/Job_Skill_Analysis)

## University program skill analysis

Extracts and categorizes the skills described in university program descriptions, for comparing what
programs teach.

[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/LAiSER-Software/laiser-cookbook/blob/main/recipes/University_Program_Skill_Analysis/University_Program_Skill_Analysis.ipynb)
&nbsp;
[View on GitHub](https://github.com/LAiSER-Software/laiser-cookbook/tree/main/recipes/University_Program_Skill_Analysis)

## Pay equity analysis

Compares compensation against the skills each role actually requires, rather than job titles alone.

[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/LAiSER-Software/laiser-cookbook/blob/main/recipes/Pay_Equity_Analysis/Pay_Equity_Analysis_Based_on_Skills.ipynb)
&nbsp;
[View on GitHub](https://github.com/LAiSER-Software/laiser-cookbook/tree/main/recipes/Pay_Equity_Analysis)
47 changes: 47 additions & 0 deletions docs/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# LAiSER

**LAiSER** (Leveraging Artificial Intelligence for Skill Extraction & Research) is a Python package
that reads free text — job postings, course descriptions, syllabi — and returns the skills, knowledge
areas and tasks it describes, each matched to an entry in a published taxonomy.

The point of the matching step is comparability. Two job postings that say "Python scripting" and
"writing Python code" should end up pointing at the same taxonomy skill, so the results can be counted,
joined and compared across employers, programs and datasets.

## How it works

For every row in an input table, LAiSER runs four stages:

1. **Extraction.** The document text is placed into a prompt for its input type (job description or
syllabus) and sent to a language model, which returns candidate skills — and, if requested, the
knowledge and tasks behind them.
2. **Parsing and deduplication.** The model's response is parsed into a list of phrases. Exact
duplicates are removed, then near-duplicates are collapsed using embedding similarity.
3. **Taxonomy alignment.** Each phrase is embedded and matched to its closest entry in bundled FAISS
indexes of taxonomy entries. Matches above a similarity threshold are kept, up to `top_k` per concept
type.
4. **Output.** Matches are returned as one table with a row per match: the phrase as extracted, the
taxonomy entry it matched, which taxonomy that entry comes from, and the similarity score.

The language model can be a hosted API or a model that runs on your own machine — see
[Model providers](providers.md). The taxonomy indexes ship with the package, so alignment runs locally
either way.

## Bundled taxonomies

| Source | `allowed_sources` value | Covers |
|---|---|---|
| ESCO — European Skills, Competences, Qualifications and Occupations | `esco` | skills, knowledge, tasks |
| O\*NET — US Occupational Information Network | `onet` | skills, knowledge, tasks |
| UK Skills Classification | `ukos` | skills, knowledge, tasks |
| Open Skills Network | `osn` | skills |

Details and entry counts are on the [Taxonomies](taxonomies.md) page.

## Where to go next

- **[Installation](installation.md)** — install from PyPI, with or without GPU support.
- **[Quickstart](quickstart.md)** — a complete run, including one that needs no API key.
- **[Usage](usage.md)** — every option of `extract_concepts` and what the output columns mean.
- **[Cookbook notebooks](cookbook.md)** — end-to-end analyses you can open in Google Colab.
- **[API reference](reference.md)** — generated from the source.
60 changes: 60 additions & 0 deletions docs/installation.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
# Installation

## Requirements

- Python **3.10 or later**. CI tests 3.10, 3.11, 3.12 and 3.13.
- A GPU is optional. Hosted providers and small local models run on CPU; larger local models are
practical only on a GPU.

## Install from PyPI

```bash
pip install laiser
```

This installs everything needed to extract with a hosted provider or a local transformers model, and to
align results against the bundled taxonomies.

### GPU inference with vLLM

```bash
pip install "laiser[gpu]"
```

The `gpu` extra adds [vLLM](https://docs.vllm.ai/) for faster local inference on CUDA devices, plus
`bitsandbytes` and `accelerate`, which LAiSER uses to load a model in 8-bit when vLLM cannot. Check that
PyTorch can see your GPU:

```bash
python -c "import torch; print(torch.cuda.is_available())"
```

### Local GGUF models with llama.cpp

To run a quantized GGUF model through llama.cpp, install its Python bindings separately:

```bash
pip install llama-cpp-python
```

See [Model providers](providers.md#llamacpp-local-gguf) for how to point LAiSER at the model file.

## Install from source

```bash
git clone https://github.com/LAiSER-Software/extract-module.git
cd extract-module
pip install -e ".[dev]"
```

The `dev` extra adds the test and lint tooling used in CI.

## What downloads on first use

!!! note "First run is slower"
The taxonomy indexes are part of the package, but two things are fetched the first time they are
needed and cached afterwards by Hugging Face:

- the sentence-embedding model used for alignment and deduplication
(`sentence-transformers/all-MiniLM-L6-v2`), and
- the language model itself, if you use a local model rather than a hosted API.
Loading
Loading