Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions .github/workflows/deploy-prod.yml
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,20 @@ jobs:
done
exit $STATUS

- name: Verify taxonomy release assets are available
run: |
python - <<'PYTHON'
import json
from pathlib import Path
from urllib.request import Request, urlopen

manifest = json.loads(Path("laiser/taxonomy-assets.json").read_text())
for kind in manifest["bundles"]:
url = f"{manifest['release_url']}/{manifest['version']}/{kind}.zip"
with urlopen(Request(url, method="HEAD"), timeout=30): # nosec B310
print(f"Available: {url}")
PYTHON

- name: Publish to PyPI
uses: pypa/gh-action-pypi-publish@release/v1
with:
Expand Down
28 changes: 28 additions & 0 deletions .github/workflows/publish-taxonomy.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
name: Publish taxonomy data

on:
push:
tags:
- "taxonomy-v*"

jobs:
publish:
runs-on: ubuntu-latest
permissions:
contents: write
steps:
- uses: actions/checkout@v4

- uses: actions/setup-python@v5
with:
python-version: "3.11"

- name: Build verified bundles
run: python scripts/build_taxonomy_bundles.py "$GITHUB_REF_NAME" dist/taxonomy

- name: Publish GitHub Release assets
env:
GH_TOKEN: ${{ github.token }}
run: |
gh release create "$GITHUB_REF_NAME" dist/taxonomy/*.zip --verify-tag --latest=false \
--title "LAiSER $GITHUB_REF_NAME" --notes-file docs/taxonomies.md
14 changes: 2 additions & 12 deletions MANIFEST.in
Original file line number Diff line number Diff line change
@@ -1,12 +1,2 @@
include laiser/public/faiss_skills.csv
include laiser/public/knowledge_df.json
include laiser/public/knowledge_taxonomy.csv
include laiser/public/knowledge_v05.index
include laiser/public/skills_df.json
include laiser/public/skills_v04.index
include laiser/public/task_taxonomy.csv
include laiser/public/tasks_df.json
include laiser/public/tasks_v05.index
exclude laiser/public/Task Statements.xlsx
exclude laiser/public/combined.csv
exclude laiser/public/skill_embeddings.npy
include laiser/taxonomy-assets.json
prune laiser/public
8 changes: 5 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,15 +33,15 @@ You give it a table of documents — job postings, course descriptions, syllabi.

Because results point at shared taxonomy entries rather than free-text phrases, they can be counted and compared across sources — the skills employers ask for can be set against the skills a program teaches, or tracked across thousands of postings.

The language model can be a hosted API (Gemini or OpenAI) or a model running on your own machine, including on CPU with no API key. Taxonomy alignment always runs locally against indexes bundled with the package.
The language model can be a hosted API (Gemini or OpenAI) or a model running on your own machine, including on CPU with no API key. Taxonomy alignment always runs locally against indexes prepared during setup.

## Architecture

LAiSER runs four stages for every document:

1. **Extraction** — The text is placed into a prompt for its input type and sent to the language model, which returns candidate concepts.
2. **Parsing and deduplication** — The response is parsed into phrases. Exact duplicates are dropped and near-duplicates are collapsed by embedding similarity.
3. **Taxonomy alignment** — Each phrase is embedded and searched against bundled FAISS indexes of taxonomy entries. Matches above a per-type similarity threshold are kept.
3. **Taxonomy alignment** — Each phrase is embedded and searched against local FAISS indexes of taxonomy entries. Matches above a per-type similarity threshold are kept.
4. **Output** — Matches are returned as a single table, optionally with graph edges linking knowledge areas to the tasks they enable.

## Requirements
Expand All @@ -56,12 +56,14 @@ LAiSER runs four stages for every document:

```shell
pip install laiser
laiser setup
```

- Install with GPU extras:

```shell
pip install "laiser[gpu]"
laiser setup
```

- Install development dependencies from source:
Expand Down Expand Up @@ -225,7 +227,7 @@ results = extractor.extract_concepts(
)
```

When reporting results, also record the LAiSER version: it fixes the bundled taxonomy data and the embedding model used for alignment (`sentence-transformers/all-MiniLM-L6-v2`). Hosted providers do not guarantee identical output even at temperature `0.0`. See [Reproducibility](https://laiser-software.github.io/extract-module/reproducibility/) for details.
When reporting results, also record the LAiSER version: it pins the taxonomy data and the embedding model used for alignment (`sentence-transformers/all-MiniLM-L6-v2`). Hosted providers do not guarantee identical output even at temperature `0.0`. See [Reproducibility](https://laiser-software.github.io/extract-module/reproducibility/) for details.

## Try it in Google Colab

Expand Down
2 changes: 2 additions & 0 deletions docs/examples.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,14 @@ This file contains copy-paste examples for the current refactored API.

```bash
pip install laiser
laiser setup
```

GPU-enabled install:

```bash
pip install "laiser[gpu]"
laiser setup
```

## 2. Skills Only From a Job Description
Expand Down
4 changes: 2 additions & 2 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ For every row in an input table, LAiSER runs four stages:
knowledge and tasks behind them.
2. **Parsing and deduplication.** The model's response is parsed into a list of phrases. Exact
duplicates are removed, then near-duplicates are collapsed using embedding similarity.
3. **Taxonomy alignment.** Each phrase is embedded and matched to its closest entry in bundled FAISS
3. **Taxonomy alignment.** Each phrase is embedded and matched to its closest entry in local FAISS
indexes of taxonomy entries. Matches above a similarity threshold are kept, up to `top_k` per concept
type.
4. **Output.** Matches are returned as one table with a row per match: the phrase as extracted, the
Expand All @@ -27,7 +27,7 @@ The language model can be a hosted API or a model that runs on your own machine
[Model providers](providers.md). The taxonomy indexes ship with the package, so alignment runs locally
either way.

## Bundled taxonomies
## Supported taxonomies

| Source | `allowed_sources` value | Covers |
|---|---|---|
Expand Down
24 changes: 15 additions & 9 deletions docs/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,15 +10,22 @@

```bash
pip install laiser
laiser setup
```

This installs everything needed to extract with a hosted provider or a local transformers model, and to
align results against the bundled taxonomies.
Run `laiser setup` once before extracting. It downloads the package's taxonomy bundles from GitHub Releases,
checks their integrity, and prepares the alignment embedding model. No provider API key is needed for setup.

The taxonomy bundles total about 125 MiB compressed and are cached under `~/.cache/laiser` by default.
Set `LAISER_CACHE_DIR` to use a different taxonomy cache directory. The embedding model uses the
Hugging Face cache, which can be set with `HF_HOME`. Use the same cache settings for setup and extraction.
Re-running setup reuses the prepared files.

### GPU inference with vLLM

```bash
pip install "laiser[gpu]"
laiser setup
```

The `gpu` extra adds [vLLM](https://docs.vllm.ai/) for faster local inference on CUDA devices, plus
Expand Down Expand Up @@ -49,12 +56,11 @@ pip install -e ".[dev]"

The `dev` extra adds the test and lint tooling used in CI.

## What downloads on first use
The checkout includes the taxonomy files. Run `laiser setup` to prepare its alignment embedding model
ahead of time as well.

!!! note "First run is slower"
The taxonomy indexes are part of the package, but two things are fetched the first time they are
needed and cached afterwards by Hugging Face:
## Language models

- the sentence-embedding model used for alignment and deduplication
(`sentence-transformers/all-MiniLM-L6-v2`), and
- the language model itself, if you use a local model rather than a hosted API.
Setup prepares the taxonomy data and the embedding model used for alignment. If you choose a local
language model for extraction, that model has its own download. Hosted providers such as Gemini still
need network access when extracting from text.
2 changes: 1 addition & 1 deletion docs/providers.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Model providers

The language model does the extraction step; alignment always runs locally against the bundled indexes.
The language model does the extraction step; alignment always runs locally against the local indexes.
Choose the model when you construct the extractor.

| You pass | Runs on | Needs |
Expand Down
4 changes: 2 additions & 2 deletions docs/reproducibility.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ For Transformers models, sampling is turned off explicitly rather than left to t
ship a `generation_config.json` that enables sampling, which would otherwise make repeated runs differ.

Alignment is deterministic as well. Each extracted phrase is embedded with a fixed sentence-transformer model
and matched by exact search against the bundled taxonomy index, so the same phrase always resolves to the same
and matched by exact search against the local taxonomy index, so the same phrase always resolves to the same
taxonomy entry with the same similarity score.

## Changing the defaults
Expand Down Expand Up @@ -67,7 +67,7 @@ results = extractor.extract_concepts(

Also record:

- **the LAiSER version** — it fixes the bundled taxonomy data and the embedding model used for alignment,
- **the LAiSER version** — it pins the taxonomy data and the embedding model used for alignment,
`sentence-transformers/all-MiniLM-L6-v2`;
- **the model revision**, for Hugging Face models that may be updated under the same name.

Expand Down
2 changes: 1 addition & 1 deletion docs/taxonomies.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Taxonomies

Extracted concepts are aligned against taxonomy indexes that ship inside the package. Each index is a
Extracted concepts are aligned against local taxonomy indexes prepared by `laiser setup`. Each index is a
FAISS inner-product index over sentence embeddings of the entries' names and descriptions, built from the
CSV files in `laiser/public/`.

Expand Down
42 changes: 41 additions & 1 deletion laiser/data_access.py
Original file line number Diff line number Diff line change
Expand Up @@ -72,10 +72,17 @@

from laiser.config import COMBINED_SKILLS_URL, DEFAULT_EMBEDDING_MODEL, ESCO_SKILLS_URL, OSN_SKILLS_URL
from laiser.exceptions import FAISSIndexError, LAiSERError
from laiser.taxonomy_assets import MANIFEST, is_source_checkout, prepared_bundle

logger = logging.getLogger(__name__)


def load_embedding_model(*, local_files_only: bool) -> SentenceTransformer:
return SentenceTransformer(
DEFAULT_EMBEDDING_MODEL, revision=MANIFEST["embedding_revision"], local_files_only=local_files_only
)


class DataAccessLayer:
"""Handles data loading and external API calls"""

Expand All @@ -89,7 +96,13 @@ def __init__(self):
def get_embedding_model(self) -> SentenceTransformer:
"""Get or initialize the embedding model"""
if self.embedding_model is None:
self.embedding_model = SentenceTransformer(DEFAULT_EMBEDDING_MODEL)
local_files_only = not is_source_checkout()
try:
self.embedding_model = load_embedding_model(local_files_only=local_files_only)
except OSError as exc:
if not local_files_only:
raise
raise LAiSERError("The alignment embedding model is missing. Run 'laiser setup'.") from exc
return self.embedding_model

# -------------------------
Expand Down Expand Up @@ -203,6 +216,14 @@ def load_faiss_index(self, file_path: str) -> Optional[faiss.IndexFlatIP]:
except Exception as e:
raise FAISSIndexError(f"Failed to load FAISS index: {e}")

def load_prepared_index(self, kind, index_name, metadata_name):
directory = prepared_bundle(kind)
index = self.load_faiss_index(str(directory / index_name))
metadata = pd.read_json(directory / metadata_name, orient="records")
if index is None or index.ntotal != len(metadata):
raise FAISSIndexError(f"The {kind} index and metadata do not match at {directory}.")
return index, metadata

def download_faiss_index(self, url: str, local_path: str) -> bool:
"""Download FAISS index from URL"""
try:
Expand Down Expand Up @@ -249,6 +270,14 @@ def initialize_index(self, force_rebuild: bool = False, debug: bool = False) ->
local_npy_path = script_dir / "public" / "skill_embeddings.npy"
local_combined_csv_path = script_dir / "public" / "faiss_skills.csv"

if not is_source_checkout():
if force_rebuild:
raise LAiSERError("Rebuilding taxonomy indexes requires a source checkout.")
self.index, self.metadata = self.data_access.load_prepared_index(
"skills", local_index_path.name, local_json_path.name
)
return self.index, self.metadata

# 1) Try to load existing index + metadata unless force_rebuild requested
if not force_rebuild:
## Issue: Embedding (npy) is not accessed. Cosine Calculations might be faster if npy is accessed.
Expand Down Expand Up @@ -692,6 +721,7 @@ class _BaseTaxonomyFAISSIndexManager:
META_FILENAME: str = ""
CSV_FILENAME: str = ""
LABEL: str = ""
BUNDLE: str = ""

def __init__(self, data_access: DataAccessLayer):
self.data_access = data_access
Expand All @@ -706,6 +736,14 @@ def initialize_index(self, force_rebuild: bool = False) -> None:
meta_path = script_dir / "public" / self.META_FILENAME
csv_path = script_dir / "public" / self.CSV_FILENAME

if not is_source_checkout():
if force_rebuild:
raise LAiSERError("Rebuilding taxonomy indexes requires a source checkout.")
self.index, self.metadata = self.data_access.load_prepared_index(
self.BUNDLE, self.INDEX_FILENAME, self.META_FILENAME
)
return

# Try loading cached index
if not force_rebuild and index_path.exists() and meta_path.exists():
try:
Expand Down Expand Up @@ -821,6 +859,7 @@ class KnowledgeFAISSIndexManager(_BaseTaxonomyFAISSIndexManager):
META_FILENAME = "knowledge_df.json"
CSV_FILENAME = "knowledge_taxonomy.csv"
LABEL = "Knowledge"
BUNDLE = "knowledge"


class TaskFAISSIndexManager(_BaseTaxonomyFAISSIndexManager):
Expand All @@ -830,3 +869,4 @@ class TaskFAISSIndexManager(_BaseTaxonomyFAISSIndexManager):
META_FILENAME = "tasks_df.json"
CSV_FILENAME = "task_taxonomy.csv"
LABEL = "Task"
BUNDLE = "tasks"
5 changes: 4 additions & 1 deletion laiser/services.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@
from laiser.data_access import DataAccessLayer, FAISSIndexManager, KnowledgeFAISSIndexManager, TaskFAISSIndexManager
from laiser.exceptions import InvalidInputError, LAiSERError
from laiser.llm_models.llm_router import LLMRouter
from laiser.taxonomy_assets import is_source_checkout

logger = logging.getLogger(__name__)

Expand Down Expand Up @@ -645,7 +646,7 @@ def __init__(
self.faiss_manager = FAISSIndexManager(self.data_access)
self.alignment_service = AlignmentService(self.data_access, self.faiss_manager)

# Knowledge + Task indexes (v0.5) — initialized lazily; no-op if CSV not yet built
# Knowledge and task indexes (v0.5)
self.knowledge_faiss = KnowledgeFAISSIndexManager(self.data_access)
self.knowledge_faiss.initialize_index(force_rebuild=False)
self.knowledge_alignment = AlignmentService(self.data_access, self.knowledge_faiss)
Expand All @@ -669,6 +670,8 @@ def __init__(

# Initialize skills FAISS index
self.faiss_manager.initialize_index(force_rebuild=False)
if not is_source_checkout():
self.data_access.get_embedding_model()
# Log router initialization state for debugging
try:
llama_present = self.backend == "llama_cpp" and getattr(self.router, "llm", None) is not None
Expand Down
37 changes: 37 additions & 0 deletions laiser/taxonomy-assets.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
{
"version": "taxonomy-v1",
"release_url": "https://github.com/LAiSER-Software/extract-module/releases/download",
"embedding_revision": "1110a243fdf4706b3f48f1d95db1a4f5529b4d41",
"bundles": {
"skills": {
"skills_v04.index": {
"size": 41702445,
"sha256": "7ce221e34d82d84834bdde281c856d4b8fbbea238d33a351b27d62fae090ddef"
},
"skills_df.json": {
"size": 17447006,
"sha256": "a2e3af040711cbdb197156a3b4fb2fddbb9549436ab8eb40dc597e697cf4748b"
}
},
"knowledge": {
"knowledge_v05.index": {
"size": 12761133,
"sha256": "0eca417c323aa8904f2958c7a9c06eca46f1cb1610a838615f0f3c30ff601782"
},
"knowledge_df.json": {
"size": 3939145,
"sha256": "5f3fef364481b1ea19b88a0fafa4576286f5d375e8e9180c029aee898b6fb3fb"
}
},
"tasks": {
"tasks_v05.index": {
"size": 80016429,
"sha256": "7386969b9ddb56303b3ffd8c7b98921ef35829b85ad679a47ee777bf6c06bb9a"
},
"tasks_df.json": {
"size": 23418740,
"sha256": "9fb5afffc24182893e5c29cf6d7055f2d7311e72ec014fea4da6cf561900d889"
}
}
}
}
Loading
Loading