Skip to content
27 changes: 27 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -192,6 +192,8 @@ print(results.head())
| `api_key` | API key for hosted providers |
| `use_gpu` | run local models on GPU where available |
| `backend` | `"llama_cpp"` to run a local GGUF model |
| `temperature` | decoding temperature for every backend (default `0.0`, greedy) |
| `seed` | seed for backends that accept one (default `42`) |
| `concepts` | `["skills"]` (default), or add `"knowledge"` and `"tasks"` |
| `allowed_sources` | taxonomies to align against: `"esco"`, `"onet"`, `"osn"`, `"ukos"` |
| `top_k` | maximum aligned matches per concept type, per document (default 25) |
Expand All @@ -200,6 +202,31 @@ print(results.head())

Every option is described in the [usage guide](https://laiser-software.github.io/extract-module/usage/), and more snippets are in [docs/examples.md](docs/examples.md).

### Reproducible results

Every backend decodes greedily by default (temperature `0.0`) and uses a fixed seed where the backend accepts one, so repeated runs over the same input give the same result. To make results reproducible for others, pin every setting that affects them:

```python
extractor = SkillExtractorRefactored(
model_id="Qwen/Qwen2.5-0.5B-Instruct", # pin the model
use_gpu=False,
temperature=0.0, # the default: greedy decoding
seed=42, # the default
)

results = extractor.extract_concepts(
data=data,
id_column="Research ID",
text_columns=["description"],
input_type="job_desc",
concepts=["skills"],
allowed_sources=["esco", "onet", "osn", "ukos"],
similarity_thresholds={"skill": 0.60, "knowledge": 0.50, "task": 0.50}, # the defaults
)
```

When reporting results, also record the LAiSER version: it fixes the bundled taxonomy data and the embedding model used for alignment (`sentence-transformers/all-MiniLM-L6-v2`). Hosted providers do not guarantee identical output even at temperature `0.0`. See [Reproducibility](https://laiser-software.github.io/extract-module/reproducibility/) for details.

## Try it in Google Colab

The [cookbook](https://github.com/LAiSER-Software/laiser-cookbook) has complete analyses that open directly in Colab:
Expand Down
6 changes: 5 additions & 1 deletion docs/providers.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,9 @@ Choose the model when you construct the extractor.

`hf_token` is only needed for gated or private Hugging Face models.

Every provider decodes greedily by default, with a fixed seed wherever it accepts one — see
[Reproducibility](reproducibility.md).

## Gemini

```python
Expand Down Expand Up @@ -70,7 +73,8 @@ because LAiSER formats the prompt with the model's chat template when it has one
## Local model on GPU

With `use_gpu=True` on a machine with CUDA, LAiSER first tries to load the model with vLLM. If vLLM cannot
load it, LAiSER loads it with transformers instead, using 8-bit quantization. Install the GPU extra, which
load it, LAiSER retries vLLM once with `TheBloke/Mixtral-7B-Instruct-v0.1-AWQ`. Only if that also fails does it
load the requested model with transformers instead, using 8-bit quantization. Install the GPU extra, which
includes vLLM plus the `bitsandbytes` and `accelerate` packages that 8-bit loading needs:

```bash
Expand Down
6 changes: 4 additions & 2 deletions docs/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,10 +95,12 @@ For the input above, the local 0.5B model on a laptop CPU returned:
| Type | Raw Concept | Taxonomy Concept | Taxonomy Source | Correlation Coefficient |
|---|---|---|---|---:|
| skill | Python programming | Program in Python | `ukos` | 0.70 |
| skill | Machine learning | Use machine learning to create or improve solutions | `ukos` | 0.66 |
| skill | Docker | Docker | `onet` | 0.75 |

Model output varies between runs and between models. A hosted or larger model typically extracts more
concepts from the same text — this small model missed "machine learning" and "AWS".
Decoding is greedy by default, so running this again returns the same rows — see
[Reproducibility](reproducibility.md). Different models extract different concepts: a hosted or larger model
typically finds more, and this small model missed "AWS".

## Next steps

Expand Down
93 changes: 93 additions & 0 deletions docs/reproducibility.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
# Reproducibility

Language models generate text by sampling, so left to their providers' defaults two runs over the same input
can return different skills. LAiSER is configured so that, by default, they do not.

## Defaults

Every backend decodes greedily: at temperature `0.0` the model always takes its most probable next token, so
the same prompt produces the same text. A fixed seed (`42`) is also supplied wherever a backend accepts one,
which keeps runs reproducible if you deliberately raise the temperature.
Comment on lines +8 to +10

| Backend | Temperature | Seed |
|---|---|---|
| Gemini | `0.0` | `42`, on google-genai releases that accept a seed |
| OpenAI | `0.0` | not accepted by the API |
| Local model through Transformers | greedy — sampling switched off | `42`, used when temperature is above `0.0` |
| Local model through vLLM | `0.0` | `42` |
| llama.cpp | `0.0` | `42`, on llama-cpp-python releases that accept a seed |

For Transformers models, sampling is turned off explicitly rather than left to the checkpoint. Some checkpoints
ship a `generation_config.json` that enables sampling, which would otherwise make repeated runs differ.

Alignment is deterministic as well. Each extracted phrase is embedded with a fixed sentence-transformer model
and matched by exact search against the bundled taxonomy index, so the same phrase always resolves to the same
taxonomy entry with the same similarity score.

## Changing the defaults

Pass `temperature` and `seed` when creating an extractor:

```python
extractor = SkillExtractorRefactored(model_id="gemini", api_key=api_key, temperature=0.7, seed=123)
```

Pass `seed=None` to send no seed. The defaults can also be changed for a whole process through environment
variables, which LAiSER reads when it is imported:

| Variable | Default | Effect |
|---|---|---|
| `LAISER_TEMPERATURE` | `0.0` | default temperature for every backend |
| `LAISER_TOP_P` | `1.0` | nucleus sampling cut-off; `1.0` leaves it off |

## Reference configuration

To let others reproduce your results, pin every setting that affects them:

```python
from laiser.skill_extractor_refactored import SkillExtractorRefactored

extractor = SkillExtractorRefactored(
model_id="Qwen/Qwen2.5-0.5B-Instruct", # pin the model
use_gpu=False,
temperature=0.0, # the default: greedy decoding
seed=42, # the default
)

results = extractor.extract_concepts(
data=data,
id_column="Research ID",
text_columns=["description"],
input_type="job_desc",
concepts=["skills"],
allowed_sources=["esco", "onet", "osn", "ukos"],
similarity_thresholds={"skill": 0.60, "knowledge": 0.50, "task": 0.50}, # the defaults
)
```

Also record:

- **the LAiSER version** — it fixes the bundled taxonomy data and the embedding model used for alignment,
`sentence-transformers/all-MiniLM-L6-v2`;
- **the model revision**, for Hugging Face models that may be updated under the same name.

## What this does not guarantee

- **Hosted models can change.** A provider can update the model behind a name, and providers do not
guarantee identical output even at temperature `0.0`.
- **Different models extract different phrases.** Alignment can map different wordings of a skill onto the
same taxonomy entry, but each phrase is matched on its own, so that is likely rather than guaranteed — and
alignment cannot recover a skill a model never extracted. Compare models on the aligned columns, and measure
their agreement rather than assuming it.
- **Hardware and library versions matter for local models.** Floating-point results can differ between
devices and library releases.

## How it is tested

`tests/test_determinism.py` runs in CI without a GPU, an API key or a language model download. The alignment
checks use the real embedding model, which Hugging Face downloads on first use. It checks that:
Comment on lines +87 to +88

- every public entry point defaults to temperature `0.0`, and every seedable one to seed `42`;
- the decoding parameters each backend actually receives match those defaults, and per-call overrides reach
every backend;
- repeated alignment of the same phrases returns an identical table, similarity scores included.
10 changes: 10 additions & 0 deletions laiser/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,16 @@
MAX_NEW_TOKENS = 1000
GENERATION_SEED = 42

# Decoding defaults. Every backend decodes greedily unless told otherwise, so that
# repeated runs over the same input produce the same generation. A temperature of
# 0.0 always takes the most probable token; top_p = 1.0 leaves nucleus filtering
# off so temperature alone controls decoding. GENERATION_SEED is passed to every
# backend that accepts one, which keeps runs reproducible when a caller samples
# deliberately with a higher temperature. Override per extractor with
# temperature= and seed=, or through these environment variables.
DEFAULT_TEMPERATURE = float(os.getenv("LAISER_TEMPERATURE", "0.0"))
DEFAULT_TOP_P = float(os.getenv("LAISER_TOP_P", "1.0"))

# SCQF Level Descriptors
SCQF_LEVELS: Dict[int, str] = {
1: "Basic awareness of simple concepts.",
Expand Down
13 changes: 13 additions & 0 deletions laiser/llm_models/anthropic.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@

import requests

from laiser.config import DEFAULT_TEMPERATURE, DEFAULT_TOP_P

_CODE_FENCE_RE = re.compile(r"```(?:json)?|```", re.IGNORECASE)


Expand All @@ -21,7 +23,15 @@ def anthropic_generate(
model: str = "claude-haiku-4-5-20251001",
timeout: int = 60,
max_tokens: int = 4096,
temperature: float = DEFAULT_TEMPERATURE,
top_p: float = DEFAULT_TOP_P,
) -> str:
"""Generate text with the Anthropic Messages API.

Decoding is greedy by default (temperature 0.0). ``top_p`` is only sent when
it differs from 1.0, because the API recommends adjusting temperature or
top_p but not both. The API accepts no seed.
"""
api_key = api_key or os.getenv("ANTHROPIC_API_KEY")
if not api_key:
raise ValueError("Anthropic API key not provided")
Expand All @@ -35,8 +45,11 @@ def anthropic_generate(
payload = {
"model": model,
"max_tokens": max_tokens,
"temperature": temperature,
"messages": [{"role": "user", "content": prompt}],
}
if top_p is not None and top_p != 1.0:
payload["top_p"] = top_p

try:
resp = requests.post(url, headers=headers, json=payload, timeout=timeout)
Expand Down
33 changes: 31 additions & 2 deletions laiser/llm_models/gemini.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,17 @@
from google.genai import errors as genai_errors
from google.genai import types

from laiser.config import DEFAULT_TEMPERATURE, DEFAULT_TOP_P, GENERATION_SEED

DEFAULT_GEMINI_MODEL = os.getenv("LAISER_GEMINI_MODEL", "gemini-2.5-flash")
DEFAULT_GEMINI_TIMEOUT = float(os.getenv("LAISER_GEMINI_TIMEOUT", "60"))
DEFAULT_MAX_OUTPUT_TOKENS = int(os.getenv("LAISER_GEMINI_MAX_OUTPUT_TOKENS", "4096"))

# Older google-genai releases have no seed field on GenerateContentConfig, and
# passing one there fails validation. Check once so a seed is only sent when the
# installed release accepts it.
_SUPPORTS_SEED = "seed" in getattr(types.GenerateContentConfig, "model_fields", {})


class GeminiAPI:
"""Small wrapper to keep backward compatibility with older imports."""
Expand All @@ -23,13 +30,19 @@ def __init__(
model_name: Optional[str] = None,
timeout: float = DEFAULT_GEMINI_TIMEOUT,
max_output_tokens: int = DEFAULT_MAX_OUTPUT_TOKENS,
temperature: float = DEFAULT_TEMPERATURE,
seed: Optional[int] = GENERATION_SEED,
top_p: float = DEFAULT_TOP_P,
):
if not api_key:
raise ValueError("Gemini API key is required")
self.api_key = api_key
self.model_name = model_name or DEFAULT_GEMINI_MODEL
self.timeout = timeout
self.max_output_tokens = max_output_tokens
self.temperature = temperature
self.seed = seed
self.top_p = top_p
self.client = genai.Client(api_key=api_key)

def generate(self, prompt: str) -> str:
Expand All @@ -43,7 +56,13 @@ def generate_with_config(
response_schema: Optional[Any] = None,
) -> str:
try:
config_kwargs = {"temperature": 0.0, "max_output_tokens": self.max_output_tokens}
config_kwargs = {
"temperature": self.temperature,
"top_p": self.top_p,
"max_output_tokens": self.max_output_tokens,
}
if self.seed is not None and _SUPPORTS_SEED:
config_kwargs["seed"] = self.seed
if response_mime_type:
config_kwargs["response_mime_type"] = response_mime_type
if response_schema is not None:
Expand Down Expand Up @@ -76,13 +95,23 @@ def gemini_generate(
max_output_tokens: int = DEFAULT_MAX_OUTPUT_TOKENS,
response_mime_type: Optional[str] = None,
response_schema: Optional[Any] = None,
temperature: float = DEFAULT_TEMPERATURE,
seed: Optional[int] = GENERATION_SEED,
top_p: float = DEFAULT_TOP_P,
) -> str:
"""Send `prompt` to Gemini and return generated text."""
"""Send `prompt` to Gemini and return generated text.

Decoding is greedy by default (temperature 0.0, top_p 1.0), and ``seed`` is
sent whenever the installed google-genai release accepts one.
"""
client = GeminiAPI(
api_key=api_key,
model_name=model_name,
timeout=timeout,
max_output_tokens=max_output_tokens,
temperature=temperature,
seed=seed,
top_p=top_p,
)
return client.generate_with_config(
prompt,
Expand Down
56 changes: 46 additions & 10 deletions laiser/llm_models/hugging_face_llm.py
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,11 @@
"""


from laiser.config import MAX_NEW_TOKENS
from typing import Optional

import torch

from laiser.config import DEFAULT_TEMPERATURE, DEFAULT_TOP_P, GENERATION_SEED, MAX_NEW_TOKENS
from laiser.llm_models.model_loader import load_model_from_transformer

try:
Expand All @@ -63,13 +67,26 @@


def llm_generate(
prompt: str, tokenizer, model, model_id: str, use_gpu: bool, max_new_tokens: int = MAX_NEW_TOKENS
prompt: str,
tokenizer,
model,
model_id: str,
use_gpu: bool,
max_new_tokens: int = MAX_NEW_TOKENS,
temperature: float = DEFAULT_TEMPERATURE,
seed: Optional[int] = GENERATION_SEED,
) -> str:
"""Generate a completion with a transformers model.

Returns only the newly generated text. ``model.generate`` returns the prompt
tokens followed by the completion; decoding all of it hands the prompt's own
example JSON to the response parser, which can match it instead of the answer.

Decoding is greedy by default. At temperature 0.0 sampling is switched off
explicitly rather than left to the checkpoint, because some checkpoints ship a
generation_config.json that enables sampling and would make repeated runs
differ. Above 0.0 the model samples, and ``seed`` is applied first so the run
is still reproducible.
"""
if tokenizer is None or model is None:
tokenizer, model = load_model_from_transformer(model_id, use_gpu=use_gpu)
Expand All @@ -84,22 +101,41 @@ def llm_generate(
inputs = tokenizer(text, return_tensors="pt").to(model.device)
pad_token_id = tokenizer.pad_token_id if tokenizer.pad_token_id is not None else tokenizer.eos_token_id

outputs = model.generate(
**inputs,
max_new_tokens=max_new_tokens,
pad_token_id=pad_token_id,
eos_token_id=tokenizer.eos_token_id,
)
do_sample = temperature is not None and temperature > 0.0
generation_kwargs = {
"max_new_tokens": max_new_tokens,
"pad_token_id": pad_token_id,
"eos_token_id": tokenizer.eos_token_id,
"do_sample": do_sample,
}
if do_sample:
generation_kwargs["temperature"] = temperature
generation_kwargs["top_p"] = DEFAULT_TOP_P
if seed is not None:
torch.manual_seed(seed)

outputs = model.generate(**inputs, **generation_kwargs)

completion = outputs[0][inputs["input_ids"].shape[1] :]
return tokenizer.decode(completion, skip_special_tokens=True).strip()


def llm_generate_vllm(prompt, llm):
def llm_generate_vllm(
prompt,
llm,
temperature: float = DEFAULT_TEMPERATURE,
seed: Optional[int] = GENERATION_SEED,
max_tokens: int = 200,
):
"""Generate a completion with a vLLM engine.

Temperature defaults to 0.0 (greedy) rather than vLLM's own default of 1.0,
and ``seed`` keeps sampled runs reproducible.
"""
if not VLLM_AVAILABLE:
raise ImportError("vLLM is not installed. Please install it to use this function.")

sampling_params = SamplingParams(max_tokens=200, seed=42)
sampling_params = SamplingParams(max_tokens=max_tokens, temperature=temperature, top_p=DEFAULT_TOP_P, seed=seed)
result = llm.generate([prompt], sampling_params=sampling_params)
raw_text = result[0].outputs[0].text.strip()
return raw_text
Loading
Loading