Skip to content
BranchesTags

Repository files navigation

Global Language Family Mapper (GLFM)

A modular, JSON-based language database and validation pipeline for over 7,900 languages (ISO 639-3), including Uralic/Finno-Ugric languages. Designed for RSS feeds, localization, and NLP workflows.

  • Author: Tuomas Lähteenmäki
  • License: MIT
  • Version: 1.0.2

Python License Status Platform

Primary Links

Features

  • 7,900+ Languages: Unified from ISO 639-1/2/3/5, Wiktionary, CLDR, Glottolog, and POS statistics.
  • Nearest Language Neighbors: Each written language includes typologically nearest languages based on URIEL/lang2vec features (syntax, phonology, inventory, family).
  • Accurate BCP-47 Tags: Language, script, and region combined automatically.
  • Fallback Chains: Automatically assigns fallback languages where applicable.
  • Uralic/Finno-Ugric Support: Detailed mapping for Finnish, Karelian, Estonian, and related languages.
  • Validation Pipeline: Full automated checks for ISO consistency, BCP-47 compliance, fallback correctness, POS stats, and Glottolog data.
  • Graphviz Visualization: Fallback relationships can be visualized.
  • Checkpoint/Resume Support: Long-running nearest-language computations can be resumed automatically from a checkpoint.

Directory Structure

├── build_unified.py                  # Build unified JSON database
├── data/                             # Raw and processed language data
│   ├── cldr/
│   │   └── likelySubtags.json
│   ├── iso_639_*.py                  # ISO datasets in Python format
│   ├── override_scripts.json
│   ├── uralic_languages.json
│   └── ...
├── loaders/                          # Data loading modules
├── logic/                            # Logic for scripts, regions, fallbacks, BCP-47
├── models/                           # Data models (Language class, ISO modules)
├── output/                           # Pipeline output
│   ├── unified/                      # unified_languages.json
│   ├── validation/                   # validation reports, errors, fallback graph
│   ├── final/
│   ├── dialects/
│   └── logs/
├── tools/                            # Validation, report generation, and helpers
│   ├── compute_nearest_languages.py  # Compute nearest language neighbors
│   └── ...
├── run_full_pipeline.py              # Run full pipeline: build → validate → report
└── init.py

Data Structure (unified_languages.json)

Each language is represented as a JSON object keyed by its ISO 639-3 code:

"fin": {
  "bcp47": "fi-Latn-FI",
  "default_region": "FI",
  "default_script": "Latn",
  "distance_bases": [
    "WALS",
    "SSWL",
    "PHOIBLE",
    "Ethnologue",
    "Glottolog"
  ],
  "distance_source": "lang2vec / URIEL",
  "distance_type": "fam",
  "fallback": "fin",
  "family": "",
  "glottocode": "",
  "glottolog": {},
  "id": "fin",
  "iso639_1": "fi",
  "iso639_2B": "fin",
  "iso639_2T": "fin",
  "iso639_3": "fin",
  "iso639_5": "",
  "name": "Finnish",
  "nearest_languages": [
    {
      "lang": "est",
      "distance": 0.0003
    },
    {
      "lang": "izh",
      "distance": 0.0003
    },
    {
      "lang": "liv",
      "distance": 0.0003
    },
    {
      "lang": "vep",
      "distance": 0.0003
    },
    {
      "lang": "vot",
      "distance": 0.0003
    }
  ],
  "official_name": "Finnish",
  "pos_stats": {},
  "uralicNLP": true,
  "written": true,
  "written_scripts": [
    "Latn"
  ]
}
  • bcp47 is automatically generated: lang[-Script][-Region].
  • fallback is the default fallback language.
  • uralicNLP is true if the language belongs to the Uralic/Finno-Ugric family.
  • written_scripts lists all known writing scripts.
  • nearest_languages ranks typologically nearest written languages by distance (0.0 = identical features, 1.0 = completely different).

Pipeline Overview

The pipeline is fully automated:

  1. Generate ISO files (tools/generate_iso_files.py)
  2. Build Unified Database (build_unified.py)
  3. Compute Nearest Languages (tools/compute_nearest_languages.py) – adds nearest_languages field to all written languages (supports checkpoint/resume for long runs)
  4. Validate:
    • BCP-47 tags (validate_bcp47.py)
    • Fallback chains (validate_fallbacks.py)
    • ISO consistency (validate_iso_consistency.py)
    • POS statistics (validate_pos_stats.py)
    • Glottolog data (validate_glottolog.py)
  5. Generate Validation Reports (generate_report.py):
    • Markdown report (validation_report.md)
    • HTML report (validation_report.html)
    • JSON error log (validation_errors.json)
  6. Visualize Fallback Graph (visualize_fallbacks.py)
  7. Build Supplemental Data:
    • Extinct languages dataset (build_extinct_json.py)
    • Top-5 fallback optimization (build_top5.py)
  8. Compress & Convert Outputs:
    • Standard library package (compress_stdlib.py)
    • Binary dataset (convert_to_msgpack.py)

Requirements

  • Python 3.8+
  • Core dependencies: numpy>=1.26.0,<2.0.0, setuptools>=69.0.0
  • Language data: lang2vec>=1.1.0
  • Data processing: requests>=2.28.0, msgpack>=1.0.5
  • Visualization (optional): graphviz>=0.20.0
  • Testing (optional): pytest>=7.0.0

Usage

Run the full pipeline

git clone [https://codeberg.org/lahtis/GLFM.git](https://codeberg.org/lahtis/GLFM.git)
cd GLFM
pip install -r requirements.txt

python3 run_full_pipeline.py

Run nearest-language step manually

If you need to run or resume the heavy nearest-languages calculation separately:

python3 tools/compute_nearest_languages.py

Note: Checkpoints are saved automatically to output/unified/nearest_languages.checkpoint.json and allow interrupted runs to resume seamlessly.

Build optimized Top-5 fallbacks dataset

Generate extinct languages dataset

python3 logic/build_extinct_json.py

Build optimized fallbacks dataset (top 20 candidates per language)

python3 tools/build_top5.py --top-k 20

Note: The build_top5.py script takes --top-k 20 (or -k 20) to specify how many fallback candidates are stored per language.

Access data in Python

import json
from pathlib import Path

unified_file = Path("output/unified/unified_languages.json")
with unified_file.open("r", encoding="utf-8") as f:
    languages = json.load(f)

# Access Finnish
finnish = languages.get("fin")
print(finnish["bcp47"])         # fi-Latn-FI
print(finnish["uralicNLP"])     # True
print(finnish["nearest_languages"][:5]) # Top 5 nearest languages

Find nearest languages

# Get 10 nearest languages to Finnish
finnish = languages.get("fin", {})
for neighbor in finnish.get("nearest_languages", [])[:10]:
    print(f"{neighbor['lang']}: {neighbor['distance']:.4f}")

Access data from compressed archive

import gzip
import json
from pathlib import Path

# Load compressed database (.json.gz)
database_path = Path("output/unified/languages_top20.json.gz")
with gzip.open(database_path, "rt", encoding="utf-8") as f:
    languages = json.load(f)

# Access language data
finnish = languages.get("fin", {})
print(finnish.get("bcp47"))  # fi-Latn-FI

# Get 10 nearest languages to Finnish
for neighbor in finnish.get("nearest_languages", [])[:10]:
    print(f"{neighbor['lang']}: {neighbor['distance']:.4f}")

Notes

  • uralic_languages.json must be present for Uralic language support; the pipeline will issue a warning if missing.
  • The unified database integrates data from ISO, Wiktionary, CLDR, and POS statistics (Glottolog integration is currently referenced in codebase for future work).
  • Fallback chains ensure robust language resolution for RSS feeds, locale matching, and software localization.
  • The nearest_languages field is computed using URIEL/lang2vec typological features (family, syntax, phonology, and inventory).
  • Distance scores range from 0.0 (identical features) to 1.0 (completely different).

Data Sources & Acknowledgments

This project aggregates and unifies data from multiple linguistic databases:

  • Glottolog: Comprehensive bibliographic data and genealogical classification for world languages. Hammarström, Harald & Forkel, Robert & Haspelmath, Martin & Bank, Sebastian. Glottolog. (glottolog.org)
  • URIEL / lang2vec: Typological vector representations derived from URIEL knowledge base.
  • ISO 639: Language code standards (ISO 639-1, 639-2, 639-3).
  • CLDR / Wiktionary: Locale display names, scripts, and part-of-speech statistics.

Contributing & Issues

This project is primary developed on Codeberg. Please submit issues, feature requests, and pull requests to the primary repository: https://codeberg.org/lahtis/GLFM

License

Copyright © 2026 Tuomas Lähteenmäki

This project is licensed under the MIT License. You may use, modify, and distribute the data, provided the original copyright notice is retained.

See LICENSE for details.

About

[Mirror] Global Language Family Mapper (GLFM) A comprehensive JSON database that maps over 7,700 languages (ISO 639-3) to their respective language families (ISO 639-5). This library was specifically developed for language validation in RSS tools and localization workflows.

codeberg.org/lahtis/GLFM

Topics

bcp47databasehistoryiso-639-3iso-639-5jsonlanguage-familieslanguage-mappinglinguisticslinuxlocalizationnlpnlp-resourcespythonrssuralic-languages

Resources

Readme
License
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages