EUBUCCO Feature Engineering

Feature engineering pipeline for the EUBUCCO project, computing comprehensive building-level features from geospatial and building stock data across European regions.

Overview

This repository extracts and computes a wide range of features for buildings in European NUTS regions, including:

• Building geometry: footprint area, perimeter, shape metrics (elongation, convexity, rectangularity), orientation, corners
• Block-level features: block characteristics and building density metrics
• Spatial context: proximity to streets, points of interest (POIs), land use patterns, and neighboring buildings
• Environmental features: elevation, ruggedness, climate data (cooling/heating degree days), population density
• Building type indicators: OSM building classifications and GHS built-up area characteristics
• Location encodings: regional identifiers and satellite-based location embeddings

Quick Start

Installation

Install dependencies:

pip install -r requirements.txt

Ensure required data files are available in the data/ directory (see Data Requirements below).

Usage

Run the feature engineering pipeline for a specific NUTS region:

python main.py <region_id>

Example:

python main.py FRK26

The pipeline processes buildings for the specified region and outputs a Parquet file to results/<region_id>.parquet.

Data Requirements

The pipeline expects the following data files in the data/ directory:

• Building geometries: bldgs/*.parquet
• Street networks: streets/*.gpkg
• Points of interest: pois/*.gpkg
• GHS built-up area: GHS_BUILT_C_MSZ_E2018_GLOBE_R2023A_54009_10_V1_0_R4_C19.tif
• CORINE land use: U2018_CLC2018_V2020_20u1.tif
• Water polygons: OSM-water-polygons-split.gpkg
• Topography: gmted2010-mea075.tif
• Climate data: CDD_historical_mean_v1.nc, HDD_historical_mean_v1.nc
• Population: GHS_POP_E2020_GLOBE_R2023A_54009_100_V1_0.tif
• Regional data: lau_nuts_extra.csv
• Satellite embeddings: satclip_europe_res8_pca64.parquet

Project Structure

• features/: Feature computation modules (building, block, street, POI, land use, etc.)
• preproc/: Data preprocessing scripts
• util/: Utility functions for data loading and processing
• slurm/: Slurm cluster configuration files and execution scripts
• tests/: Test suite and test data for pipeline validation
• results/: Output directory for computed features
• data/: Input data directory (not tracked in git)

Development

Pre-Commit Hooks

This project uses pre-commit hooks to maintain code quality:

1. Install pre-commit:

   pip install pre-commit

2. Install hooks:

   pre-commit install

3. Run hooks manually:

   pre-commit run --all-files

The hooks enforce code formatting with Black, linting with Flake8, and import sorting with isort.

Testing

Run the test suite with pytest:

pytest tests/

The test suite includes end-to-end pipeline validation using test data in tests/data/.

Slurm Pipeline

The pipeline is orchestrated on HPC Slurm clusters using the slurm-pipeline orchestrator. Configuration files in slurm/ define job parameters, resources, and dependencies. Resource allocations automatically scale for large regions (e.g., increased CPUs and memory for regions with building files >100MB).