Revival 3/4: study areas from OpenStreetMap and a synthetic crime generator

.github/workflows/ci.yml

@@ -37,7 +37,7 @@ jobs:
           python-version: ${{ matrix.python-version }}
           cache: pip
       - name: Install
-        run: python -m pip install -e ".[dev,contour]"
+        run: python -m pip install -e ".[dev,osm,contour]"
       - name: Test
         run: pytest --cov --cov-report=term-missing
 

CHANGELOG.md

@@ -12,6 +12,11 @@ Revival release: the code base now targets Python 3.10+ with current
 versions of pandas (>= 2.2), GeoPandas (>= 1.0), scikit-learn and statsmodels.
 
 ### Added
+- `crime_mapping.load_study_area("City, Country")` fetches a study area
+  boundary from OpenStreetMap via `osmnx` (`pip install predspot[osm]`).
+- `synthetic.generate_crimes` generates synthetic events inside any study
+  area: Gaussian hotspots plus uniform background, with trend, annual cycle,
+  day-of-week and hour-of-day patterns; reproducible with `seed`.
 - `QuadratCount` mapping (event counts per cell) as a first-class alternative
   to `KDE`, usable with hexagonal (`create_gridhexagonal`) and square
   (`create_gridsquares`) grids inside `PredictionPipeline`.

CONTRIBUTING.md

@@ -8,7 +8,7 @@ Thanks for your interest in Predspot! Issues and pull requests are welcome.
 git clone https://github.com/adaj/predspot.git
 cd predspot
 python -m venv .venv && source .venv/bin/activate
-pip install -e ".[dev,contour]"
+pip install -e ".[dev,osm,contour]"
 ```
 
 ## Checks
@@ -17,6 +17,7 @@ pip install -e ".[dev,contour]"
 ruff check src tests          # lint
 ruff format src tests         # format
 pytest                        # tests (~10 s)
+PREDSPOT_NETWORK_TESTS=1 pytest tests/test_load_study_area.py  # also query OpenStreetMap
 ```
 
 CI runs the same checks on Python 3.10 to 3.13 for every pull request.

README.md

@@ -67,6 +67,23 @@ from predspot.crime_mapping import QuadratCount, create_gridhexagonal
 mapping = QuadratCount(tfreq='W', grid=create_gridhexagonal(study_area_gdf, resolution=1))
 ```
 
+### Study area from OpenStreetMap and synthetic data
+
+You do not need real data to try Predspot. Fetch a city boundary from
+OpenStreetMap (`pip install "predspot[osm]"`) and generate synthetic events
+with spatial hotspots and realistic temporal patterns (trend, annual cycle,
+day-of-week and hour-of-day profiles):
+
+```python
+from predspot import Dataset, load_study_area, generate_crimes
+
+study_area = load_study_area("Natal, Rio Grande do Norte, Brazil")
+crimes = generate_crimes(study_area, n_events=5000, n_hotspots=4,
+                         start="2019-01-01", end="2020-12-31", seed=0)
+dataset = Dataset(crimes, study_area)
+dataset.plot()
+```
+
 Or run the default pipeline in one call:
 
 ```python
@@ -80,7 +97,7 @@ print(pipeline.evaluate('r2', cv=3))
 
 ## Development ⚡
 
-Predspot has four main modules:
+Predspot has five main modules:
 
 `dataset_preparation`: Module for preparing and managing crime datasets and study areas.
 
@@ -90,12 +107,15 @@ Predspot has four main modules:
 
 `ml_modelling`: Module that implements the prediction pipeline and model evaluation.
 
+`synthetic`: Module that generates synthetic crime events (hotspots + temporal patterns) inside any study area.
+
 ### Installation steps 🛠️
 
 Predspot requires Python 3.10 or newer.
 
 ```bash
 pip install predspot              # from PyPI
+pip install "predspot[osm]"       # + study areas from OpenStreetMap (osmnx)
 pip install "predspot[contour]"   # + GeoJSON contour export (geojsoncontour)
 ```
 
@@ -104,7 +124,7 @@ From source, for development:
 ```bash
 git clone https://github.com/adaj/predspot.git
 cd predspot
-pip install -e ".[dev,contour]"
+pip install -e ".[dev,osm,contour]"
 ```
 
 Core dependencies (installed automatically): pandas, geopandas, shapely,

src/predspot/__init__.py

@@ -14,6 +14,7 @@
     dataset_preparation,
     feature_engineering,
     ml_modelling,
+    synthetic,
     utilities,
 )
 from predspot.crime_mapping import (
@@ -22,9 +23,11 @@
     create_gridhexagonal,
     create_gridpoints,
     create_gridsquares,
+    load_study_area,
 )
 from predspot.dataset_preparation import Dataset
 from predspot.ml_modelling import PredictionPipeline
+from predspot.synthetic import generate_crimes
 from predspot.utilities import PandasFeatureUnion
 
 __version__ = "0.2.0"
@@ -38,6 +41,9 @@
     "create_gridpoints",
     "create_gridhexagonal",
     "create_gridsquares",
+    "load_study_area",
+    "generate_crimes",
+    "synthetic",
     "crime_mapping",
     "dataset_preparation",
     "feature_engineering",

src/predspot/crime_mapping.py

@@ -14,6 +14,9 @@
 Both produce the same output format, a :class:`pandas.Series` named
 ``crime_density`` indexed by ``(t, places)``, so they are interchangeable
 inside :class:`predspot.ml_modelling.PredictionPipeline`.
+
+The study area itself can be fetched from OpenStreetMap with
+:func:`load_study_area` (requires the optional ``osmnx`` dependency).
 """
 
 __author__ = "Adelson Araujo"
@@ -83,6 +86,54 @@ def tfreq_offset(tfreq):
     return pd.offsets.Day(1)
 
 
+def load_study_area(place, crs=WGS84, which_result=None):
+    """
+    Fetch the boundary polygon of a place from OpenStreetMap.
+
+    Uses `osmnx <https://osmnx.readthedocs.io>`_ (optional dependency:
+    ``pip install predspot[osm]``) to geocode ``place`` with Nominatim and
+    return its administrative boundary, ready to be used as the
+    ``study_area`` of :class:`predspot.Dataset` or as the ``bbox`` of the
+    ``create_grid*`` functions.
+
+    Args:
+        place (str or list): Name of the place as Nominatim understands it,
+            e.g. ``"Natal, Rio Grande do Norte, Brazil"``. A list of names
+            returns one row per place.
+        crs: CRS of the returned GeoDataFrame (default WGS84).
+        which_result (int, optional): Forwarded to
+            ``osmnx.geocode_to_gdf`` to pick a specific Nominatim match when
+            the first one is not the boundary you want.
+
+    Returns:
+        GeoDataFrame: One row per place with ``name``, ``display_name``,
+        ``osm_type``, ``osm_id`` and a (Multi)Polygon ``geometry``.
+
+    Raises:
+        ImportError: If ``osmnx`` is not installed.
+        ValueError: If Nominatim returns a point instead of a boundary
+            polygon for the query.
+    """
+    try:
+        import osmnx as ox
+    except ImportError as exc:
+        raise ImportError(
+            "load_study_area requires the optional dependency `osmnx`: pip install predspot[osm]"
+        ) from exc
+    logger.debug("Geocoding study area %r with osmnx", place)
+    gdf = ox.geocode_to_gdf(place, which_result=which_result)
+    if not gdf.geom_type.isin(["Polygon", "MultiPolygon"]).all():
+        bad = gdf.loc[~gdf.geom_type.isin(["Polygon", "MultiPolygon"]), "display_name"]
+        raise ValueError(
+            "OpenStreetMap returned a non-polygon geometry for "
+            f"{bad.tolist()}. Try a more specific query (e.g. add the state "
+            "and country) or a different `which_result`."
+        )
+    columns = [c for c in ("name", "display_name", "osm_type", "osm_id") if c in gdf.columns]
+    gdf = gdf[columns + ["geometry"]].reset_index(drop=True)
+    return gdf.to_crs(crs)
+
+
 def _check_bbox(bbox):
     if not isinstance(bbox, gpd.GeoDataFrame):
         raise TypeError("bbox must be a geopandas GeoDataFrame.")

src/predspot/pipeline.py

@@ -25,6 +25,7 @@
 from sklearn.preprocessing import QuantileTransformer
 
 from predspot import crime_mapping, dataset_preparation, feature_engineering, ml_modelling
+from predspot.synthetic import generate_crimes
 from predspot.utilities import PandasFeatureUnion
 
 logger = logging.getLogger(__name__)
@@ -35,7 +36,10 @@
 
 def generate_testdata(n_points, start_time, end_time, bounds=DEFAULT_BOUNDS, seed=None):
     """
-    Generate uniformly random crime events inside a rectangular study area.
+    Generate synthetic crime events inside a rectangular study area.
+
+    A thin wrapper around :func:`predspot.synthetic.generate_crimes` with
+    three hotspots and the default temporal patterns.
 
     Args:
         n_points (int): Number of events.
@@ -48,27 +52,13 @@ def generate_testdata(n_points, start_time, end_time, bounds=DEFAULT_BOUNDS, see
         tuple: ``(crimes, study_area)`` — a DataFrame with ``tag``, ``t``,
         ``lon``, ``lat`` and a one-row GeoDataFrame with the study area.
     """
-    rng = np.random.default_rng(seed)
     west, south, east, north = bounds
     study_area = gpd.GeoDataFrame(
         {"name": ["study_area"]}, geometry=[box(west, south, east, north)], crs="EPSG:4326"
     )
-    start, end = pd.Timestamp(start_time), pd.Timestamp(end_time)
-    seconds = rng.integers(0, int((end - start).total_seconds()), n_points)
-    tags = rng.choice(
-        ["burglary", "assault", "drugs", "homicide"],
-        size=n_points,
-        p=np.array([1000, 100, 10, 1]) / 1111,
-    )
-    crimes = pd.DataFrame(
-        {
-            "tag": tags,
-            "t": start + pd.to_timedelta(seconds, unit="s"),
-            "lon": rng.uniform(west, east, n_points),
-            "lat": rng.uniform(south, north, n_points),
-        }
+    crimes = generate_crimes(
+        study_area, n_events=n_points, start=start_time, end=end_time, seed=seed
     )
-    logger.debug("Generated %d synthetic events", n_points)
     return crimes, study_area