pricewatch-il

Daily ingestion of Israeli grocery prices. The 2014 transparency law obliges every chain to publish its full price list daily. Nothing obliges them to make it usable, and they don't — the files are scattered across three portal types, the schemas disagree, and chains change format without notice. This turns that into one clean, partitioned, public dataset.

Status: v0.1.0 — ingestion only, on purpose. This repository is running months ahead of the analysis it exists to support. Prices are published for today only: there is no archive and no backfill, so a day not captured is gone permanently. Basket indices, shrinkflation detection, and entity resolution all need history that does not exist yet, so the cron ships first and the analysis comes later. See ADR 0001.

What runs today

A GitHub Actions cron at 04:00 UTC that discovers, downloads, parses, validates, and publishes one day of prices to a Hugging Face Dataset.

	Measured 2026-09-01
Chains live	Shufersal (no credentials needed)
Stores discovered	417
Items per store	~6,835
Rows per day	~2.85 million
Parquet per day	46 MB (zstd)
Projected per year	~16.7 GB

With all ~10 chains enabled the projection is ~11.4M rows and ~184 MB/day. That scale is precisely why the dataset lives on the Hub rather than in this repository — see ADR 0002.

Architecture

flowchart TB
    subgraph discover["Discovery — one interface, three portal shapes"]
        SHUF["Shufersal<br/>paginated Azure blob index<br/>no auth · 417 stores"]
        CERB["Cerberus portal<br/>~10 chains behind<br/>published per-chain logins"]
        SKIP{{"Unconfigured chains<br/>skipped, not failed"}}
        CERB -.->|"no credentials"| SKIP
    end

    subgraph fetch["Fetch"]
        RETRY["Bounded concurrency<br/>exponential backoff<br/>empty page = retry, not EOF"]
        IDEM[("Raw cache<br/>re-runs skip existing files")]
    end

    subgraph parse["Parse & validate"]
        XML["Tolerant XML parse<br/>BOM · alias tags · recover mode"]
        CONTRACT{{"Data contract"}}
        QUAR[("quarantine/<br/>file + reason")]
        CONTRACT -->|"violation"| QUAR
    end

    subgraph publish["Publish"]
        PARQ["Parquet partitioned by<br/>date, then chain"]
        HUB[("Hugging Face Dataset")]
    end

    SHUF --> RETRY
    CERB --> RETRY
    RETRY --> IDEM --> XML --> CONTRACT
    CONTRACT -->|"pass"| PARQ --> HUB
    PARQ --> REPORT["Run report<br/>rows · failures · quarantined"]
    REPORT -->|"zero rows"| FAIL["CI fails loudly"]

    style QUAR fill:#78350f,color:#fff
    style CONTRACT fill:#78350f,color:#fff
    style SKIP fill:#1e3a5f,color:#fff
    style FAIL fill:#7f1d1d,color:#fff
    style HUB fill:#14532d,color:#fff

Loading

Quickstart

Works from a cold clone. Requires uv and Python 3.11 or 3.12.

git clone https://github.com/DataScientist13/pricewatch-il && cd pricewatch-il
just setup            # deps, pre-commit, and the git-level commit-message guard
just test             # 51 tests, fully offline

just chains           # which adapters are configured
just discover         # what the portals are serving right now
just smoke            # a real two-file run, end to end, in under a minute
just run              # a full day — ~417 files, several minutes

Inspect any single file, including anything that landed in quarantine:

just inspect data/raw/shufersal/PriceFull7290027600007-001-001-20260901-030000.gz

rows      6,532
skipped   0
store     001
contract  PASS

Three things that will silently corrupt this data

All three were found against the live feeds, and each has a test pinning it.

1. Coordinates of the problem: an empty page is not the end of the list. Shufersal's paginated index intermittently returns zero files under load — during verification, pages 18 and 22 came back empty while 19–21 each returned 20. Treating the first empty page as the end of the listing truncates the crawl and produces a day of partial data that looks complete. Empty pages are retryable failures here, and only three consecutive ones end the walk.

2. A chain switching to agorot looks exactly like inflation. Nothing crashes; the basket index just drifts by 100x. The contract rejects any file whose median price exceeds 500 ₪, and files are quarantined, never repaired — a rescaled file is an assumption presented as an observation. See ADR 0003.

3. PriceFull and Price are not the same thing. The latter is a delta. Ingesting it alongside snapshots yields a partial view that looks whole. Only full snapshots are ingested.

Plus the ordinary landmines: a UTF-8 BOM with no encoding declaration (lxml refuses it raw), per-chain tag casing (ChainID / chainid / productcode), and Hebrew names containing gershayim and asterisks — ניילון נצמד 30ס"מ*30 מטר.

The dataset

Published to DataScientist13/pricewatch-il:

data/observed_for=YYYY-MM-DD/chain=<slug>/part-0.parquet

20 columns per row — barcode, Hebrew item name, manufacturer, price, unit price, quantity, unit of measure, weighted flag, store and chain ids, and the source filename for provenance.

import polars as pl

df = pl.read_parquet("hf://datasets/DataScientist13/pricewatch-il/data/**/*.parquet")

Enabling more chains

Shufersal needs nothing. The Cerberus portal hosts roughly ten more chains behind per-chain logins, which are not secrets — the regulation requires each chain to publish its own access details:

export PRICEWATCH_CERBERUS_CREDENTIALS="chainuser:,otherchain:somepass"
just chains

An unconfigured chain is skipped, not failed: a partial day of real data beats no data, and the run report names every chain it skipped.

Engineering

Tests	51 passing, 91% coverage, fully offline
Type checking	mypy --strict clean across 15 modules
Lint	ruff clean (E, F, I, N, UP, B, SIM, RUF, ANN, D)
Idempotence	re-running a day skips downloads and rewrites the partition
Failure isolation	a dead chain or a bad file costs only itself
CI	tests on 3.11 and 3.12, plus a live upstream probe on every push

The upstream probe is the gate that matters for this repository: the realistic failure is not a bug in the code but a chain quietly changing its format, so CI parses a real file on every push and warns when one gets quarantined.

Ethical position

These files are published under a legal transparency mandate, so consuming them is exactly what they are for. The obligations are still real and are implemented rather than asserted: an identifying User-Agent, bounded concurrency, exponential backoff, and a raw cache so an unchanged file is never re-fetched. Re-running a day costs the portals nothing.

Limitations

1. Shufersal only, today. Roughly ten further chains are one configuration value away, but until they are enabled the dataset represents one chain — about a quarter of Israeli grocery retail.
2. Chains enabled later will have shorter histories. Any analysis must handle ragged start dates and state its window.
3. No entity resolution yet. Barcodes join the easy majority; fresh produce, private label, and weighted goods do not. That is the hard part and it is deliberately not attempted here.
4. A day missed is gone. There is no backfill. The cron failing quietly is the one unrecoverable failure mode, which is why it exits non-zero when a run captures nothing.
5. The contract is heuristic. It catches order-of-magnitude errors. A subtle format change — a field quietly meaning something new — will still get through.
6. Promotions are discovered and discarded. PromoFull files are filtered out; promo depth analysis will need them ingested separately.
7. No price-change detection yet. Every day is a full snapshot, so the dataset is larger than it strictly needs to be. Deltas are a later optimisation, not a correctness issue.

Licence

MIT. The underlying price data is published by Israeli retail chains under the transparency regulation; this repository redistributes none of it, only the code that fetches it.