Open Social Datasets Initiative

Our mission is to make open social data from sources like national statistics agencies (e.g., Stats NZ, ABS) accessible, understandable, and usable for a wide audience.

This repository aims to be a central hub for curated links to datasets, their comprehensive documentation, and guides to help everyone from researchers to the general public explore and utilize valuable social information.

Project Philosophy

• Openness & Accessibility: Publicly available data and documentation under open licenses.
• Understandability: Clear, concise documentation for technical and non-technical users, including plain language guides.
• Usability & Reusability: Common data formats and clear access instructions.
• Transparency & Reproducibility: Documented processes for data handling.
• Collaboration & Community: Encouraging contributions and feedback.
• Accuracy & Reliability: Striving for up-to-date information, clearly linking to original sources.
• Ethical Considerations: Highlighting responsible data use.

How to Navigate This Repository

• /datasets: Contains organized collections of datasets from Stats NZ, ABS, AIHW, and the New Zealand Ministry of Health.
  • Each dataset will have its own directory including:
    • README.md: Specific information about the dataset, its source, structure, and how to access it.
    • data_dictionary.md: Detailed explanation of all variables.
    • accessible_guide.md: A plain-language guide to understanding the dataset.
    • Potentially data/ (for small, directly hosted data snippets or examples) or scripts/ (for fetching/processing).
• /docs: General guides, technical documentation, and articles related to using social data.
  • docs/guides/: Accessible, topic-based guides.
  • docs/technical/: More in-depth technical papers or API usage examples.
• /templates: Contains templates for creating standardized dataset documentation.
• ROADMAP.md: Outlines future plans and milestones for this project.
• TODO.md: Lists specific tasks to be done.
• CHANGELOG.md: Tracks changes and additions to the repository.
• CONTRIBUTING.md: Guidelines on how to contribute to this project.
• SESSION_LOG_TEMPLATE.md: A template for documenting data work sessions.

Rust Data Engine Preview

This repository now includes an early Rust data engine and CLI scaffold for provider-backed row-level data fetches, quality checks, Parquet export, and local catalog metadata.

Common commands:

cargo run --bin open-social-data-cli -- list
cargo run --bin open-social-data-cli -- list --provider abs
cargo run --bin open-social-data-cli -- list --provider digitalnz
cargo run --bin open-social-data-cli -- list --provider data_gov_au
cargo run --bin open-social-data-cli -- status
cargo run --bin open-social-data-cli -- fetch abs QBIS --output datasets/abs/qbis_business_indicators/data/qbis.parquet --quality-report .open-social-data/qbis-quality.json
cargo run --bin open-social-data-cli -- fetch digitalnz nz_gazette --output datasets/digitalnz/nz_gazette.parquet
cargo run --bin open-social-data-cli -- catalog sync
cargo run --bin open-social-data-cli -- catalog sync --provider abs
cargo run --bin open-social-data-cli -- catalog sync --sqlite .open-social-data/catalog.sqlite
cargo run --bin open-social-data-cli -- catalog list
cargo run --bin open-social-data-cli -- catalog list --sqlite .open-social-data/catalog.sqlite
cargo run --bin open-social-data-cli -- catalog search qbis
cargo run --bin open-social-data-cli -- catalog search qbis --sqlite .open-social-data/catalog.sqlite
cargo run --bin open-social-data-cli -- validate dataset-packs
cargo run --bin open-social-data-cli -- validate source-metadata
cargo run --bin open-social-data-cli -- validate medium-term --run-examples
cargo run --bin open-social-data-cli -- examples myhospitals-summary --limit 5
cargo run --bin open-social-data-cli -- examples source-metadata-inventory

Local runtime metadata is written under .open-social-data/ by default. The default CLI catalog path is JSON-backed for simple file exchange, and catalog list/search/sync --sqlite <path> uses an embedded SQLite catalog for queryable local metadata storage. Catalog entries preserve provider source URLs, quality report paths, ETag values, and Last-Modified values. Fetches reuse cached ETag and Last-Modified metadata and preserve existing outputs when a provider returns 304 Not Modified. ABS fetches parse SDMX-JSON observations into rows, Stats NZ fetches flatten OData value rows, and the Data.gov.au provider consumes the public CKAN metadata API plus the current OAIC FOI-statistics CSV resource. The Rust CLI is the supported user-facing entry point for release validation and local examples; Python scripts remain as maintainer helpers and dataset wrappers, and R examples are retired from the validated capability set.

On this Windows workspace, run Rust validation with a target directory outside OneDrive to avoid damaged inherited ACLs on the repository target/ directory:

set CARGO_TARGET_DIR=C:\tmp\open_social_data_target2
cargo check --all-targets
set CARGO_PROFILE_TEST_DEBUG=0
cargo test
cargo clippy --all-targets -- -D warnings

The repo includes a GNU-target Cargo linker setting so the active stable-x86_64-pc-windows-gnu toolchain uses gcc instead of accidentally resolving Git for Windows' link.exe.

Release-readiness checks for the current dataset packs are documented in docs/technical/release_readiness_checklist.md. External source and future expansion follow-ups that are not roadmap blockers are tracked in docs/external_future_followups.md.

Current Dataset Coverage

Current pack-level coverage includes population, prices, labour, national accounts, business indicators, household income and housing costs, earnings, hospital measures, and New Zealand Health Survey key results. Most packs are source-linked and script-backed; some are endpoint-driven because official export URLs are generated by agency data explorers.

General interpretation guides live in docs/guides/, and small local examples live in examples/.

How to Contribute

We welcome contributions! Please see CONTRIBUTING.md for more details. In general, contributions can include:

• Adding new datasets (following the established structure and templates).
• Improving documentation.
• Reporting issues or suggesting improvements.
• Sharing use cases or analysis scripts.

License

• The content of this repository, including documentation and scripts generated for this project, is licensed under the MIT License unless otherwise stated.
• Individual datasets will clearly state their original source licenses. Please ensure you adhere to those terms.

Contact & Acknowledgements

• For questions or suggestions regarding this project, please open an issue in the repository.
• We acknowledge and thank the national statistics agencies and other open data providers for making their data publicly available.

---

This README provides an overview. For more detailed plans, see ROADMAP.md.