datasets.md

Datasets

• Free Public API's 233 for students and developers

• Simple data pipeline powertools: sqlite, pandas, gnuplot and friends

  • Open [data] is also an open [api]
  • Amazing use of curl | gunzip | sql | query

• Mock

  • fakenamegenerator.com

Using real data is inspiring

• datacamp.com - Build data skills online
  • Data leaning courses/examples available
• Open Source Intelligence - Framework
  • Loads of links to cool datasets
• huggingface.co/datasets for machine learning

https://developers.google.com/search/docs/data-types/dataset

• W3C's Data Catalog Vocabulary: https://www.w3.org/TR/vocab-dcat-3/
• Schema.org dataset: https://schema.org/Dataset
• CSVW Namespace Vocabulary Terms: https://www.w3.org/ns/csvw
• Generating RDF from Tabular Data on the Web (examples on how to use CSVW): https://www.w3.org/TR/csv2rdf/

• Data sets

  • Kaggle Jupiter notebook with datasets live on the web
    • Girls-Who-Code-At-Home-Depicting-Data.pdf
  • The Ultimate Python Seaborn Tutorial: Gotta Catch ‘Em All
    • Pokemon gen1 data (csv) mined with python
  • Google Dataset Search
  • awesome-public-datasets
  • free-datasets-for-projects
  • OpenNeuro - MRI datasets
  • DataLad Scientific Data Community? Explorable versioned data? API for data?
  • Our World in Data - Research and data to make progress against the world’s largest problems
    • Who do we spend time with across our lifetime?
  • data.gov.uk
  • National Office of Statistics
    • Deaths - Deaths broken down by age, sex, area and cause of death.
  • UK Data Service international social science datasets
  • University of Bath Library: Finding Data links
  • WorldBank Data
    • WDI - World Development Data
  • tableau 7 public data sets you can analyse for free right now
    1. Google Trends
    2. National Climatic Data Center
    3. Global Health Observatory data
    4. Data.gov.sg
    5. Earthdata
    6. Amazon Web Services Open Data Registry
       1. 1000 genome project
    7. Pew Internet
  • Met Office (UK)
    • Digital Library and Archive
    • Historic Weather Data
      • sheffield yearly
      • Example global temp visulisation
    • datapoint api
  • Co2: Growth Rate
  • spaceweatherlive.com
  • National Centers for Environmental Information (US)
    • global hourly
  • NOAA STAR - Global Vegetation Health Products : Introduction
    • Breathing Earth
  • Severe Weather Europe - Follow severe weather as it happens. Anywhere. Any time
    • (more writeups rather than data)
  • Open-Metro
    • free weather data for non comercial use
  • Local
    • propertydata.co.uk
      • Planning Permissions
    • planningfinder.co.uk - planning permission on map with emial alerts
    • Official statistics from MHCLG (Ministry of Housing, Communities & Local Government)
      • 200+ datasets published under open licences covering MHCLG’s priorities including: deprivation, housing, local government finance, planning and other themes.
    • local.gov.uk - Improving services through information
      • The local area benchmarking tool from the Local Government Association
  • Maps
    • OpenStreetMap Javascript Editor - ID
      • StreetComplete - Easy to use OpenStreetMap editor for Android
      • GeoJSON Tools
      • Mapillary Images to maps
      • How to make fancy road trip maps with R and OpenStreetMap
        • Use R to get geocoded location and routing data from OpenStreetMap and explore our family’s impending 5,000 mile road trip around the USA
    • Live Realtime
      • Lightening Maps
      • Windy.com
      • Rain-dar
    • Electricity Map of the World - Ranked by carbon intensity of electricity consumed (gCO₂eq/kWh)
    • SlowWays - footpaths between cities
    • opencellid.org - The world's largest Open Database of Cell Towers
      • Locate devices without GPS, explore Mobile Operator coverage and more!
      • View individual radio masts
      • [networks] [maps]
      • https://wigle.net/
    • shipmap.org
    • Earth Engine - A planetary-scale platform for Earth science data & analysis
      • timelaps
    • Global: Notable People

• Traff - Traffic Feed Format, a generic data format for live road traffic updates

• Planning permission (UK)

  • Scraping 241 UK council planning portals – 2.6M decisions so far

Senate Stock Watcher US Senators invest in socks. Before CoVid 19 many senators sold stocks in travel/touisum and invested in medical supplies while telling the US population 'not to worry'. Senetors report their stocks with paper written notes. It takes multiple weeks to process them and they can be incorrect/misinterpreted as handwriting can be hard to read.

https://www.metoffice.gov.uk/pub/data/weather/uk/climate/datasets/Rainfall/date/England_SE_and_Central_S.txt https://www.ethnicity-facts-figures.service.gov.uk/crime-justice-and-the-law/courts-sentencing-and-tribunals/prosecutions-and-convictions/latest https://data.cambridgeshireinsight.org.uk/dataset/mill-road-project-traffic-sensor-data/resource/67873c4e-2d50-4d31-b2a0-68f1df7f081e

• Analysis of distances between bus stops London 2023

• apps.london.gov.uk/street-trees Web app to see all street trees in London

Data formats

• [data-formats]
• Indoor Mapping Data Format

Examples of use of data

• [maps]
• desirable-streets in Boston
  • Beautiful visual web presentation ofb data
• hedonometer.org
  • How happy is twitter - sentiment analysis from 2009 to 2021

etymology

• Surprising shared word etymologies
  • list of common English words
  • a database of word etymologies
  • GloVe - Global Vectors for Word Representation

Space

• Didnt know where to put this .. but OH MY GOD
  • This New 10 Terapixel Image of the Night Sky Contains 1 Billion Galaxies
    • viewer
• Satelite orbits? - plot in 3d?
  • app.keeptrack.space
  • orbit viewer
  • Orbit Dataset
  • Orbit descriptions
  • Japan developing wooden satellites to cut space junk
  • How "See A Satellite Tonight" Works
    • Mobile app! for seeing satellites! and setting alarms to see them
  • Orbidium
    • Visualize every astroid in the solar system locally
    • An app demo showing asteroid orbits using NASA MPC database.
  • Every Satellite Orbiting Earth and Who Owns Them 2023
• spaceweatherlive.com
• Logarithmic Maps of the Universe

Medical

• WHO - Model List of Essential Medicines

Imaging

• hyper-resolution.org - web viewer for gigapixel images

Unsorted

• energydashboard.co.uk

  • See the UK energy generation

• eia US Energy Information Administration - Independent Statistics and Analysis Gasoline explained Where our gasoline comes from

• wikipedia.org/wiki/List_of_rivers_by_discharge

• American Solar Farms

  • Some dude used stellite images to estimate the power output of all us solar farms

• National Grid: Live

  • The National Grid is the electric power transmission network for Great Britain