Preprocessing

Unfortunately, the raw data is always incomplete. There are multiple ways to deal with it, and ours is to employ imputation. Whereas working with complete records is a viable approach also we cannot use it: a single incomplete individual record in a particular household means the latter has to be removed completely. This introduces even more bias and potentially removes certain population groups from consideration.

Convert original data

Setup

The recommended way to set up a working environment and get it running is to clone the repository and open it in PyCharm. The first time you do this your IDE throws a warning that there is no interpreter for this project and provides instructions how to install it - use the miniconda option. Once the interpreter is there and working, you can install all required dependencies - the IDE is smart, it detects what is needed and asks you to install everything. Finally, open the terminal within the IDE and run pip install -e .; create a new Jupyter notebook in the same directory to run your code in it.

Data handling

The very first step would be to download data as STATA files. The format choice was primarily based on possibility to work with people who don't know much R or Python, and is not critical. Unpack it somewhere (here, it's ~/Work/data/understanding_society) and use the following code to extract only the columns that are needed and store them in CSV files:

from synthwave.utils.uk.understanding_society import preprocess_usoc_data

DATA_PATH = "~/Work/data/"

preprocess_usoc_data(DATA_PATH, , skip_conversion=False)

The conversion is typically done only once as most of the changes happen in the actual pre-processing code, so you can do subsequent runs with the True flag:

preprocess_usoc_data(DATA_PATH, , skip_conversion=True)

Now, there is a new directory called synthwave in your data folder, with the sub-directory md containing *.pickle files with raw but human-readable individual and household data subsets. This is done to avoid data extraction every time there is a change in the pre-processing procedure. The final, non-imputed data is stored in your synthwave directory as *.parquet files with the following names: adults_non_imputed_middle_fidelity, children_non_imputed_middle_fidelity, households_non_imputed_middle_fidelity.

This is a step that's typically done once unless one needs more data from the original files. All subsequent runs for whatever reason can read pre-processed data from the new file directly.

Internal steps

1. We keep only households with non-zero cross-sectional weights.
2. The weights are scaled up and rounded so that they can be represented by integers.
3. All individuals aged 15 and below are seen as adults, this is to match the Census definition of a child (at least one of them)
4. We replace all encoded missing values with NA
5. We correct where possible inconsistent records
6. We scale up the dataset with adults using weights obtained earlier

Imputation

Unfortunately, we have to resort to R for the imputation step. The file is located in src/synthwave/synthesizer/imputation/adults_imputation.R, note though it takes a very long time to do this. Just run it in the same directory as your adults_non_imputed_middle_fidelity file and let it cook.

Legacy

The original approach was to use Annual Population Survey/Labour Populations Survey for this. However, APS lacks links between individuals i.e. household ids and therefore doesn't fit our approach. We're aware of plans to add this information to the existing datasets, but there is no ETA.