New to MIMIC database - preprocessing issues

[DaisyP2025]

Hi everyone,

I'm a research scientist at King's College London and I'm relatively new to working with MIMIC data. I've been trying to get started with MIMIC-III and IV by downloading the CSV files and working with them in Python/pandas. So far, my experience has been... challenging.

For example, when I try to download sepsis patients with 1Hz vital sign data, I need to:

• Download several large compressed CSV files (multiple GB each)
• Spend a lot of time figuring out which tables have what data
• Write scripts to join different tables together (PATIENTS, ADMISSIONS, CHARTEVENTS, LABEVENTS, etc.)
• Understand the data structure and relationships
• Start over each time when I need a different cohort (e.g., COPD vs sepsis)

I'm about 2 weeks in and still haven't gotten to my actual analysis yet.

From reading online, I see people mention:

• Setting up local PostgreSQL databases
• Using BigQuery on GCP
• Tools like MIMIC-Extract (but it seems outdated?)
• PyHealth (just learned about this)

I'm genuinely curious:

1. Is this normal? Does it get easier once you learn the system?
2. What workflow do experienced MIMIC users actually use?
3. Are there tools or resources I should know about that would help?

I don't want to reinvent the wheel if there's a better approach. Any guidance from folks who've been through this learning curve would be really helpful.

Thank you!

[tompollard]

Hi @DaisyP2025, I would suggest working with the copy of MIMIC available in BigQuery. See: https://mimic.mit.edu/docs/gettingstarted/cloud/ for more details.

You can create a Colab notebook that connects directly to the BigQuery instance for your analysis. Outdated, but e.g. https://github.com/MIT-LCP/2019_aarhus_critical_data/blob/master/tutorials/mimic-iii/mimic-iii-tutorial-aarhus.ipynb

You'll find code to help with extracting cohorts etc at: https://github.com/MIT-LCP/mimic-code (also some notebooks, e.g. https://github.com/MIT-LCP/mimic-code/tree/main/mimic-iv/notebooks, though I realise these are outdated...we will try to add some more current ones).