-
Notifications
You must be signed in to change notification settings - Fork 1
Open
Labels
Description
The current mongodb implementation does the following things daily -
- Uses the existing SQLite intermediate databases for all stations.
- Drops existing mongodb database.
- Computes data states for all stations and all dates (from 01/01/1994 to today).
- Inserts these values into a new mongodb database.
The current process takes upwards of 24 hours to process approximately upto data states 10,000 GPS Stations. Daily re-computation of Historical data states is redundant and can be avoided.
A simple performance improvement would be compute only the end_date's data state and attach it to the mongodb collection everyday, instead of dropping and creating from scratch.
Reactions are currently unavailable