Skip to content

Improve performance of daily mongodb loads #6

@Ameya05

Description

@Ameya05

The current mongodb implementation does the following things daily -

  1. Uses the existing SQLite intermediate databases for all stations.
  2. Drops existing mongodb database.
  3. Computes data states for all stations and all dates (from 01/01/1994 to today).
  4. Inserts these values into a new mongodb database.

The current process takes upwards of 24 hours to process approximately upto data states 10,000 GPS Stations. Daily re-computation of Historical data states is redundant and can be avoided.

A simple performance improvement would be compute only the end_date's data state and attach it to the mongodb collection everyday, instead of dropping and creating from scratch.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions