Skip to content

Breaking up dependency hash logic to reduce duplicate partition in Dagster #1365

@tech3371

Description

@tech3371

Yeah, we would need to find some way to remove the spacecraft clock kernel from all the existing hashes. Seems like the best way would be to just use is_duplicate_job and figure out some way to distinguish the old files. Maybe we could like make the dependency hash one character longer? Then it would be immediately easy to tell if it's in the old system and already run, VS a processing job...

Originally posted by @maxinelasp in #1353 (comment)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    Status

    Todo

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions