What should the new database schemas be called?

[MoralCode]

In an upcoming version of CollectOSS we plan to include a database migration that updates the name of the database schemas.

I wanted to start a community discussion over the new naming we should use.

This change will likely require all downstream that are relying on our core database tables (including 8Knot and probably the grafana monitoring stack) to update some or all of their queries where the schema is being hard-coded.

Many such queries within CollectOSS itself will likely change.

Our current affected schemas are called:

• augur_data (for collected data used as part of analysis)
• augur_operations (for metadata relating to the collection process)

[MoralCode]

Option 1: basic

• collectoss_data / collectoss_operations

Option 2: generic

• collection_data / collection_operations

Option 3: snazzy

• forge_data / collection_operations

Misc other ideas:

• collected_data
• health_data
• core_data

[cdolfi]

I dont have any strong opinions, good with any of the 3!

[shlokgilda]

I prefer 2.

[MoralCode]

collection_data / collection_operations it is

[MoralCode]

On talking with Josh Berkus, his recommendation is not to use a prefix since CollectOSS is so heavy on databases (and IO) at large scales, so it doesnt really make much sense for it to be sharing a database with other applications anyway.

Also given the plans to attempt to codify a separate schema thats explicitly an external interface, i propose this naming scheme:

augur_data -> internal
augur_operations -> operations
new schema for views (materialized or not) that downstream projects use -> data

[cdolfi]

I find the proposed naming changes to be confusing. While I know that represents what your goals are for how our database is used, I don't think it's fair to anticipate others will use it the same way. I feel strongly that what is now augur_data should have data in its name. This change will break every notebook that currently exists and believe there should be a straight forward transition

[MoralCode]

yeah i was noodling on that transition and trying to figure out how to make it as smooth as possible for downstreams. The other way we could do it is:

augur_data -> data
augur_operations -> operations
new schema -> internal

Then we can move the tables to the "internal" schema piece by piece and replace them with views/materialized views as we go so that outward interface stays the same.

Ultimately though, anything that depends directly on core tables (without going through a view or materialized view) is going to be potentially subject to breaking changes as the schema improves (which is likely as we refactor things to be simpler and add support for additional platforms).

I definitely understand that people may be using the tables in ways I'm not aware of and want as many usecases as possible to remain supported. At this point my logic is that its better to do a single, well-documented round of breaking changes with a migration guide, rather than putting ourselves in a situation where we will need to drip feed the breaking changes out over a long period of time.

I.e. i think my solution would be:

1. go with renaming augur_data -> internal
2. ensure the existing materialized views are moved over to the data schema
3. for all current tables in the internal schema, create a view in the data schema that is a select * from the corresponding internal table (possibly with some small renames, like having the commits table be exposed as commit_files)

This would make the data schema a drop-in for the old augur_data one, meaning the migration for downstreams (8knot) is as simple as augur_data -> data, while also codifying the data schema that boundary layer. I can then use some of the testing techniques we have talked about for 8Knot to create tests that ensure the structure of the data schema does not change accidentally/without being thoroughly documented in release notes, meaning we can document a commitment to keeping that stable and following semantic versioning with releases (ultimately making life easier for downstreams)

How does that sound?

[cdolfi]

This feels very overcomplicated but maybe im missing something. @jberkus what are your thoughts on this?

[MoralCode]

internally, it is more complicated than a simple rename, but externally (as far as the migration guide/what downstreams will need to do) I think this is the path that allows us to have more control over breaking changes (and being able to avoid or schedule them without delaying the CollectOSS roadmap nearly as much), which is going to be better/more stable for downstreams long term.

Technical questions for josh:

• will using a view that is essentially select * from table automatically pass through any indexes?
• will we be able to logical database replication from this "data" (views) schema?