fix: running two models at exact same moment with reusable_session set to true causes AlreadyExistsException error.

[akmalsoliev]

Description

running two models at exact same moment with reusable_session set to true causes AlreadyExistsException error.

This is mainly present when running dbt-glue on Airflow with Cosmos dbt plugin using sub-processes, that runs dbt independently.

Checklist

• I have signed the CLA
• I have run this code in development and it appears to resolve the stated issue
• This PR includes tests, or tests are not required/relevant for this PR
• I have updated the CHANGELOG.md and added information about my change to the "dbt-glue next" section.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[jausanca]

Is it safe to have two concurrent processes reuse the same session? It would be equivalent to two threads pushing code to be executed to the same single process session, where statements, I think, are executed in order, same as they would be when running cells on a jupiter notebook.

[akmalsoliev]

Is it safe to have two concurrent processes reuse the same session? It would be equivalent to two threads pushing code to be executed to the same single process session, where statements, I think, are executed in order, same as they would be when running cells on a jupiter notebook.

If your model takes a significantly low compute and you provide extra workers then it is fine(I am running on g1x with 4 worker). Never had an issue on my end as my models merge on a heavy model, which is a single running model. In case of switching to session/model would kill my run time (1-3 minutes of session provisioning)

[jausanca]

I'm not asking in terms of compute provisioning, but in terms of how code is evaluated on the session. Does it run sequentially, or do statements actually run in parallel if they are executed concurrently? My assumption is that there's a single python interpreter and statements are queued (even if there's parallelism handling the actual requested data computation). Would be nice if we had someone from AWS confirm if that's the case or not.

It's true that having to wait for model provisioning on a cosmos execution would introduce too much inefficiency but, if the computation steps are executed sequentially on the session's end, maybe you don't win anything by trying to have the models executed in parallel.

[akmalsoliev]

I'm not asking in terms of compute provisioning, but in terms of how code is evaluated on the session. Does it run sequentially, or do statements actually run in parallel if they are executed concurrently? My assumption is that there's a single python interpreter and statements are queued (even if there's parallelism handling the actual requested data computation). Would be nice if we had someone from AWS confirm if that's the case or not.

It's true that having to wait for model provisioning on a cosmos execution would introduce too much inefficiency but, if the computation steps are executed sequentially on the session's end, maybe you don't win anything by trying to have the models executed in parallel.

This is completely out of scope of this PR, would recommend opening a discussion issue.

But yes models do build in parallel when even running on a single session.