perf: Logical replication perf experiments

[bdewilde]

changes

• adds msgspec as package dependency and sets MsgSpecWriter as tap's message writer class
• minimizes variable/function access in loop over messages
• consolidates row update operations when consuming a message
• reduces replication status interval from 5s to 1s (afacit this matches what the pipelinewise variant uses; I figured I'd give it a shot)

context

Diving deeper into the code wrt issue #587

These changes may or may not be of interest for merging; they're mostly just a demonstration of what I found.

details

All together, these changes reduced runtimes for my example E+L pipeline from ~112s to ~107s. Not very impressive, I know. Here's the py-spy flame graph based on this repo's main branch:

And here's the (very similar) equivalent based on the latest version of my branch:

My changes only touch a small part of the tap's call stack:

Unfortunately, I wasn't able to do anything about the slowest parts of the tap:

• checking if the db cursor has has had any messages within the timeout before breaking out of the loop, here
• writing messages to stdout via the meltano sdk, here

For what it's worth, here is the corresponding functionality in the pipelinewise variant's code. There's a lot of additional logic there, but I had a hard time understanding the key differences with this variant.

questions

• Is there any way to make the check for whether or not to break out of the messages loop faster?
• any idea why writing messages is so slow, even using msgspec?

[edgarrmondragon]

Hi @bdewilde, thanks for taking the time to post this!

Is there any way to make the check for whether or not to break out of the messages loop faster?

I'm not sure what this refers to. Is there a specific location in the code where you see this happening?

any idea why writing messages is so slow, even using msgspec?

I'm reading through the performance docs¹ and I don't see anything obvious we're missing other than using msgspec.Struct for messages.

The serialization itself with msgspec should be considerably faster even without using Structs, so I think we're doing something wrong elsewhere. I know try/except blocks inside loops can be expensive, and get_records also seems to be taking longer that it probably should, so going one lever down from SQLAlchemy to the native postgres client is another thing to explore there.

Footnotes

1. https://jcristharif.com/msgspec/perf-tips.html#use-structs ↩

[bdewilde]

Is there any way to make the check for whether or not to break out of the messages loop faster?

I'm not sure what this refers to. Is there a specific location in the code where you see this happening?

Sorry about the confusion! This was in reference to an earlier comment in the PR about apparent perf bottlenecks:

checking if the db cursor has has had any messages within the timeout before breaking out of the loop, here

any idea why writing messages is so slow, even using msgspec?

I'm reading through the performance docs1 and I don't see anything obvious we're missing other than using msgspec.Struct for messages.

The serialization itself with msgspec should be considerably faster even without using Structs, so I think we're doing something wrong elsewhere. I know try/except blocks inside loops can be expensive, and get_records also seems to be taking longer that it probably should, so going one lever down from SQLAlchemy to the native postgres client is another thing to explore there.

I'll try to take a peek at the sdk's usage of msgspec, thanks for the link to perf tips 👍