Further Optimize Maintainer Loop Logging

[adeutscher]

As I'm local-testing #75 , I find myself already a bit annoyed by the idempotency monitor announcement that I set up yesterday. It still maintains an important purpose, but also there's no point in re-checking if we know for a fact that there aren't any messages to handle. I think that we can improve on that.

Rough ideas with minimal impact to existing infrastructure:

• Rename the maintainer delay to be more indicative of our plans, like MaintainerWaitAsync
• If the number of in-flight jobs stored in the applied execution end arbiter is greater than zero, then sleep for the appointed amount of time through sleep service as we currently do.
  • For the idempotency monitor specifically, this is "good enough", since getting callbacks when the message state is adjusted is already documented elsewhere as a 'cure-worse-than-the-disease' kind of problem for similar logistics.
• If the number of of in-flight jobs stored in the applied execution end arbiter is zero, then pass off to waiting on a manual-reset-event.
  • Event state is managed by update callback for watched jobs.
  • This way, the heartbeat monitor and idempotency monitors only loop when there's a point in looping.
• Possible alternative idea: The manual-reset-event is always awaited before the standard delay, thus preventing monitor checks from running the instant a relevant job comes in. A brand-new job doesn't need to be heartbeated.
  • Maybe still gate an informative log message on event state? (related topic: can we gate an if statement on the event's state? Question for later).

[adeutscher]

Local testing for this is going exactly the way that I'd hoped:

worker-1  | DBG Hosting started
worker-1  | VRB Heartbeat Monitor: Waiting for watched events
worker-1  | VRB Idempotency Monitor: Waiting for watched events
worker-1  | VRB Worker thread for JobSubscriberManager done. Response: NotEnabled
worker-1  | INF Health listening on port 8081
worker-1  | VRB Fetched 0 jobs in 00:00:00.1155575
worker-1  | VRB Waiting before pulling more jobs, retrying in 2 s
worker-1  | VRB Fetched 1 jobs in 00:00:00.0082861
worker-1  | VRB Executor 0 received message 7d3e26c7-d569-4666-9200-7a44957be8ea
worker-1  | VRB Idempotency Monitor: 00:00:30 until next follow-up check
worker-1  | VRB Heartbeat Monitor: 00:03:44.9935558 until next heartbeat check
worker-1  | INF Sleeping for 2 seconds                     
worker-1  | VRB Executor 0 finished processing message 7d3e26c7-d569-4666-9200-7a44957be8ea. Result: Success
worker-1  | VRB Fetched 0 jobs in 00:00:00.0015752
worker-1  | VRB Waiting before pulling more jobs, retrying in 2 s
< Job loader noise >
worker-1  | VRB Fetched 0 jobs in 00:00:00.0020725
worker-1  | VRB Waiting before pulling more jobs, retrying in 16 s
worker-1  | VRB Idempotency Monitor: Waiting for watched events
worker-1  | VRB Fetched 0 jobs in 00:00:00.0017232
worker-1  | VRB Waiting before pulling more jobs, retrying in 30 s
worker-1  | VRB Fetched 0 jobs in 00:00:00.0017292
< More job loader noise >
worker-1  | VRB Fetched 0 jobs in 00:00:00.0017315
worker-1  | VRB Waiting before pulling more jobs, retrying in 30 s
worker-1  | VRB Heartbeat Monitor: Waiting for watched events

This probably only really affects local testing, but an if it helps to grease the wheels of progress then an optimization is an optimization.