Swartzn/feat/add badgerdb block cache size config

[swartzn]

What does this PR do / why do we need it?

Required for all PRs.

These commits allow admins to properly size the block cache for their environment and workload. In local testing, both sync and remote, block cache is forced to evict entries before the job or work can complete. This leads to log messages such as,

(Info) Block cache metrics: hit: 7431865 miss: 330069 keys-added: 247014 keys-updated: 0 keys-evicted: 225046 cost-added: 768665002 cost-evicted: 701538757 sets-dropped: 0 sets-rejected: 18626 gets-dropped: 43712 gets-kept: 7710336 gets-total: 7761934 hit-ratio: 0.96 {"component": "workmgr", "database": "workJournal"}

(Warn) Block cache might be too small. Metrics: hit: 603894 miss: 953385 keys-added: 143312 keys-updated: 639 keys-evicted: 104512 cost
(Warn) added: 951290201 cost-evicted: 682855830 sets-dropped: 0 sets-rejected: 80025 gets-dropped: 56256 gets-kept: 1490752 gets-total: 1557279 hit-ratio: 0.39 {"component": "job", "database": "pathDB"}
Cache life expectancy (in seconds):
-- Histogram:
Min value: 0
Max value: 134
Count: 14598
50p: 8.00
75p: 16.00
90p: 16.00
[0, 2) 1240 8.49% 8.49%
[2, 4) 2293 15.71% 24.20%
[4, 8) 4899 33.56% 57.76%
[8, 16) 6007 41.15% 98.91%
[32, 64) 32 0.22% 99.13%
[64, 128) 122 0.84% 99.97%
[128, 256) 5 0.03% 100.00%

These are BadgerDB's builtin cache diagnostics. The Info log shows that everything is fine and that the majority of workJournal's entries are retrieved from cache. The Warn logs shows less than 40% of entries retrieved are from cache and it might be undersized.

In practice block cache could be hard to size but it should be approximately equal to job creation rate * average job lifespan * bytes per entry.

Related Issue(s)

Required when applicable.

Where should the reviewer(s) start reviewing this?

Only required for larger PRs when this may not be immediately obvious.

Are there any specific topics we should discuss before merging?

Not required.

What are the next steps after this PR?

Not required.

Checklist before merging:

Required for all PRs.

When creating a PR these are items to keep in mind that cannot be checked by GitHub actions:

• Documentation:
  • Does developer documentation (code comments, readme, etc.) need to be added or updated?
  • Does the user documentation need to be expanded or updated for this change?
• Testing:
  • Does this functionality require changing or adding new unit tests?
  • Does this functionality require changing or adding new integration tests?
• Git Hygiene:
  • Do commits adhere to Conventional Commits?
  • Is the commit history squashed down reasonably?

For more details refer to the Go coding standards and the pull request process.