test(dursto): add regression testing for Database

[victor-dumitrescu]

Closes RV-980.

What

Adds regression testing for Database along with the functionality required to (re-)generate regression test inputs.

Why

To make sure the semantics of Database operations don't unexpectedly change as we stabilise the NDS.

How

A new test_database_regression test is added which reads Database operations from checked-in JSON files, runs them on both backends via kv_test!, produces traces, and compares them with the corresponding checked-in JSON traces.

An xtask crate is added for durable-storage with a single command at the moment which will regenerate the inputs and run the test in order to also regenerate the outputs. Inputs are generated using database_operations_strategy in the range of 90..100 operations.

Under the unstable-test-utils flag, serialisation and deserialisation is implemented or derived for types which are used in either operations of traces.

Manually Testing

make all

Regressions

New regressions for Database are introduced in durable-storage/tests/inputs and expected outputs in durable-storage/tests/expected

Tasks for the Author

• Link all Linear issues related to this MR using magic words (e.g. part of, relates to, closes).
• Eliminate dead code and other spurious artefacts introduced in your changes.
• Document new public functions, methods and types.
• Make sure the documentation for updated functions, methods, and types is correct.
• Add tests for bugs that have been fixed.
• Explain changes to regression test captures when applicable.
• Write commit messages in agreement with our guidelines.
• Self-review your changes to ensure they are high-quality.
• Complete all of the above before assigning this MR to reviewers.

[codecov]

Codecov Report

❌ Patch coverage is 53.12500% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.09%. Comparing base (e57c89f) to head (174c72a).

Files with missing lines	Patch %	Lines
data/src/hash.rs	23.07%	10 Missing ⚠️
durable-storage/src/database.rs	72.72%	2 Missing and 1 partial ⚠️
durable-storage/src/key.rs	75.00%	0 Missing and 2 partials ⚠️

❌ Your patch check has failed because the patch coverage (53.12%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1074      +/-   ##
==========================================
- Coverage   90.14%   90.09%   -0.05%     
==========================================
  Files         130      130              
  Lines       28198    28230      +32     
  Branches    28198    28230      +32     
==========================================
+ Hits        25418    25435      +17     
- Misses       2031     2042      +11     
- Partials      749      753       +4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

Benchmark results for revision 97e2863:

Metric	Duration	TPS
Mean	1.520596116s	26.306
Worst	1.525708106s	26.217
Best	1.509626693s	26.497
Standard Deviation	±4.158066ms	±0.072

Full results

Run	Transfers	Duration	TPS
1	40	1.524532416s	26.238
2	40	1.525385835s	26.223
3	40	1.512048266s	26.454
4	40	1.525708106s	26.217
5	40	1.525370717s	26.223
6	40	1.521342686s	26.293
7	40	1.52201632s	26.281
8	40	1.519878293s	26.318
9	40	1.509626693s	26.497
10	40	1.52242297s	26.274
11	40	1.517957699s	26.351
12	40	1.523314897s	26.259
13	40	1.518418805s	26.343
14	40	1.520080982s	26.314
15	40	1.519732463s	26.320
16	40	1.518521386s	26.341
17	40	1.525354988s	26.223
18	40	1.521210197s	26.295
19	40	1.517405883s	26.361
20	40	1.52159271s	26.288

Compare the results above with those for the default branch.